Packages

  • package root
    Definition Classes
    root
  • package org
    Definition Classes
    root
  • package scalatest

    ScalaTest's main traits, classes, and other members, including members supporting ScalaTest's DSL for the Scala interpreter.

    ScalaTest's main traits, classes, and other members, including members supporting ScalaTest's DSL for the Scala interpreter.

    Definition Classes
    org
  • package compatible
    Definition Classes
    scalatest
  • package concurrent

    ScalaTest's main traits, classes, and other members, including members supporting ScalaTest's DSL for the Scala interpreter.

    ScalaTest's main traits, classes, and other members, including members supporting ScalaTest's DSL for the Scala interpreter.

    Definition Classes
    scalatest
  • package enablers
    Definition Classes
    scalatest
  • package events
    Definition Classes
    scalatest
  • package exceptions
    Definition Classes
    scalatest
  • package featurespec
    Definition Classes
    scalatest
  • package fixture

    Package fixture deprecated types.

    Package fixture deprecated types.

    Definition Classes
    scalatest
  • package flatspec
    Definition Classes
    scalatest
  • package freespec
    Definition Classes
    scalatest
  • package funspec
    Definition Classes
    scalatest
  • package funsuite
    Definition Classes
    scalatest
  • package matchers
    Definition Classes
    scalatest
  • package path
    Definition Classes
    scalatest
  • package prop

    Scalatest support for Property-based testing.

    Scalatest support for Property-based testing.

    Introduction to Property-based Testing

    In traditional unit testing, you write tests that describe precisely what the test will do: create these objects, wire them together, call these functions, assert on the results, and so on. It is clear and deterministic, but also limited, because it only covers the exact situations you think to test. In most cases, it is not feasible to test all of the possible combinations of data that might arise in real-world use.

    Property-based testing works the other way around. You describe properties -- rules that you expect your classes to live by -- and describe how to test those properties. The test system then generates relatively large amounts of synthetic data (with an emphasis on edge cases that tend to make things break), so that you can see if the properties hold true in these situations.

    As a result, property-based testing is scientific in the purest sense: you are stating a hypothesis about how things should work (the property), and the system is trying to falsify that hypothesis. If the tests pass, that doesn't prove the property holds, but it at least gives you some confidence that you are probably correct.

    Property-based testing is deliberately a bit random: while the edge cases get tried upfront, the system also usually generates a number of random values to try out. This makes things a bit non-deterministic -- each run will be tried with somewhat different data. To make it easier to debug, and to build regression tests, the system provides tools to re-run a failed test with precisely the same data.

    Background

    TODO: Bill should insert a brief section on QuickCheck, ScalaCheck, etc, and how this system is similar and different.

    Using Property Checks

    In order to use the tools described here, you should import this package:

    import org.scalatest._
    import org.scalatest.prop._

    This library is designed to work well with the types defined in Scalactic, and some functions take types such as PosZInt as parameters. So it can also be helpful to import those with:

    import org.scalactic.anyvals._

    In order to call forAll, the function that actually performs property checks, you will need to either extend or import GeneratorDrivenPropertyChecks, like this:

    class DocExamples extends FlatSpec with Matchers with GeneratorDrivenPropertyChecks {

    There's nothing special about FlatSpec, though -- you may use any of ScalaTest's styles with property checks. GeneratorDrivenPropertyChecks extends CommonGenerators, so it also provides access to the many utilities found there.

    What Does a Property Look Like?

    Let's check a simple property of Strings -- that if you concatenate a String to itself, its length will be doubled:

    "Strings" should "have the correct length when doubled" in {
      forAll { (s: String) =>
        val s2 = s * 2
        s2.length should equal (s.length * 2)
      }
    }

    (Note that the examples here are all using the FlatSpec style, but will work the same way with any of ScalaTest's styles.)

    As the name of the tests suggests, the property we are testing is the length of a String that has been doubled.

    The test begins with forAll. This is usually the way you'll want to begin property checks, and that line can be read as, "For all Strings, the following should be true".

    The test harness will generate a number of Strings, with various contents and lengths. For each one, we compute s * 2. (* is a function on String, which appends the String to itself as many times as you specify.) And then we check that the length of the doubled String is twice the length of the original one.

    Using Specific Generators

    Let's try a more general version of this test, multiplying arbitrary Strings by arbitrary multipliers:

    "Strings" should "have the correct length when multiplied" in {
      forAll { (s: String, n: PosZInt) =>
        val s2 = s * n.value
        s2.length should equal (s.length * n.value)
      }
    }

    Again, you can read the first line of the test as "For all Strings, and all non-negative Integers, the following should be true". (PosZInt is a type defined in Scalactic, which can be any positive integer, including zero. It is appropriate to use here, since multiplying a String by a negative number doesn't make sense.)

    This intuitively makes sense, but when we try to run it, we get a JVM Out of Memory error! Why? Because the test system tries to test with the "edge cases" first, and one of the more important edge cases is Int.MaxValue. It is trying to multiply a String by that, which is far larger than the memory of even a big computer, and crashing.

    So we want to constrain our test to sane values of n, so that it doesn't crash. We can do this by using more specific Generators.

    When we write a forAll test like the above, ScalaTest has to generate the values to be tested -- the semi-random Strings, Ints and other types that you are testing. It does this by calling on an implicit Generator for the desired type. The Generator generates values to test, starting with the edge cases and then moving on to randomly-selected values.

    ScalaTest has built-in Generators for many major types, including String and PosZInt, but these Generators are generic: they will try any value, including values that can break your test, as shown above. But it also provides tools to let you be more specific.

    Here is the fixed version of the above test:

    "Strings" should "have the correct length when multiplied" in {
      forAll(strings, posZIntsBetween(0, 1000))
      { (s: String, n: PosZInt) =>
        val s2 = s * n.value
        s2.length should equal (s.length * n.value)
      }
    }

    This is using a variant of forAll, which lets you specify the Generators to use instead of just picking the implicit one. CommonGenerators.strings is the built-in Generator for Strings, the same one you were getting implicitly. (The other built-ins can be found in CommonGenerators. They are mixed into GeneratorDrivenPropertyChecks, so they are readily available.)

    But CommonGenerators.posZIntsBetween is a function that creates a Generator that selects from the given values. In this case, it will create a Generator that only creates numbers from 0 to 1000 -- small enough to not blow up our computer's memory. If you try this test, this runs correctly.

    The moral of the story is that, while using the built-in Generators is very convenient, and works most of the time, you should think about the data you are trying to test, and pick or create a more-specific Generator when the test calls for it.

    CommonGenerators contains many functions that are helpful in common cases. In particular:

    • xxsBetween (where xxs might be Int, Long, Float or most other significant numeric types) gives you a value of the desired type in the given range, as in the posZIntsBetween() example above.
    • CommonGenerators.specificValue and CommonGenerators.specificValues create Generators that produce either one specific value every time, or one of several values randomly. This is useful for enumerations and types that behave like enumerations.
    • CommonGenerators.evenly and CommonGenerators.frequency create higher-level Generators that call other Generators, either more or less equally or with a distribution you define.

    Testing Your Own Types

    Testing the built-in types isn't very interesting, though. Usually, you have your own types that you want to check the properties of. So let's build up an example piece by piece.

    Say you have this simple type:

    sealed trait Shape {
      def area: Double
    }
    case class Rectangle(width: Int, height: Int) extends Shape {
      require(width > 0)
      require(height > 0)
      def area: Double = width * height
    }

    Let's confirm a nice straightforward property that is surely true: that the area is greater than zero:

    "Rectangles" should "have a positive area" in {
       forAll { (w: PosInt, h: PosInt) =>
         val rect = Rectangle(w, h)
         rect.area should be > 0.0
       }
     }

    Note that, even though our class takes ordinary Ints as parameters (and checks the values at runtime), it is actually easier to generate the legal values using Scalactic's PosInt type.

    This should work, right? Actually, it doesn't -- if we run it a few times, we quickly hit an error!

    [info] Rectangles
    [info] - should have a positive area *** FAILED ***
    [info]   GeneratorDrivenPropertyCheckFailedException was thrown during property evaluation.
    [info]    (DocExamples.scala:42)
    [info]     Falsified after 2 successful property evaluations.
    [info]     Location: (DocExamples.scala:42)
    [info]     Occurred when passed generated values (
    [info]       None = PosInt(399455539),
    [info]       None = PosInt(703518968)
    [info]     )
    [info]     Init Seed: 1568878346200

    TODO: fix the above error to reflect the better errors we should get when we merge in the code being forward-ported from 3.0.5.

    Looking at it, we can see that the numbers being used are pretty large. What happens when we multiply them together?

    scala> 399455539 * 703518968
    res0: Int = -2046258840

    We're hitting an Int overflow problem here: the numbers are too big to multiply together and still get an Int. So we have to fix our area function:

    case class Rectangle(width: Int, height: Int) extends Shape {
      require(width > 0)
      require(height > 0)
      def area: Double = width.toLong * height.toLong
    }

    Now, when we run our property check, it consistently passes. Excellent -- we've caught a bug, because ScalaTest tried sufficiently large numbers.

    Composing Your Own Generators

    Doing things as shown above works, but having to generate the parameters and construct a Rectangle every time is a nuisance. What we really want is to create our own Generator that just hands us Rectangles, the same way we can do for PosInt. Fortunately, this is easy.

    Generators can be composed in for comprehensions. So we can create our own Generator for Rectangle like this:

    implicit val rectGenerator = for {
      w <- posInts
      h <- posInts
    }
      yield Rectangle(w, h)

    Taking that line by line:

    w <- posInts

    CommonGenerators.posInts is the built-in Generator for positive Ints. So this line puts a randomly-generated positive Int in w, and

    h <- posInts

    this line puts another one in h. Finally, this line:

    yield Rectangle(w, h)

    combines w and h to make a Rectangle.

    That's pretty much all you need in order to build any normal case class -- just build it out of the Generators for the type of each field. (And if the fields are complex data structures themselves, build Generators for them the same way, until you are just using primitives.)

    Now, our property check becomes simpler:

    "Generated Rectangles" should "have a positive area" in {
       forAll { (rect: Rectangle) =>
         rect.area should be > 0.0
       }
     }

    That's about as close to plain English as we can reasonably hope for!

    Filtering Values with whenever()

    Sometimes, not all of your generated values make sense for the property you want to check -- you know (via external information) that some of these values will never come up. In cases like this, you can create a custom Generator that only creates the values you do want, but it's often easier to just use Whenever.whenever. (Whenever is mixed into GeneratorDrivenPropertyChecks, so this is available when you need it.)

    The Whenever.whenever function can be used inside of GeneratorDrivenPropertyChecks.forAll. It says that only the filtered values should be used, and anything else should be discarded. For example, look at this property:

    "Fractions" should "get smaller when squared" in {
      forAll { (n: Float) =>
        whenever(n > 0 && n < 1) {
          (n * n) should be < n
        }
      }
    }

    We are testing a property of numbers less than 1, so we filter away everything that is not the numbers we want. This property check succeeds, because we've screened out the values that would make it fail.

    Discard Limits

    You shouldn't push Whenever.whenever too far, though. This system is all about trying random data, but if too much of the random data simply isn't usable, you can't get valid answers, and the system tracks that.

    For example, consider this apparently-reasonable test:

    "Space Chars" should "not also be letters" in {
      forAll { (c: Char) =>
        whenever (c.isSpaceChar) {
          assert(!c.isLetter)
        }
      }
    }

    Although the property is true, this test will fail with an error like this:

    [info] Lowercase Chars
    [info] - should upper-case correctly *** FAILED ***
    [info]   Gave up after 0 successful property evaluations. 49 evaluations were discarded.
    [info]   Init Seed: 1568855247784

    Because the vast majority of Chars are not spaces, nearly all of the generated values are being discarded. As a result, the system gives up after a while. In cases like this, you usually should write a custom Generator instead.

    The proportion of how many discards to permit, relative to the number of successful checks, is configuration-controllable. See GeneratorDrivenPropertyChecks for more details.

    Randomization

    The point of Generator is to create pseudo-random values for checking properties. But it turns out to be very inconvenient if those values are actually random -- that would mean that, when a property check fails occasionally, you have no good way to invoke that specific set of circumstances again for debugging. We want "randomness", but we also want it to be deterministic, and reproducible when you need it.

    To support this, all "randomness" in ScalaTest's property checking system uses the Randomizer class. You start by creating a Randomizer using an initial seed value, and call that to get your "random" value. Each call to a Randomizer function returns a new Randomizer, which you should use to fetch the next value.

    GeneratorDrivenPropertyChecks.forAll uses Randomizer under the hood: each time you run a forAll-based test, it will automatically create a new Randomizer, which by default is seeded based on the current system time. You can override this, as discussed below.

    Since Randomizer is actually deterministic (the "random" values are unobvious, but will always be the same given the same initial seed), this means that re-running a test with the same seed will produce the same values.

    If you need random data for your own Generators and property checks, you should use Randomizer in the same way; that way, your tests will also be re-runnable, when needed for debugging.

    Debugging, and Re-running a Failed Property Check

    In Testing Your Own Types above, we found to our surprise that the property check failed with this error:

    [info] Rectangles
    [info] - should have a positive area *** FAILED ***
    [info]   GeneratorDrivenPropertyCheckFailedException was thrown during property evaluation.
    [info]    (DocExamples.scala:42)
    [info]     Falsified after 2 successful property evaluations.
    [info]     Location: (DocExamples.scala:42)
    [info]     Occurred when passed generated values (
    [info]       None = PosInt(399455539),
    [info]       None = PosInt(703518968)
    [info]     )
    [info]     Init Seed: 1568878346200

    There must be a bug here -- but once we've fixed it, how can we make sure that we are re-testing exactly the same case that failed?

    This is where the pseudo-random nature of Randomizer comes in, and why it is so important to use it consistently. So long as all of our "random" data comes from that, then all we need to do is re-run with the same seed.

    That's why the Init Seed shown in the message above is crucial. We can re-use that seed -- and therefore get exactly the same "random" data -- by using the -S flag to ScalaTest.

    So you can run this command in sbt to re-run exactly the same property check:

    testOnly *DocExamples -- -z "have a positive area" -S 1568878346200

    Taking that apart:

    • testOnly *DocExamples says that we only want to run suites whose paths end with DocExamples
    • -z "have a positive area" says to only run tests whose names include that string.
    • -S 1568878346200 says to run all tests with a "random" seed of 1568878346200

    By combining these flags, you can re-run exactly the property check you need, with the right random seed to make sure you are re-creating the failed test. You should get exactly the same failure over and over until you fix the bug, and then you can confirm your fix with confidence.

    Configuration

    In general, forAll() works well out of the box. But you can tune several configuration parameters when needed. See GeneratorDrivenPropertyChecks for info on how to set configuration parameters for your test.

    Table-Driven Properties

    Sometimes, you want something in between traditional hard-coded unit tests and Generator-driven, randomized tests. Instead, you sometimes want to check your properties against a specific set of inputs.

    (This is particularly useful for regression tests, when you have found certain inputs that have caused problems in the past, and want to make sure that they get consistently re-tested.)

    ScalaTest supports these, by mixing in TableDrivenPropertyChecks. See the documentation for that class for the full details.

    Definition Classes
    scalatest
  • package propspec
    Definition Classes
    scalatest
  • AnyPropSpec
  • AnyPropSpecLike
  • FixtureAnyPropSpec
  • FixtureAnyPropSpecLike
  • package refspec
    Definition Classes
    scalatest
  • package tagobjects
    Definition Classes
    scalatest
  • package tags
    Definition Classes
    scalatest
  • package time
    Definition Classes
    scalatest
  • package tools
    Definition Classes
    scalatest
  • package words
    Definition Classes
    scalatest
  • package wordspec
    Definition Classes
    scalatest
p

org.scalatest

propspec

package propspec

Type Members

  1. class AnyPropSpec extends AnyPropSpecLike

    A suite of property-based tests.

    A suite of property-based tests.

    Recommended Usage: Class AnyPropSpec is a good fit for teams that want to write tests exclusively in terms of property checks, and is also a good choice for writing the occasional test matrix when a different style trait is chosen as the main unit testing style.

    Here's an example AnyPropSpec:

    package org.scalatest.examples.propspec
    
    import org.scalatest._
    import prop._
    import scala.collection.immutable._
    
    class SetSpec extends propspec.AnyPropSpec with TableDrivenPropertyChecks with Matchers {
    
      val examples =
        Table(
          "set",
          BitSet.empty,
          HashSet.empty[Int],
          TreeSet.empty[Int]
        )
    
      property("an empty Set should have size 0") {
        forAll(examples) { set =>
          set.size should be (0)
        }
      }
    
      property("invoking head on an empty set should produce NoSuchElementException") {
        forAll(examples) { set =>
          a [NoSuchElementException] should be thrownBy { set.head }
        }
      }
    }
    

    You can run a AnyPropSpec by invoking execute on it. This method, which prints test results to the standard output, is intended to serve as a convenient way to run tests from within the Scala interpreter. For example, to run SetSpec from within the Scala interpreter, you could write:

    scala> org.scalatest.run(new SetSpec)
    

    And you would see:

    SetSpec:
    - an empty Set should have size 0
    - invoking head on an empty Set should produce NoSuchElementException
    

    Or, to run just the “an empty Set should have size 0” method, you could pass that test's name, or any unique substring of the name, such as "size 0" or even just "0". Here's an example:

    scala> org.scalatest.run(new SetSpec, "size 0")
    SetSpec:
    - an empty Set should have size 0
    

    You can also pass to execute a config map of key-value pairs, which will be passed down into suites and tests, as well as other parameters that configure the run itself. For more information on running in the Scala interpreter, see the documentation for execute (below) and the ScalaTest shell.

    The execute method invokes a run method that takes two parameters. This run method, which actually executes the suite, will usually be invoked by a test runner, such as run, tools.Runner, a build tool, or an IDE.

    property” is a method, defined in AnyPropSpec, which will be invoked by the primary constructor of SetSpec. You specify the name of the test as a string between the parentheses, and the test code itself between curly braces. The test code is a function passed as a by-name parameter to property, which registers it for later execution.

    A AnyPropSpec's lifecycle has two phases: the registration phase and the ready phase. It starts in registration phase and enters ready phase the first time run is called on it. It then remains in ready phase for the remainder of its lifetime.

    Tests can only be registered with the property method while the AnyPropSpec is in its registration phase. Any attempt to register a test after the AnyPropSpec has entered its ready phase, i.e., after run has been invoked on the AnyPropSpec, will be met with a thrown TestRegistrationClosedException. The recommended style of using AnyPropSpec is to register tests during object construction as is done in all the examples shown here. If you keep to the recommended style, you should never see a TestRegistrationClosedException.

    Ignored tests

    To support the common use case of temporarily disabling a test, with the good intention of resurrecting the test at a later time, AnyPropSpec provides registration methods that start with ignore instead of property. Here's an example:

    package org.scalatest.examples.suite.ignore
    
    import org.scalatest._
    import prop._
    import scala.collection.immutable._
    
    class SetSpec extends propspec.AnyPropSpec with TableDrivenPropertyChecks with Matchers {
    
      val examples =
        Table(
          "set",
          BitSet.empty,
          HashSet.empty[Int],
          TreeSet.empty[Int]
        )
    
      ignore("an empty Set should have size 0") {
        forAll(examples) { set =>
          set.size should be (0)
        }
      }
    
      property("invoking head on an empty set should produce NoSuchElementException") {
        forAll(examples) { set =>
          a [NoSuchElementException] should be thrownBy { set.head }
        }
      }
    }
    

    If you run this version of SetSuite with:

    scala> org.scalatest.run(new SetSpec)
    

    It will run only the second test and report that the first test was ignored:

    SetSuite:
    - an empty Set should have size 0 !!! IGNORED !!!
    - invoking head on an empty Set should produce NoSuchElementException
    

    Informers

    One of the parameters to AnyPropSpec's run method is a Reporter, which will collect and report information about the running suite of tests. Information about suites and tests that were run, whether tests succeeded or failed, and tests that were ignored will be passed to the Reporter as the suite runs. Most often the reporting done by default by AnyPropSpec's methods will be sufficient, but occasionally you may wish to provide custom information to the Reporter from a test. For this purpose, an Informer that will forward information to the current Reporter is provided via the info parameterless method. You can pass the extra information to the Informer via its apply method. The Informer will then pass the information to the Reporter via an InfoProvided event. Here's an example that shows both a direct use as well as an indirect use through the methods of GivenWhenThen:

    package org.scalatest.examples.propspec.info
    
    import org.scalatest._
    import prop._
    import collection.mutable
    
    class SetSpec extends propspec.AnyPropSpec with TableDrivenPropertyChecks with GivenWhenThen {
    
      val examples =
        Table(
          "set",
          mutable.BitSet.empty,
          mutable.HashSet.empty[Int],
          mutable.LinkedHashSet.empty[Int]
        )
    
      property("an element can be added to an empty mutable Set") {
    
        forAll(examples) { set =>
    
          info("----------------")
    
          Given("an empty mutable " + set.getClass.getSimpleName)
          assert(set.isEmpty)
    
          When("an element is added")
          set += 99
    
          Then("the Set should have size 1")
          assert(set.size === 1)
    
          And("the Set should contain the added element")
          assert(set.contains(99))
        }
      }
    }
    

    If you run this AnyPropSpec from the interpreter, you will see the following output:

    scala> org.scalatest.run(new SetSpec)
    SetSpec:
    - an element can be added to an empty mutable Set
      + ----------------
      + Given an empty mutable BitSet
      + When an element is added
      + Then the Set should have size 1
      + And the Set should contain the added element
      + ----------------
      + Given an empty mutable HashSet
      + When an element is added
      + Then the Set should have size 1
      + And the Set should contain the added element
      + ----------------
      + Given an empty mutable LinkedHashSet
      + When an element is added
      + Then the Set should have size 1
      + And the Set should contain the added element
    

    Documenters

    AnyPropSpec also provides a markup method that returns a Documenter, which allows you to send to the Reporter text formatted in Markdown syntax. You can pass the extra information to the Documenter via its apply method. The Documenter will then pass the information to the Reporter via an MarkupProvided event.

    Here's an example AnyPropSpec that uses markup:

    package org.scalatest.examples.propspec.markup
    
    import org.scalatest._
    import prop._
    import collection.mutable
    
    class SetSpec extends propspec.AnyPropSpec with TableDrivenPropertyChecks with GivenWhenThen {
    
      markup { """
    
    Mutable Set
    -----------
    
    A set is a collection that contains no duplicate elements.
    
    To implement a concrete mutable set, you need to provide implementations
    of the following methods:
    
        def contains(elem: A): Boolean
        def iterator: Iterator[A]
        def += (elem: A): this.type
        def -= (elem: A): this.type
    
    If you wish that methods like `take`,
    `drop`, `filter` return the same kind of set,
    you should also override:
    
        def empty: This
    
    It is also good idea to override methods `foreach` and
    `size` for efficiency.
    
      """ }
    
      val examples =
        Table(
          "set",
          mutable.BitSet.empty,
          mutable.HashSet.empty[Int],
          mutable.LinkedHashSet.empty[Int]
        )
    
      property("an element can be added to an empty mutable Set") {
    
        forAll(examples) { set =>
    
          info("----------------")
    
          Given("an empty mutable " + set.getClass.getSimpleName)
          assert(set.isEmpty)
    
          When("an element is added")
          set += 99
    
          Then("the Set should have size 1")
          assert(set.size === 1)
    
          And("the Set should contain the added element")
          assert(set.contains(99))
        }
    
        markup("This test finished with a **bold** statement!")
      }
    }
    

    Although all of ScalaTest's built-in reporters will display the markup text in some form, the HTML reporter will format the markup information into HTML. Thus, the main purpose of markup is to add nicely formatted text to HTML reports. Here's what the above SetSpec would look like in the HTML reporter:

    Notifiers and alerters

    ScalaTest records text passed to info and markup during tests, and sends the recorded text in the recordedEvents field of test completion events like TestSucceeded and TestFailed. This allows string reporters (like the standard out reporter) to show info and markup text after the test name in a color determined by the outcome of the test. For example, if the test fails, string reporters will show the info and markup text in red. If a test succeeds, string reporters will show the info and markup text in green. While this approach helps the readability of reports, it means that you can't use info to get status updates from long running tests.

    To get immediate (i.e., non-recorded) notifications from tests, you can use note (a Notifier) and alert (an Alerter). Here's an example showing the differences:

    package org.scalatest.examples.propspec.note
    
    import org.scalatest._
    import prop._
    import collection.mutable
    
    class SetSpec extends propspec.AnyPropSpec with TableDrivenPropertyChecks {
    
      val examples =
        Table(
          "set",
          mutable.BitSet.empty,
          mutable.HashSet.empty[Int],
          mutable.LinkedHashSet.empty[Int]
        )
    
      property("an element can be added to an empty mutable Set") {
    
        info("info is recorded")
        markup("markup is *also* recorded")
        note("notes are sent immediately")
        alert("alerts are also sent immediately")
    
        forAll(examples) { set =>
    
          assert(set.isEmpty)
          set += 99
          assert(set.size === 1)
          assert(set.contains(99))
        }
      }
    }
    

    Because note and alert information is sent immediately, it will appear before the test name in string reporters, and its color will be unrelated to the ultimate outcome of the test: note text will always appear in green, alert text will always appear in yellow. Here's an example:

    scala> org.scalatest.run(new SetSpec)
    SetSpec:
      + notes are sent immediately
      + alerts are also sent immediately
    - an element can be added to an empty mutable Set
      + info is recorded
      + markup is *also* recorded
    

    Another example is slowpoke notifications. If you find a test is taking a long time to complete, but you're not sure which test, you can enable slowpoke notifications. ScalaTest will use an Alerter to fire an event whenever a test has been running longer than a specified amount of time.

    In summary, use info and markup for text that should form part of the specification output. Use note and alert to send status notifications. (Because the HTML reporter is intended to produce a readable, printable specification, info and markup text will appear in the HTML report, but note and alert text will not.)

    Pending tests

    A pending test is one that has been given a name but is not yet implemented. The purpose of pending tests is to facilitate a style of testing in which documentation of behavior is sketched out before tests are written to verify that behavior (and often, before the behavior of the system being tested is itself implemented). Such sketches form a kind of specification of what tests and functionality to implement later.

    To support this style of testing, a test can be given a name that specifies one bit of behavior required by the system being tested. The test can also include some code that sends more information about the behavior to the reporter when the tests run. At the end of the test, it can call method pending, which will cause it to complete abruptly with TestPendingException.

    Because tests in ScalaTest can be designated as pending with TestPendingException, both the test name and any information sent to the reporter when running the test can appear in the report of a test run. (The code of a pending test is executed just like any other test.) However, because the test completes abruptly with TestPendingException, the test will be reported as pending, to indicate the actual test, and possibly the functionality, has not yet been implemented.

    You can mark tests pending in AnyPropSpec like this:

    import org.scalatest._
    import prop._
    import scala.collection.immutable._
    
    class SetSpec extends propspec.AnyPropSpec with TableDrivenPropertyChecks with Matchers {
    
      val examples =
        Table(
          "set",
          BitSet.empty,
          HashSet.empty[Int],
          TreeSet.empty[Int]
        )
    
      property("an empty Set should have size 0") (pending)
    
      property("invoking head on an empty set should produce NoSuchElementException") {
        forAll(examples) { set =>
          a [NoSuchElementException] should be thrownBy { set.head }
        }
      }
    }
    

    (Note: "(pending)" is the body of the test. Thus the test contains just one statement, an invocation of the pending method, which throws TestPendingException.) If you run this version of SetSuite with:

    scala> org.scalatest.run(new SetSuite)
    

    It will run both tests, but report that first test is pending. You'll see:

    SetSuite:
    - An empty Set should have size 0 (pending)
    - Invoking head on an empty Set should produce NoSuchElementException
    

    One difference between an ignored test and a pending one is that an ignored test is intended to be used during a significant refactorings of the code under test, when tests break and you don't want to spend the time to fix all of them immediately. You can mark some of those broken tests as ignored temporarily, so that you can focus the red bar on just failing tests you actually want to fix immediately. Later you can go back and fix the ignored tests. In other words, by ignoring some failing tests temporarily, you can more easily notice failed tests that you actually want to fix. By contrast, a pending test is intended to be used before a test and/or the code under test is written. Pending indicates you've decided to write a test for a bit of behavior, but either you haven't written the test yet, or have only written part of it, or perhaps you've written the test but don't want to implement the behavior it tests until after you've implemented a different bit of behavior you realized you need first. Thus ignored tests are designed to facilitate refactoring of existing code whereas pending tests are designed to facilitate the creation of new code.

    One other difference between ignored and pending tests is that ignored tests are implemented as a test tag that is excluded by default. Thus an ignored test is never executed. By contrast, a pending test is implemented as a test that throws TestPendingException (which is what calling the pending method does). Thus the body of pending tests are executed up until they throw TestPendingException. The reason for this difference is that it enables your unfinished test to send InfoProvided messages to the reporter before it completes abruptly with TestPendingException, as shown in the previous example on Informers that used the GivenWhenThen trait.

    Tagging tests

    A AnyPropSpec's tests may be classified into groups by tagging them with string names. As with any suite, when executing a AnyPropSpec, groups of tests can optionally be included and/or excluded. To tag a AnyPropSpec's tests, you pass objects that extend class org.scalatest.Tag to methods that register tests. Class Tag takes one parameter, a string name. If you have created tag annotation interfaces as described in the Tag documentation, then you will probably want to use tag names on your test functions that match. To do so, simply pass the fully qualified names of the tag interfaces to the Tag constructor. For example, if you've defined a tag annotation interface with fully qualified names, com.mycompany.tags.DbTest, then you could create a matching tag for AnyPropSpecs like this:

    package org.scalatest.examples.propspec.tagging
    
    import org.scalatest.Tag
    
    object DbTest extends Tag("com.mycompany.tags.DbTest")
    

    Given these definitions, you could place AnyPropSpec tests into groups with tags like this:

    import org.scalatest._
    import prop._
    import tagobjects.Slow
    import scala.collection.immutable._
    
    class SetSpec extends propspec.AnyPropSpec with TableDrivenPropertyChecks with Matchers {
    
      val examples =
        Table(
          "set",
          BitSet.empty,
          HashSet.empty[Int],
          TreeSet.empty[Int]
        )
    
      property("an empty Set should have size 0", Slow) {
        forAll(examples) { set =>
          set.size should be (0)
        }
      }
    
      property("invoking head on an empty set should produce NoSuchElementException",
          Slow, DbTest) {
    
        forAll(examples) { set =>
          a [NoSuchElementException] should be thrownBy { set.head }
        }
      }
    }
    

    This code marks both tests with the org.scalatest.tags.Slow tag, and the second test with the com.mycompany.tags.DbTest tag.

    The run method takes a Filter, whose constructor takes an optional Set[String] called tagsToInclude and a Set[String] called tagsToExclude. If tagsToInclude is None, all tests will be run except those those belonging to tags listed in the tagsToExclude Set. If tagsToInclude is defined, only tests belonging to tags mentioned in the tagsToInclude set, and not mentioned in tagsToExclude, will be run.

    Shared fixtures

    A test fixture is composed of the objects and other artifacts (files, sockets, database connections, etc.) tests use to do their work. When multiple tests need to work with the same fixtures, it is important to try and avoid duplicating the fixture code across those tests. The more code duplication you have in your tests, the greater drag the tests will have on refactoring the actual production code.

    ScalaTest recommends three techniques to eliminate such code duplication:

    • Refactor using Scala
    • Override withFixture
    • Mix in a before-and-after trait

    Each technique is geared towards helping you reduce code duplication without introducing instance vars, shared mutable objects, or other dependencies between tests. Eliminating shared mutable state across tests will make your test code easier to reason about and more amenable for parallel test execution.

    The techniques in AnyPropSpec are identical to those in FunSuite, but with “test” replaced by “property”. The following table summarizes the options with a link to the relevant documentation for trait FunSuite:

    Refactor using Scala when different tests need different fixtures.
    get-fixture methods The extract method refactor helps you create a fresh instances of mutable fixture objects in each test that needs them, but doesn't help you clean them up when you're done.
    fixture-context objects By placing fixture methods and fields into traits, you can easily give each test just the newly created fixtures it needs by mixing together traits. Use this technique when you need different combinations of mutable fixture objects in different tests, and don't need to clean up after.
    loan-fixture methods Factor out dupicate code with the loan pattern when different tests need different fixtures that must be cleaned up afterwards.
    Override withFixture when most or all tests need the same fixture.
    withFixture(NoArgTest) The recommended default approach when most or all tests need the same fixture treatment. This general technique allows you, for example, to perform side effects at the beginning and end of all or most tests, transform the outcome of tests, retry tests, make decisions based on test names, tags, or other test data. Use this technique unless:
    Different tests need different fixtures (refactor using Scala instead)
    An exception in fixture code should abort the suite, not fail the test (use a before-and-after trait instead)
    You have objects to pass into tests (override withFixture(OneArgTest) instead)
    withFixture(OneArgTest) Use when you want to pass the same fixture object or objects as a parameter into all or most tests.
    Mix in a before-and-after trait when you want an aborted suite, not a failed test, if the fixture code fails.
    BeforeAndAfter Use this boilerplate-buster when you need to perform the same side-effects before and/or after tests, rather than at the beginning or end of tests.
    BeforeAndAfterEach Use when you want to stack traits that perform the same side-effects before and/or after tests, rather than at the beginning or end of tests.

    Using AnyPropSpec to implement a test matrix

    Using fixture-context objects in a AnyPropSpec is a good way to implement a test matrix. What is the matrix? A test matrix is a series of tests that you need to run on a series of subjects. For example, The Scala API contains many implementations of trait Set. Every implementation must obey the contract of Set. One property of any Set is that an empty Set should have size 0, another is that invoking head on an empty Set should give you a NoSuchElementException, and so on. Already you have a matrix, where rows are the properties and the columns are the set implementations:

     BitSetHashSetTreeSet
    An empty Set should have size 0passpasspass
    Invoking head on an empty set should produce NoSuchElementExceptionpasspasspass

    One way to implement this test matrix is to define a trait to represent the columns (in this case, BitSet, HashSet, and TreeSet) as elements in a single-dimensional Table. Each element in the Table represents one Set implementation. Because different properties may require different fixture instances for those implementations, you can define a trait to hold the examples, like this:

    trait SetExamples extends Tables {
    
      def examples = Table("set", bitSet, hashSet, treeSet)
    
      def bitSet: BitSet
      def hashSet: HashSet[Int]
      def treeSet: TreeSet[Int]
    }
    

    Given this trait, you could provide empty sets in one implementation of SetExamples, and non-empty sets in another. Here's how you might provide empty set examples:

    class EmptySetExamples extends SetExamples {
      def bitSet = BitSet.empty
      def hashSet = HashSet.empty[Int]
      def treeSet = TreeSet.empty[Int]
    }
    

    And here's how you might provide set examples with one item each:

    class SetWithOneItemExamples extends SetExamples {
      def bitSet = BitSet(1)
      def hashSet = HashSet(1)
      def treeSet = TreeSet(1)
    }
    

    Armed with these example classes, you can define checks of properties that require empty or non-empty set fixtures by using instances of these classes as fixture-context objects. In other words, the columns of the test matrix are implemented as elements of a one-dimensional table of fixtures, the rows are implemented as property clauses of a AnyPropSpec.

    Here's a complete example that checks the two properties mentioned previously:

    package org.scalatest.examples.propspec.matrix
    
    import org.scalatest._
    import org.scalatest.prop._
    import scala.collection.immutable._
    
    trait SetExamples extends Tables {
    
      def examples = Table("set", bitSet, hashSet, treeSet)
    
      def bitSet: BitSet
      def hashSet: HashSet[Int]
      def treeSet: TreeSet[Int]
    }
    
    class EmptySetExamples extends SetExamples {
      def bitSet = BitSet.empty
      def hashSet = HashSet.empty[Int]
      def treeSet = TreeSet.empty[Int]
    }
    
    class SetSpec extends propspec.AnyPropSpec with TableDrivenPropertyChecks with Matchers {
    
      property("an empty Set should have size 0") {
        new EmptySetExamples {
          forAll(examples) { set =>
            set.size should be (0)
          }
        }
      }
    
      property("invoking head on an empty set should produce NoSuchElementException") {
        new EmptySetExamples {
          forAll(examples) { set =>
            a [NoSuchElementException] should be thrownBy { set.head }
          }
        }
      }
    }
    

    One benefit of this approach is that the compiler will help you when you need to add either a new row or column to the matrix. In either case, you'll need to ensure all cells are checked to get your code to compile.

    Shared tests

    Sometimes you may want to run the same test code on different fixture objects. That is to say, you may want to write tests that are "shared" by different fixture objects. You accomplish this in a AnyPropSpec in the same way you would do it in a FunSuite, except instead of test you say property, and instead of testsFor you say propertiesFor. For more information, see the Shared tests section of FunSuite's documentation.

  2. trait AnyPropSpecLike extends TestSuite with TestRegistration with Informing with Notifying with Alerting with Documenting

    Implementation trait for class AnyPropSpec, which represents a suite of property-based tests.

    Implementation trait for class AnyPropSpec, which represents a suite of property-based tests.

    AnyPropSpec is a class, not a trait, to minimize compile time given there is a slight compiler overhead to mixing in traits compared to extending classes. If you need to mix the behavior of AnyPropSpec into some other class, you can use this trait instead, because class AnyPropSpec does nothing more than extend this trait and add a nice toString implementation.

    See the documentation of the class for a detailed overview of AnyPropSpec.

  3. abstract class FixtureAnyPropSpec extends FixtureAnyPropSpecLike

    A sister class to org.scalatest.propspec.AnyPropSpec that can pass a fixture object into its tests.

    A sister class to org.scalatest.propspec.AnyPropSpec that can pass a fixture object into its tests.

    Recommended Usage: Use class FixtureAnyPropSpec in situations for which AnyPropSpec would be a good choice, when all or most tests need the same fixture objects that must be cleaned up afterwards. Note: FixtureAnyPropSpec is intended for use in special situations, with class AnyPropSpec used for general needs. For more insight into where FixtureAnyPropSpec fits in the big picture, see the withFixture(OneArgTest) subsection of the Shared fixtures section in the documentation for class AnyPropSpec.

    Class FixtureAnyPropSpec behaves similarly to class org.scalatest.propspec.AnyPropSpec, except that tests may have a fixture parameter. The type of the fixture parameter is defined by the abstract FixtureParam type, which is a member of this class. This class also has an abstract withFixture method. This withFixture method takes a OneArgTest, which is a nested trait defined as a member of this class. OneArgTest has an apply method that takes a FixtureParam. This apply method is responsible for running a test. This class's runTest method delegates the actual running of each test to withFixture, passing in the test code to run via the OneArgTest argument. The withFixture method (abstract in this class) is responsible for creating the fixture argument and passing it to the test function.

    Subclasses of this class must, therefore, do three things differently from a plain old org.scalatest.propspec.AnyPropSpec:

    • define the type of the fixture parameter by specifying type FixtureParam
    • define the withFixture(OneArgTest) method
    • write tests that take a fixture parameter
    • (You can also define tests that don't take a fixture parameter.)

    Here's an example:

    package org.scalatest.examples.fixture.propspec
    
    import org.scalatest._
    import prop.PropertyChecks
    import java.io._
    
    class ExampleSpec extends propspec.FixtureAnyPropSpec with PropertyChecks with Matchers {
    
      // 1. define type FixtureParam
      type FixtureParam = FileReader
    
      // 2. define the withFixture method
      def withFixture(test: OneArgTest) = {
    
        val FileName = "TempFile.txt"
    
        // Set up the temp file needed by the test
        val writer = new FileWriter(FileName)
        try {
          writer.write("Hello, test!")
        }
        finally {
          writer.close()
        }
    
        // Create the reader needed by the test
        val reader = new FileReader(FileName)
    
        try {
          // Run the test using the temp file
          test(reader)
        }
        finally {
          // Close and delete the temp file
          reader.close()
          val file = new File(FileName)
          file.delete()
        }
      }
    
      // 3. write property-based tests that take a fixture parameter
      // (Hopefully less contrived than the examples shown here.)
      property("can read from a temp file") { reader =>
        var builder = new StringBuilder
        var c = reader.read()
        while (c != -1) {
          builder.append(c.toChar)
          c = reader.read()
        }
        val fileContents = builder.toString
        forAll { (c: Char) =>
          whenever (c != 'H') {
            fileContents should not startWith c.toString
          }
        }
      }
    
      property("can read the first char of the temp file") { reader =>
        val firstChar = reader.read()
        forAll { (c: Char) =>
          whenever (c != 'H') {
            c should not equal firstChar
          }
        }
      }
    
      // (You can also write tests that don't take a fixture parameter.)
      property("can write tests that don't take the fixture") { () =>
        forAll { (i: Int) => i + i should equal (2 * i) }
      }
    }
    

    Note: to run the examples on this page, you'll need to include ScalaCheck on the classpath in addition to ScalaTest.

    In the previous example, withFixture creates and initializes a temp file, then invokes the test function, passing in a FileReader connected to that file. In addition to setting up the fixture before a test, the withFixture method also cleans it up afterwards. If you need to do some clean up that must happen even if a test fails, you should invoke the test function from inside a try block and do the cleanup in a finally clause, as shown in the previous example.

    If a test fails, the OneArgTest function will result in a Failed wrapping the exception describing the failure. The reason you must perform cleanup in a finally clause is that in case an exception propagates back through withFixture, the finally clause will ensure the fixture cleanup happens as that exception propagates back up the call stack to runTest.

    If a test doesn't need the fixture, you can indicate that by providing a no-arg instead of a one-arg function. In other words, instead of starting your function literal with something like “reader =>”, you'd start it with “() =>”, as is done in the third test in the above example. For such tests, runTest will not invoke withFixture(OneArgTest). It will instead directly invoke withFixture(NoArgTest).

    Passing multiple fixture objects

    If the fixture you want to pass into your tests consists of multiple objects, you will need to combine them into one object to use this class. One good approach to passing multiple fixture objects is to encapsulate them in a case class. Here's an example:

    case class FixtureParam(builder: StringBuilder, buffer: ListBuffer[String])
    

    To enable the stacking of traits that define withFixture(NoArgTest), it is a good idea to let withFixture(NoArgTest) invoke the test function instead of invoking the test function directly. To do so, you'll need to convert the OneArgTest to a NoArgTest. You can do that by passing the fixture object to the toNoArgTest method of OneArgTest. In other words, instead of writing “test(theFixture)”, you'd delegate responsibility for invoking the test function to the withFixture(NoArgTest) method of the same instance by writing:

    withFixture(test.toNoArgTest(theFixture))
    

    Here's a complete example:

    package org.scalatest.examples.fixture.propspec.multi
    
    import org.scalatest._
    import prop.PropertyChecks
    import scala.collection.mutable.ListBuffer
    
    class ExampleSpec extends propspec.FixtureAnyPropSpec with PropertyChecks with Matchers {
    
      case class FixtureParam(builder: StringBuilder, buffer: ListBuffer[String])
    
      def withFixture(test: OneArgTest) = {
    
        // Create needed mutable objects
        val stringBuilder = new StringBuilder("ScalaTest is ")
        val listBuffer = new ListBuffer[String]
        val theFixture = FixtureParam(stringBuilder, listBuffer)
    
        // Invoke the test function, passing in the mutable objects
        withFixture(test.toNoArgTest(theFixture))
      }
    
      property("testing should be easy") { f =>
        f.builder.append("easy!")
        assert(f.builder.toString === "ScalaTest is easy!")
        assert(f.buffer.isEmpty)
        val firstChar = f.builder(0)
        forAll { (c: Char) =>
          whenever (c != 'S') {
            c should not equal firstChar
          }
        }
        f.buffer += "sweet"
      }
    
      property("testing should be fun") { f =>
        f.builder.append("fun!")
        assert(f.builder.toString === "ScalaTest is fun!")
        assert(f.buffer.isEmpty)
        val firstChar = f.builder(0)
        forAll { (c: Char) =>
          whenever (c != 'S') {
            c should not equal firstChar
          }
        }
      }
    }
    

  4. trait FixtureAnyPropSpecLike extends fixture.TestSuite with fixture.TestRegistration with Informing with Notifying with Alerting with Documenting

    Implementation trait for class FixtureAnyPropSpec, which is a sister class to org.scalatest.propspec.AnyPropSpec that can pass a fixture object into its tests.

    Implementation trait for class FixtureAnyPropSpec, which is a sister class to org.scalatest.propspec.AnyPropSpec that can pass a fixture object into its tests.

    FixtureAnyPropSpec is a class, not a trait, to minimize compile time given there is a slight compiler overhead to mixing in traits compared to extending classes. If you need to mix the behavior of FixtureAnyPropSpec into some other class, you can use this trait instead, because class FixtureAnyPropSpec does nothing more than extend this trait and add a nice toString implementation.

    See the documentation of the class for a detailed overview of FixtureAnyPropSpec.

Ungrouped