s2"""
The `capitalize` method verifies
${ "hello".capitalize === "Hello" }
${ "Hello".capitalize === "Hello" }
${ "hello world".capitalize === "Hello world" }
"""
The `capitalize` method verifies
+ "hello".capitalize === "Hello"
+ "Hello".capitalize === "Hello"
+ "hello world".capitalize === "Hello world"
It turns out that the method interpolating the s2 extractor is using a macro to extract the text for each interpolated expression and so, if on a given line there is no preceding text, we take the captured expression as the example description. It is important to note that this will only properly work if you enable the -Yrangepos scalac option (in sbt: scalacOptions in Test := Seq("-Yrangepos")).
However the drawback of using that option is the compilation speed cost which you can incur (around 10% in my own measurements). If you don't want (or you forget :-)) to use that option there is a default implementation which should do the trick in most cases but which might not capture all the text in some edge cases.
Scripts
The work on Given/When/Then specifications has led to a nice generalisation. Since the new GWT trait decouples the specification text from the steps and examples to create, we can push this idea a bit further and create "classical" specifications where the text is not annotated at all and examples are described somewhere else.
Let's see what we can do with the org.specs2.specification.script.Specification class:
import org.specs2._
import specification._
class StringSpecification extends script.Specification with Grouped { def is = s2"""
Addition
========
It is possible to add strings with the + operator
+ one string and an empty string
+ 2 non-empty strings
Multiplication
==============
It is also possible to duplicate a string with the * operator
+ using a positive integer duplicates the string
+ using a negative integer returns an empty string
"""
"addition" - new group {
eg := ("hello" + "") === "hello"
eg := ("hello" + " world") === "hello world"
}
"multiplication" - new group {
eg := ("hello" * 2) === "hellohello"
eg := ("hello" * -1) must beEmpty
}
}
With script.Specifications you just provide a piece of text where examples are starting with a + sign and you specify examples groups. Example groups were introduced in a previous version of specs2 with the idea of providing standard names for examples in Acceptance specifications.
When the specification is executed, the first 2 example lines are mapped to the examples of the first group, and the examples lines from the next block (as delimited with a Markdown title) are used to build examples by taking expectations in the second group (those group are automatically given names, g1 and g2, but you can specify them yourself: "addition" - new g1 {...).
This seems to be a lot of "convention over configuration" but this is actually all configurable! The script.Specification class is an example of a Script and it is associated with a ScriptTemplate which defines how to parse text to create fragments based on the information contained in the Script (we will see another example of this in action below with the GWT trait which proposes another type of Script named Scenario to define Given/When/Then steps).
There are lots of advantages in adopting this new script.Specification class:
it is "operator-free", there's no need to annotate your specification on the right with strange symbols
tags are automatically inserted for you so that it's easy to re-run a specific example or group of examples by name: test-only StringSpecification -- include g2.e1
examples are marked as pending if you haven't yet implemented them
it is configurable to accomodate for other templates (you could even create Cucumber-like specifications if that's your thing!)
The obvious drawback is the decoupling between the text and the examples code. If you restructure the text you will have to restructure the examples accordingly and knowing which example is described by which piece of text is not obvious. This, or operators on the right-hand side, choose your poison :-)
Compilation times
Scala's typechecking and JVM interoperability comes with a big price in terms of compilation times. Moderately-sized projects can take minutes to compile which is very annoying for someone coming from Java or Haskell.
Bill Venners has tried to do a systematic study of which features in testing libraries seems to have the biggest impact. It turns out that implicits, traits and byname parameters have a significant impact on compilation times. Since specs2 is using those features more than any other test library, I tried to do something about it.
The easiest thing to do was to make Specification an abstract class, not a trait (and provide the SpecificationLike trait in its place). My unscientific estimation is that this single change removed 0.5 seconds per compiled file (from 313s to 237s for the specs2 build, and a memory reduction of 55Mb, from 225Mb to 170Mb).
Then, the next very significant improvement was to use interpolated specifications instead of the previous style of Acceptance specifications. The result is impressive: from 237 seconds to 150 seconds and a memory reduction of more than 120Mb, from 170Mb to 47Mb!
On the other hand, when I tried to remove some of the byname parameters (the left part of a must_== b) I didn't observe a real impact on compilation times (only 15% less memory).
The last thing I did was to remove some of the default matchers (and to add a few others). Those matchers are the "content" matchers: XmlMatchers, JsonMatchers, FileMatchers, ContentMatchers (and I added instead the TryMatchers). I did this to remove some implicits from the scope when compiling code but also to reduce the namespace footprint everytime you extend the Specification class. However I couldn't see a major improvement to compile-time performances with this change.
Snippets
One frustration of software documentation writers is that it is very common to have stale or incorrect code because the API has moved on. What if it was possible to write some code, in the documentation, that will be checked by the compiler? And automatically refactored when you change a method name?
This is exactly what Snippets will do for you. When you want to capture and display a piece of code in a Specification you create a Snippet:
s2"""
This is an example of addition: ${snippet{
// who knew?
1 + 1 == 2
}}
"""
This renders as:
This is an example of addition
// who knew?
1 + 1 == 2
And yes, you guessed it right, the Snippet above was extracted by using another Snippet! I encourage you to read the documentation on Snippets to see what you can do with them, the main features are:
code evaluation: the last value can be displayed as a result
checks: the last value can be checked and reported as a failure in the Specification
code hiding: it is possible to hide parts of the code (initialisations, results) by enclosing them in "scissors" comments of the form // 8<--
Example factory
Every now and then I get a question from users who want to intercept the creation of examples and use the example description to do interesting things before or after the example execution. It is now possible to do so by providing another ExampleFactory rather than the default one:
import specification._
class PrintBeforeAfterSpec extends Specification { def is =
"test" ! ok
case class BeforeAfterExample(e: Example) extends BeforeAfter {
def before = println("before "+e.desc)
def after = println("after "+e.desc)
}
override def exampleFactory = new ExampleFactory {
def newExample(e: Example) = {
val context = BeforeAfterExample(e)
e.copy(body = () => context(e.body()))
}
}
}
The PrintBeforeAfterSpec will print the name of each example before and after executing it.
Apologies
the >> / in problem
This issue has come up at different times and one lesson is: Unit means "anything" so don't try to be too smart about it. So I owe an apology to the users for this poor API design choice and for the breaking API change that is now ensuing. Please read the thread in the Github issue to learn how to fix compile errors that would result from this change.
API breaks
While we're on the subject of API breaks, let's make a list:
Unit values in >> / in: now you need to explicitly declare if you mean "a list of examples created with foreach" or "a list of expectations created with foreach"
Specification is not a trait anymore so you should use the SpecificationLike trait instead if that's what you need (see the Compilation times section)
Some matchers traits have been removed from the default matchers (XML, JSON, File, Content) so you need to explicitly mix them in (see the Compilation times section)
The Given/When/Then functionality has been extracted as a deprecated trait specification.GivenWhenThen (see the Given/When/Then? section)
the negation of the Map matchers has changed (this can be considered as a fix but this might be a run-time break for some of you)
many of the Traversable matchers have been deprecated (see the next section)
Traversable matchers
I've had this nagging thought in my mind for some time now but it only reached my conscience recently. I always felt that specs2 matchers for collections were a bit ad-hoc, with not-so-obvious ways to do simple things. After lots of fighting with implicit classes, overloading and subclassing, I think that I have something better to propose.
With the new API we generalize the type of checks you can perform on elements:
Seq(1, 2, 3) must contain(2) just checks for the presence of one element in the sequence
this is equivalent to writing Seq(1, 2, 3) must contain(equalTo(2)) which means that you can pass a matcher to the contain method. For example containAnyOf(1, 2, 3) is contain(anyOf(1, 2, 3)) where anyOf is just another matcher
and more generally, you can pass any function returning a result! Seq(1, 2, 3) must contain((i: Int) => i must beEqualTo(2)) or Seq(1, 2, 3) must contain((i: Int) => i == 2) (you can even return a ScalaCheck Prop if you want)
Then we can use combinators to specify how many times we want the check to be performed:
Seq(1, 2, 3) must contain(2) is equivalent to Seq(1, 2, 3) must contain(2).atLeastOnce
Seq(1, 2, 3) must contain(2).atMostOnce
Seq(1, 2, 3) must contain(be_>=(2)).atLeast(2.times)
Seq(1, 2, 3) must contain(be_>=(2)).between(1.times, 2.times)
This covers lots of cases where you would previously use must have oneElementLike(partialFunction) or must containMatch(...). This also can be used instead of the forall, atLeastOnce methods. For example forall(Seq(1, 2, 3)) { (i: Int) => i must be_>=(0) } is Seq(1, 2, 3) must contain((i: Int) => i must be_>=(0)).forall.
The other type of matching which you want to perform on collections is with several checks at the time. For example:
Seq(1, 2, 3) must contain(allOf(2, 3))
This seems similar to the previous case but the combinators you might want to use with several checks are different. exactly is one of them:
Seq(1, 2, 3) must contain(exactly(3, 1, 2)) // we don't expect ordered elements by default
Or inOrder
Seq(1, 2, 3) must contain(exactly(be_>(0), be_>(1), be_>(2)).inOrder) // with matchers here
One important thing to note though is that, when you are not using inOrder, the comparison is done greedily, we don't try all the possible combinations of input elements and checks to see if there would be a possibility for the whole expression to match.
Please explore this new API and report any issue (bug, compilation error) you will find. Most certainly the failure reporting can be improved. The description of failures is much more centralized with this new implementation but also a bit more generic. For now, the failure messages are just listing which elements were not passing the checks but they do not output something nice like: The sequence 'Seq(1, 2, 3) does not contain exactly the elements 4 and 3 in order: 4 is not found'.
AroundOutside vs Fixture
My approach to context management in specs2 has been very progressive. First I provided the ability to insert code (and more precisely effects) before or after an Example, reproducing here standard JUnit capabilities. Then I've introduced Around to place things "in" a context, and Outside to pass data to an example. And finally AroundOutside as the ultimate combination of both capabilities.
I thought that with AroundOutside you could do whatever you needed to do, end of story. It turns out that it's not so simple. AroundOutside is not general enough because the generation of Outside data cannot be controled by the Around context. This proved to be very problematic for me on a specific use case where I needed to re-run the same example, based on different parameters, with slightly different input data each time. AroundOutside was just not doing it. The solution? A good old Fixture. Very simple, a Fixture[T], is a trait like that:
trait Fixture[T] {
def apply[R : AsResult](f: T => R): Result
}
You can define an implicit fixture for all the examples:
class s extends Specification { def is = s2"""
first example using the magic number $e1
second example using the magic number $e1
"""
implicit def magicNumber = new specification.Fixture[Int] {
def apply[R : AsResult](f: Int => R) = AsResult(f(10))
}
def e1 = (i: Int) => i must be_>(0)
def e2 = (i: Int) => i must be_<(100)
}
I'm not particularly happy to add this to the API because it adds to the overall API footprint and learning curve, but in some scenarios this is just indispensable.
Given/When/Then?
With the new "interpolated" style I had to find another way to write Given/When/Then (GWT) steps. But this is tricky. The trouble with GWT steps is that they are intrisically dependent. You cannot have a Then step being defined before a When step for example.
The "classic" style of acceptance specification is enforcing this at compile time because, in that style, you explicitly chain calls and the types have to "align":
class GivenWhenThenSpec extends Specification with GivenWhenThen { def is =
"A given-when-then example for a calculator" ^ br^
"Given the following number: ${1}" ^ aNumber^
"And a second number: ${2}" ^ aNumber^
"And a third number: ${6}" ^ aNumber^
"When I use this operator: ${+}" ^ operator^
"Then I should get: ${9}" ^ result^
end
val aNumber: Given[Int] = (_:String).toInt
val operator: When[Seq[Int], Operation] = (numbers: Seq[Int]) => (s: String) => Operation(numbers, s)
val result: Then[Operation] = (operation: Operation) => (s: String) => { operation.calculate must_== s.toInt }
case class Operation(numbers: Seq[Int], operator: String) {
def calculate: Int = if (operator == "+") numbers.sum else numbers.product
}
}
We can probably do better than this. What is required?
- to extract strings from text and transform them to well-typed values
- to define functions using those values so that types are respected
- to restrict the composition of functions so that a proper order of
Given/When/Then is respected
- to transform all of this into
Steps and Examples
So, with apologies for coming up with yet-another-way of doing the same thing, let me introduce you to the GWT trait:
