The latest release of specs2 (2.0) deserves a little bit more than just release notes. It needs explanations, apologies and a bit of celebration!
Explanations
- why is there another (actually several!) new style(s) of writing acceptance specifications
- what are
Scripts andScriptTemplates - what has been done for compilation times
- what you can do with
Snippets - what is an
ExampleFactory
Apologies
- the
>>/inproblem - API breaks
TraversablematchersAroundOutsideandFixture- the never-ending quest for
Given/When/Thenspecifications
Celebration
- compiler-checked documentation!
- "operator-less" specifications!
- more consistent
Traversablematchers!
Explanations
Scala 2.10 is a game changer for specs2, thanks to 2 features: String interpolation and Macros.
String interpolation
Specs2 has been designed from the start with the idea that it should be immutable by default. This has led to the definition of Acceptance specifications with lots of operators, or, as someone put it elegantly, "code on the left, brainfuck on the right":
class HelloWorldSpec extends Specification { def is =
"This is a specification to check the 'Hello world' string" ^
p^
"The 'Hello world' string should" ^
"contain 11 characters" ! e1^
"start with 'Hello'" ! e2^
"end with 'world'" ! e3^
end
def e1 = "Hello world" must have size(11)
def e2 = "Hello world" must startWith("Hello")
def e3 = "Hello world" must endWith("world")
}
Fortunately Scala 2.10 now offers a great alternative with String interpolation. In itself, String interpolation is not revolutionary. A string starting with s can have interpolated variables:
val name = "Eric"
s"Hello $name!"
Hello Eric!
But the great powers behind Scala realized that they could both provide standard String interpolation and give you the ability to make your own. Exactly what I needed to make these pesky operators disappear!
class HelloWorldSpec extends Specification { def is = s2"""
This is a specification to check the 'Hello world' string
The 'Hello world' string should
contain 11 characters $e1
start with 'Hello' $e2
end with 'world' $e3
"""
def e1 = "Hello world" must have size(11)
def e2 = "Hello world" must startWith("Hello")
def e3 = "Hello world" must endWith("world")
}
What has changed in the specification above is that text Fragments are now regular strings in the multiline s2 string and the examples are now inserted as interpolated variables. Let's explore in more details some aspects of this new feature:
- layout
- examples descriptions
- other fragments
- implicit conversions
- auto-examples
Layout
If you run the HelloWorldSpec you will see that the indentation of each example is respected in the output:
This is a specification to check the 'Hello world' string
The 'Hello world' string should
+ contain 11 characters
+ start with 'Hello'
+ end with 'world'
This means that you don't have to worry anymore about the layout of text and use the p, t, bt, end, endp formatting fragments as before.
Examples descriptions
On the other hand, the string which is taken as the example description is not as well delimited anymore, so it is now choosen by convention to be everything that is on the same line. For example this is what you get with the new interpolated string:
s2"""
My software should
do something that it pretty long to explain,
so long that it needs 2 lines" ${ 1 must_== 1 }
"""
My software should
do something that it pretty long to explain,
+ so long that it needs 2 lines"
If you want the 2 lines to be included in the example description you will need to use the "old" form of creating an example:
s2"""
My software should
${ """do something that it pretty long to explain,
so long that it needs 2 lines""" ! { 1 must_== 1 } }
"""
My software should+ do something that it pretty long to explain,
so long that it needs 2 lines
But I suspect that there will be very few times when you will want to do that.
Other fragments and variables
Inside the s2 string you can interpolate all the usual specs2 fragments: Steps, Actions, included specifications, Forms... However you will quickly realize that you can not interpolate arbitrary objects. Indeed, excepted specs2 objects, the only other 2 types which you can use as variables are Snippets (see below) and Strings.
The restriction is there to remind you that, in general, interpolated expressions are "unsafe". If the expression you're interpolating is throwing an Exception, as it is commonly the case with tested code, there is no way to catch that exception. If that exception is uncaught, the whole specification will fail to be built. Why is that?
Implicit conversions
When I first started to experiment with interpolated strings I thought that they could even be used to write Unit Specifications:
s2"""
This is an example of conversion using integers ${
val (a, b) = ("1".toInt, "2".toInt)
(a + b) must_== 3
}
"""
Unfortunately such specifications will horribly break if there is an error in one of the examples. For instance if the example was:
This is an example of conversion using integers ${
// oops, this is going to throw a NumberFormatException!
val (a, b) = ("!".toInt, "2".toInt)
(a + b) must_== 3
}
Then the whole string and the whole specification will fail to be instantiated!
The reason is that everything you interpolate is converted, through an implicit conversion, to a "SpecPart" which will be interpreted differently depending on its type. If it is a Result then we will interpret this as the body of an Example and use the preceding text as the description. If it is just a simple string then it is just inserted in the specification as a piece of text. But implicit conversions of a block of code, as above, are not converting the whole block. They are merely converting the last value! So if anything before the last value throws an Exception you will have absolutely no way to catch it and it will bubble up to the top.
That means that you need to be very prudent when interpolating arbitrary blocks. One work-around is to do something like that
import execute.{AsResult => >>}
s2"""
This is an example of conversion using integers ${>>{
val (a, b) = ("!".toInt, "2".toInt)
(a + b) must_== 3
}}
"""
But you have to admit that the whole ${>>{...}} is not exactly gorgeous.
Auto-examples
One clear win of Scala 2.10 for specs2 is the use of macros to capture code expressions. This particularly interesting with so-called "auto-examples". This feature is really useful when your examples are so self-descriptive that a textual description feels redundant. For example if you want to specify the String.capitalize method:
s2"""
The `capitalize` method verifies
${ "hello".capitalize === "Hello" }
${ "Hello".capitalize === "Hello" }
${ "hello world".capitalize === "Hello world" }
"""
The `capitalize` method verifies
+ "hello".capitalize === "Hello"
+ "Hello".capitalize === "Hello"
+ "hello world".capitalize === "Hello world"
It turns out that the method interpolating the s2 extractor is using a macro to extract the text for each interpolated expression and so, if on a given line there is no preceding text, we take the captured expression as the example description. It is important to note that this will only properly work if you enable the -Yrangepos scalac option (in sbt: scalacOptions in Test := Seq("-Yrangepos")).
However the drawback of using that option is the compilation speed cost which you can incur (around 10% in my own measurements). If you don't want (or you forget :-)) to use that option there is a default implementation which should do the trick in most cases but which might not capture all the text in some edge cases.
Scripts
The work on Given/When/Then specifications has led to a nice generalisation. Since the new GWT trait decouples the specification text from the steps and examples to create, we can push this idea a bit further and create "classical" specifications where the text is not annotated at all and examples are described somewhere else.
Let's see what we can do with the org.specs2.specification.script.Specification class:
import org.specs2._
import specification._
class StringSpecification extends script.Specification with Grouped { def is = s2"""
Addition
========
It is possible to add strings with the + operator
+ one string and an empty string
+ 2 non-empty strings
Multiplication
==============
It is also possible to duplicate a string with the * operator
+ using a positive integer duplicates the string
+ using a negative integer returns an empty string
"""
"addition" - new group {
eg := ("hello" + "") === "hello"
eg := ("hello" + " world") === "hello world"
}
"multiplication" - new group {
eg := ("hello" * 2) === "hellohello"
eg := ("hello" * -1) must beEmpty
}
}
With script.Specifications you just provide a piece of text where examples are starting with a + sign and you specify examples groups. Example groups were introduced in a previous version of specs2 with the idea of providing standard names for examples in Acceptance specifications.
When the specification is executed, the first 2 example lines are mapped to the examples of the first group, and the examples lines from the next block (as delimited with a Markdown title) are used to build examples by taking expectations in the second group (those group are automatically given names, g1 and g2, but you can specify them yourself: "addition" - new g1 {...).
This seems to be a lot of "convention over configuration" but this is actually all configurable! The script.Specification class is an example of a Script and it is associated with a ScriptTemplate which defines how to parse text to create fragments based on the information contained in the Script (we will see another example of this in action below with the GWT trait which proposes another type of Script named Scenario to define Given/When/Then steps).
There are lots of advantages in adopting this new script.Specification class:
it is "operator-free", there's no need to annotate your specification on the right with strange symbols
tags are automatically inserted for you so that it's easy to re-run a specific example or group of examples by name:
test-only StringSpecification -- include g2.e1examples are marked as pending if you haven't yet implemented them
it is configurable to accomodate for other templates (you could even create Cucumber-like specifications if that's your thing!)
The obvious drawback is the decoupling between the text and the examples code. If you restructure the text you will have to restructure the examples accordingly and knowing which example is described by which piece of text is not obvious. This, or operators on the right-hand side, choose your poison :-)
Compilation times
Scala's typechecking and JVM interoperability comes with a big price in terms of compilation times. Moderately-sized projects can take minutes to compile which is very annoying for someone coming from Java or Haskell.
Bill Venners has tried to do a systematic study of which features in testing libraries seems to have the biggest impact. It turns out that implicits, traits and byname parameters have a significant impact on compilation times. Since specs2 is using those features more than any other test library, I tried to do something about it.
The easiest thing to do was to make Specification an abstract class, not a trait (and provide the SpecificationLike trait in its place). My unscientific estimation is that this single change removed 0.5 seconds per compiled file (from 313s to 237s for the specs2 build, and a memory reduction of 55Mb, from 225Mb to 170Mb).
Then, the next very significant improvement was to use interpolated specifications instead of the previous style of Acceptance specifications. The result is impressive: from 237 seconds to 150 seconds and a memory reduction of more than 120Mb, from 170Mb to 47Mb!
On the other hand, when I tried to remove some of the byname parameters (the left part of a must_== b) I didn't observe a real impact on compilation times (only 15% less memory).
The last thing I did was to remove some of the default matchers (and to add a few others). Those matchers are the "content" matchers: XmlMatchers, JsonMatchers, FileMatchers, ContentMatchers (and I added instead the TryMatchers). I did this to remove some implicits from the scope when compiling code but also to reduce the namespace footprint everytime you extend the Specification class. However I couldn't see a major improvement to compile-time performances with this change.
Snippets
One frustration of software documentation writers is that it is very common to have stale or incorrect code because the API has moved on. What if it was possible to write some code, in the documentation, that will be checked by the compiler? And automatically refactored when you change a method name?
This is exactly what Snippets will do for you. When you want to capture and display a piece of code in a Specification you create a Snippet:
s2"""
This is an example of addition: ${snippet{
// who knew?
1 + 1 == 2
}}
"""
This renders as:
This is an example of addition
// who knew?
1 + 1 == 2
And yes, you guessed it right, the Snippet above was extracted by using another Snippet! I encourage you to read the documentation on Snippets to see what you can do with them, the main features are:
code evaluation: the last value can be displayed as a result
checks: the last value can be checked and reported as a failure in the Specification
code hiding: it is possible to hide parts of the code (initialisations, results) by enclosing them in "scissors" comments of the form
// 8<--
Example factory
Every now and then I get a question from users who want to intercept the creation of examples and use the example description to do interesting things before or after the example execution. It is now possible to do so by providing another ExampleFactory rather than the default one:
import specification._
class PrintBeforeAfterSpec extends Specification { def is =
"test" ! ok
case class BeforeAfterExample(e: Example) extends BeforeAfter {
def before = println("before "+e.desc)
def after = println("after "+e.desc)
}
override def exampleFactory = new ExampleFactory {
def newExample(e: Example) = {
val context = BeforeAfterExample(e)
e.copy(body = () => context(e.body()))
}
}
}
The PrintBeforeAfterSpec will print the name of each example before and after executing it.
Apologies
the >> / in problem
This issue has come up at different times and one lesson is: Unit means "anything" so don't try to be too smart about it. So I owe an apology to the users for this poor API design choice and for the breaking API change that is now ensuing. Please read the thread in the Github issue to learn how to fix compile errors that would result from this change.
API breaks
While we're on the subject of API breaks, let's make a list:
Unit values in
>>/in: now you need to explicitly declare if you mean "a list of examples created with foreach" or "a list of expectations created with foreach"Specificationis not a trait anymore so you should use theSpecificationLiketrait instead if that's what you need (see the Compilation times section)Some matchers traits have been removed from the default matchers (XML, JSON, File, Content) so you need to explicitly mix them in (see the Compilation times section)
The
Given/When/Thenfunctionality has been extracted as a deprecated traitspecification.GivenWhenThen(see the Given/When/Then? section)the negation of the
Mapmatchers has changed (this can be considered as a fix but this might be a run-time break for some of you)many of the
Traversablematchers have been deprecated (see the next section)
Traversable matchers
I've had this nagging thought in my mind for some time now but it only reached my conscience recently. I always felt that specs2 matchers for collections were a bit ad-hoc, with not-so-obvious ways to do simple things. After lots of fighting with implicit classes, overloading and subclassing, I think that I have something better to propose.
With the new API we generalize the type of checks you can perform on elements:
Seq(1, 2, 3) must contain(2)just checks for the presence of one element in the sequencethis is equivalent to writing
Seq(1, 2, 3) must contain(equalTo(2))which means that you can pass a matcher to thecontainmethod. For examplecontainAnyOf(1, 2, 3)iscontain(anyOf(1, 2, 3))whereanyOfis just another matcherand more generally, you can pass any function returning a result!
Seq(1, 2, 3) must contain((i: Int) => i must beEqualTo(2))orSeq(1, 2, 3) must contain((i: Int) => i == 2)(you can even return a ScalaCheckPropif you want)
Then we can use combinators to specify how many times we want the check to be performed:
Seq(1, 2, 3) must contain(2)is equivalent toSeq(1, 2, 3) must contain(2).atLeastOnceSeq(1, 2, 3) must contain(2).atMostOnceSeq(1, 2, 3) must contain(be_>=(2)).atLeast(2.times)Seq(1, 2, 3) must contain(be_>=(2)).between(1.times, 2.times)
This covers lots of cases where you would previously use must have oneElementLike(partialFunction) or must containMatch(...). This also can be used instead of the forall, atLeastOnce methods. For example forall(Seq(1, 2, 3)) { (i: Int) => i must be_>=(0) } is Seq(1, 2, 3) must contain((i: Int) => i must be_>=(0)).forall.
The other type of matching which you want to perform on collections is with several checks at the time. For example:
Seq(1, 2, 3) must contain(allOf(2, 3))
This seems similar to the previous case but the combinators you might want to use with several checks are different. exactly is one of them:
Seq(1, 2, 3) must contain(exactly(3, 1, 2))// we don't expect ordered elements by default
Or inOrder
Seq(1, 2, 3) must contain(exactly(be_>(0), be_>(1), be_>(2)).inOrder)// with matchers here
One important thing to note though is that, when you are not using inOrder, the comparison is done greedily, we don't try all the possible combinations of input elements and checks to see if there would be a possibility for the whole expression to match.
Please explore this new API and report any issue (bug, compilation error) you will find. Most certainly the failure reporting can be improved. The description of failures is much more centralized with this new implementation but also a bit more generic. For now, the failure messages are just listing which elements were not passing the checks but they do not output something nice like: The sequence 'Seq(1, 2, 3) does not contain exactly the elements 4 and 3 in order: 4 is not found'.
AroundOutside vs Fixture
My approach to context management in specs2 has been very progressive. First I provided the ability to insert code (and more precisely effects) before or after an Example, reproducing here standard JUnit capabilities. Then I've introduced Around to place things "in" a context, and Outside to pass data to an example. And finally AroundOutside as the ultimate combination of both capabilities.
I thought that with AroundOutside you could do whatever you needed to do, end of story. It turns out that it's not so simple. AroundOutside is not general enough because the generation of Outside data cannot be controled by the Around context. This proved to be very problematic for me on a specific use case where I needed to re-run the same example, based on different parameters, with slightly different input data each time. AroundOutside was just not doing it. The solution? A good old Fixture. Very simple, a Fixture[T], is a trait like that:
trait Fixture[T] {
def apply[R : AsResult](f: T => R): Result
}
You can define an implicit fixture for all the examples:
class s extends Specification { def is = s2"""
first example using the magic number $e1
second example using the magic number $e1
"""
implicit def magicNumber = new specification.Fixture[Int] {
def apply[R : AsResult](f: Int => R) = AsResult(f(10))
}
def e1 = (i: Int) => i must be_>(0)
def e2 = (i: Int) => i must be_<(100)
}
I'm not particularly happy to add this to the API because it adds to the overall API footprint and learning curve, but in some scenarios this is just indispensable.
Given/When/Then?
With the new "interpolated" style I had to find another way to write Given/When/Then (GWT) steps. But this is tricky. The trouble with GWT steps is that they are intrisically dependent. You cannot have a Then step being defined before a When step for example.
The "classic" style of acceptance specification is enforcing this at compile time because, in that style, you explicitly chain calls and the types have to "align":
class GivenWhenThenSpec extends Specification with GivenWhenThen { def is =
"A given-when-then example for a calculator" ^ br^
"Given the following number: ${1}" ^ aNumber^
"And a second number: ${2}" ^ aNumber^
"And a third number: ${6}" ^ aNumber^
"When I use this operator: ${+}" ^ operator^
"Then I should get: ${9}" ^ result^
end
val aNumber: Given[Int] = (_:String).toInt
val operator: When[Seq[Int], Operation] = (numbers: Seq[Int]) => (s: String) => Operation(numbers, s)
val result: Then[Operation] = (operation: Operation) => (s: String) => { operation.calculate must_== s.toInt }
case class Operation(numbers: Seq[Int], operator: String) {
def calculate: Int = if (operator == "+") numbers.sum else numbers.product
}
}
We can probably do better than this. What is required?
- to extract strings from text and transform them to well-typed values
- to define functions using those values so that types are respected
- to restrict the composition of functions so that a proper order of
Given/When/Thenis respected - to transform all of this into
StepsandExamples
So, with apologies for coming up with yet-another-way of doing the same thing, let me introduce you to the GWT trait:
import org.specs2._
import specification.script.{GWT, StandardRegexStepParsers}
class GWTSpec extends Specification with GWT with StandardRegexStepParsers { def is = s2"""
A given-when-then example for a calculator ${calculator.start}
Given the following number: 1
And a second number: 2
And a third number: 6
When I use this operator: +
Then I should get: 9
And it should be >: 0 ${calculator.end}
"""
val anOperator = readAs(".*: (.)$").and((s: String) => s)
val calculator =
Scenario("calculator").
given(anInt).
given(anInt).
given(anInt).
when(anOperator) { case op :: i :: j :: k :: _ => if (op == "+") (i+j+k) else (i*j*k) }.
andThen(anInt) { case expected :: sum :: _ => sum === expected }.
andThen(anInt) { case expected :: sum :: _ => sum must be_>(expected) }
}
In the specification above, calculator is a Scenario object which declares some steps through the given/when/andThen methods. The Scenario class provides a fluent interface in order to restrict the order of calls. For example, if you try to call a given step after a when step you will get a compilation error. Furthermore steps which are using extracted values from previous steps must use the proper types, what you pass to the when step has to be a partial function taking in a Shapeless HList of the right type.
You will also notice that the calculator is using anInt, anOperator. Those are StepParsers, which are simple objects extracting values from a line of text and returning Either[Exception, T] depending on the correct conversion of text to a type T. By default you have access to 2 types of parsers. The first one is DelimitedStepParser which expects that values to extract are enclosed in {} delimiters (this is configurable). The other one is RegexStepParser which uses a regular expression with groups in order to know what to extract. For example anOperator defines that the operator to extract will be just after the column at the end of the line.
Finally the calculator scenario is inserted into the s2 interpolated string to delimitate the text it applies to. Scenario being a specific kind of Script it has an associated ScriptTemplate which defines that the last lines of the text should be paired with the corresponding given/when/then method declarations. This is configurable and we can imagine other ways of pairing text to steps (see the org.specs2.specification.script.GWT.BulletTemplate class for example).
For reasons which are too long to expose here I've never been a big fan of Given/When/Then specifications and I guess that the multitude of ways to do that in specs2 shows it. I hope however that the GWT fans will find this approach satisfying and customisable to their taste.
Celebration!
I think there are some really exciting things in this upcoming specs2 release for "Executable Software Specifications" lovers.
Compiler-checked documentation
Having compiler-checked snippets is incredibly useful. I've fixed quite a few bugs in boths specs2 and Scoobi user guides and I hope that I made them more resistant to future changes that will happen through refactoring (when just renaming things for example). I'm also very happy that, thanks to macros, the ability to capture code was extended to "auto-examples". In previous specs2 versions, this is implemented by looking at stack traces and doing horrendous calculations on where a piece of code would be. This gives me the shivers everytime I have to look at that code!
No operators
The second thing is Scripts and ScriptTemplates. There is a trade-off when writing specifications. On one hand we would like to read pure text, without the encumbrance of implementation code, on the other hand, when we read specification code, it's nice to have a short sentence explaining what it does. With this new release there is a continuum of solutions on this trade-off axis:
- you can have pure text, with no annotations but no navigation is possible to the code (with
org.specs2.specification.script.Specification) - you can have annotated text, with some annotations to access the code (with
org.specs2.Specification) - you can have text interspersed with the code (with
org.specs2.mutable.Specification)
New matchers
I'm pretty happy to have new Traversable matchers covering a lot more use cases than before in a straight-forward manner. I hope this will reduce the thinking time between "I need to check that" and "Ok, this is how I do it".
Please try out the new Release Candidate, report bugs, propose enhancements and have fun!
No comments:
Post a Comment