If at first it doesn't succeed...

If at first it doesn’t succeed, try something else

Today we start from a very common programmer need and we end up checking laws! With some considerations about software modularity.

I am currently implementing a command-line application in Haskell. This application needs a OAuthToken to access some webservice. There are 2 ways to get a token:

• with an environment variable: OAUTH_TOKEN
• with a call to another command-line tool, ztoken, which gives you a token based on your credentials

This translates to the following Haskell functions

getTokenFromEnvironment :: ExceptT GetTokenError IO OAuthToken

getTokenFromCommandLine :: ExceptT GetTokenError IO OAuthToken

Here I am using:

• IO as my base monad for interacting with the external world

• ExceptT GetTokenError as a way to declare an error if I cannot get a token because either I don’t have an OAUTH_TOKEN variable in my environment or ztoken does not accept my credentials. I leave other issues, like network access problems to exceptions in the IO monad

data GetTokenError =
  CommandLineError String |
  EnvironmentError String

What I really want now is a way to combine those 2 calls into one

getToken :: ExceptT GetTokenError IO OAuthToken

The first thing which came to my mind was the Alternative type class and in particular the <|> operator:

(<|>) :: f a -> f a -> f a

This typeclass has an instance for ExceptT e provided that there is a Monoid e, meaning that I can accumulate errors. Actually what I just need is the Alt typeclass which just requires a Semigroup e. For this I need to adapt my error type a little bit:

data GetTokenError =
  CommandLineError String |
  EnvironmentError String |
  RepeatedErrors GetTokenError GetTokenError
  deriving (Eq, Show)

The RepeatedErrors case gives us the possibility to accumulate errors in the case of repeated failed calls to retrieve an OAuthToken. You could argue that it is not particularly well modelled because RepeatedErrors stores errors more like a tree rather than a list. But let’s leave it at that for now.

The Semigroup instance for GetTokenError looks like this:

instance Semigroup GetTokenError where
  e1 <> e2 = RepeatedErrors e1 e2

I can finally define:

getToken :: ExceptT GetTokenError IO OAuthToken
getToken =
  getTokenFromEnvironment <|>
  getTokenFromCommandLine

This is really nice because this is exactly what I want to express!

• get the token from the environment
• if that doesn’t work get it from the command line

More abstraction on the way

This looks all good but something is annoying. My functions are using ExceptT GetTokenError IO which is a very concrete monad stack. I am going to make further calls to other monad stacks to make HTTP calls and I might have to lift all over the place to align all the different stacks. This is even more annoying if I create a small library for getting tokens because I impose my stack choices on all clients of the API.

There is a way out of this, the monad transformers library (mtl). In the mtl there are all sorts of typeclasses abstracting features provided by monad transformers. One of them is MonadError:

class Monad m => MonadError e m | m -> e where
  throwError :: e -> m a
  catchError :: m a -> (e -> m a) -> m a

If a Monad has a MonadError instance then you can catch and throw errors. Nice. However there is a catch (no pun intended :-)). The | m -> e part of MonadError means that the Monad m you are going to eventually select to support the MonadError functionality is dependent on the error type e.

Concretely this means that this type signature propagates to the top of the application:

getToken :: (MonadError GetTokenError m, MonadIO m) => m OAuthToken

Indeed, if I mix other calls to getToken, for example

getPartitions :: (MonadError GetTokenError m, MonadIO m) => m [Partition]
getPartitions =
  do token <- getToken
     callService token partitionsRequest

Then the MonadError GetTokenError m constraints stays because m is tied to it forever. And if callService declares its own error type, I won’t be able to mix both getToken and callService calls:

callService :: (MonadError HttpError m, MonadIO m) :: OauthToken -> Request a -> a

Maybe one solution would be to define getPartitions as:

getPartitions :: (MonadError (GetTokenError Either ServiceError) m, MonadIO m) => m [Partition]

In a way we are back to the previous problem. We now have a “concrete stack” of errors where we just want “a” structure capable of holding both GetTokenError and HttpError.

Lenses to the rescue

A partial solution to the problem is given by lenses and is wonderfully explained by Georges Wilson in “next level mtl with classy optics” (I also recommend this gist by Nick Partridge):

-- create prisms for GetTokenError
makeClassyPrisms ''GetTokenError

-- now e is anything having a Prism allowing us to extract or inject a GetTokenError
getToken :: (MonadError e m, MonadIO m, AsGetTokenError e) => m OAuthToken

So it becomes kind of easier to mix calls having different error types:

getPartitions :: (MonadError e m, AsGetTokenError e, AsHttpError e, MonadIO m) => m [Partition]

This is probably a better way to mix different error types with MonadError. We don’t get rid of the functional constraint though. The error types still “bubble-up” to the top and the method used to do the authentication is exposed to the clients of getPartitions. So if we switch the authentication mechanism and get different error types we will have to change all the functions calling getPartitions.

I think we need to really take care of this kind of situation because it makes software a lot harder to evolve when a small change has ripple effects across all the software layers.

Errors translation / encapsulation

What we can do is to define a new error type subsuming GetTokenError and HttpError:

data ServiceError =
  AuthenticationError GetTokenError |
  CallError HttpError

makeClassy ''ServiceError

instance AsGetTokenError ServiceError
instance AsHttpError ServiceError

And we need a bit of boilerplate to do the translation between getToken and callService. To be able to “liberate” those 2 functions from their MonadError e m, MonadIO m constraints we can run them with the minimum stack which satisfies these constraints, this is ExceptT <error type> IO a:

runOAuthToken :: (MonadError e m, AsCliError e, MonadIO m) => ExceptT GetTokenError IO a -> m a
runOAuthToken = runExceptTIO (review _AuthenticationError)

runCallService :: (MonadError e m, AsCliError e, MonadIO m) => ExceptT HttpError IO a -> m a
runCallService = runExceptTIO (review _HttpError)

runExceptTIO :: (MonadError e m, MonadIO m) => (f -> e) -> ExceptT f IO a -> m a
runExceptTIO mapError ioa =
  do valueOrError <- liftIO (runExceptT ioa)
     fromEitherM (throwError . mapError) valueOrError

(fromEitherM comes from the from-sum package)

Then we can make both calls and “unify” them under new constraints:

getPartitions :: (MonadError e m, AsServiceError e, MonadIO m) => m [Partition]
getPartitions =
  do token <- runToken getToken
     runService (callService token partitionsRequest)

Alternatives?

Unfortunately being more abstract with getToken breaks the use of <|> which was so convenient. One option for fixing it is to define a similar operator for mtl classes:

(<|>) :: (MonadError e m, Semigroup e) => m a -> m a -> m a
(<|>) ma1 ma2 =
  catchError ma1 (\e1 ->
    catchError ma2 (\e2 ->
      throwError (e1 <> e2)))

But this still doesn’t work given the way we have defined our errors as prisms over a general error type! We need to define a extension of <|> which uses the appropriate Prism to aggregate errors:

(<|?>) :: (MonadError e m, Semigroup o) => Prism' e o -> m a -> m a -> m a
(<|?>) p ma1 ma2 = catchError ma1 (\e1 ->
  case e1 ^? p of
    Nothing -> throwError e1
    Just o1 ->
      catchError ma2 (\e2 ->
        case e2 ^? p of
          Nothing -> throwError e1
          Just o2 -> throwError (review p (o1 <> o2))))

And finally

getToken :: (MonadError e m, MonadIO m, AsGetTokenError e) => m OAuthToken
getToken =
  (<|?>) _GetTokenError
  getTokenFromEnvironment
  getTokenFromCommandLine

This is not syntactically as nice as before but we can improve that:

infix 4 <!>
infix 3 <?>

data Alternating a = Alternating a a

(<!>) :: MonadError e m => m a -> m a -> Alternating (m a)
(<!>) = Alternating

(<?>) :: (MonadError e m, Semigroup o) => Prism' e o -> Alternating (m a) -> m a
(<?>) = error "left to the reader"

getToken =
  _GetTokenError <?>
  getTokenFromEnvironment <|>
  getTokenFromCommandLine

What about laws?

This is not the happy end of the story. The Alternative type class specifies that <|> must be an associative operation. This means that we probably need to prove that our <|> and its evil twin <|?> are associative operators. What does that mean for instances of MonadError e m? By the way what is even a MonadError e m?

I was pretty shocked to realize that MonadError doesn’t come up with any laws in Haskell! We need to turn to Purescript for this:

-- | - Left zero: `throwError e >>= f = throwError e`
-- | - Catch:     `catchError (throwError e) f = f e`
-- | - Pure:      `catchError (pure a) f = pure a`

And we probably need an additional law for our <|> operator:

-- | - Catch associativity: `catchError (catchError ma f1) f2 =
--                           catchError ma (\e -> catchError (f1 e) f2)`

Since many instances already exist for MonadError the best we can do is to check if those laws hold for these instances.

I was at first very enthusiastic about that. There are 2 libraries in Haskell which can be used to retroactively find properties for a give API: quickspec and speculate. However I was unable to get any of them to compile / find laws. Since I had been shaving too many yaks on this application I decided to stop there. But I hope I will get some time to get those libraries to find laws.

Anyway, the best I could do was to check the laws for all the instances of MonadError: EitherT, ListT, MaybeT, ExceptT and so on. I verified a bunch of them using QuickCheck and they seem to hold. Actually if that wasn’t the case I would be surprised and would probably question the implementation. I also proved that the laws were holding for the Either instance of MonadError. It goes like this, for the associativity of catch:

-- this is what we want to prove
catchError (catchError m k2) k1 == catchError m (\\e -> catchError (k2 e) k1)

--> case 1: m = Left t
catchError (catchError (Left t) k2) k1 == catchError (Left t) (\\e -> catchError (k2 e) k1)

-- we apply on each side the definition of catchError for Either when the (m a)
-- value is a Left value which is to call the handler, CQFD
catchError (k2 t) k1 = catchError (k2 t) k1

--> case 2: m = Right a
catchError (catchError (Right a) k2) k1 == catchError (Right a) (\\e -> catchError (k2 e) k1)

-- we apply on each side the definition of catchError for Either when the (m a) value
-- is a Right value which is to leave it as it is, CQFD
catchError (Right a) k1 = Right a

-- we do it once more on the left side of the equation
Right a = Right a

I suspect that it is possible to prove the laws for other MonadError instances, StateT would probably be interesting.

What have we learned?

There are plenty of lessons for me in this exercise:

• Haskell is full of interesting generalization, <|> is one of them and worth spotting in production code

• Haskell libraries are not necessarily complete and can probably benefit from contributions

• The modularity story of Haskell is not as obvious as what you might think. I had to think hard to find a satisfying solution and one huge problem is still unsolved. How can I test my full application code with a dummy authentication, which would always succeed? This will be the subject of a following post!

ICFP 2017

I already blogged last year on ICFP, the awesome conference on Functional Programming, to share a neat trick.

This year I would like to do a small brain dump on some of the sessions which I really liked.

Workshop on higher-programming with effects

Programming a web server with effects

This is one of several talks showing the benefits of programming with a language with natively supports the description of effects in the type system: Koka. Not only Daan Leijen is a great presenter whose enthusiasm is contagious but with Koka it feels entirely natural to write asynchronous code with interleaving, cancellation, resource-cleanup and so on.

The also hints at something which got me really interested, the yield operator to create iterators can be seen as an effect. Coupled to the async support above this makes me think that a whole streaming library could be created as a set of effects!

Workshop on type-driven development

Type-directed diffing of structured data

A fantastic idea: can be do better than line-diffing to diff code and do AST diffs? The presenter introduced a language for representing AST patches (not as easy as it seems) then different possibilities for algorithms doing the diff (clearly a hard problem if you want to find the minimum diff).

ICFP

Super 8 languages for making movies

I’m not a big fan of dynamic languages but the power of Racket is impressive. The presenter shows how she created a “Tower of DSLs” to create a video-editing tool. Complete with parser, GUI, documentation, integration with video libraries in a lot less lines of code than anything else I know.

Faster coroutine pipelines

Coroutine pipelines are the functional way to create streaming libraries. However they can be made faster. How? But using an encoding based on continuations rather than the “direct” encoding. This trick seems to be pervasive. For example, for algebraic effects, ScalaEffekt uses such an encoding and gets much better performances than eff.

A unified approach for solving seven programming problems

Racket is at it again but with a secret weapon, miniKanren. If you view a computation as a “logical” relationship between a program and some outputs, it is very reasonable to ask your solver to find the programs which give you the right outputs. And find, for example, Quines which are programs which output themselves!

Generic functional parallel algorithms: parallel scan

Scans, computing the prefix sums of a list of ints, are an important ingredient in some algorithms, like regular expression search. If they can be performed in parallel it’s even better! This talk shows that a scan algorithm can be defined for generic datastructures, just defined by sums, products and functor composition. Then by adjusting the decomposition you get different amounts of parallelism and work to compose results.

Effect-driven quickchecking of compilers

Very entertaining presentation showing that it is entirely possible to generate well-type terms to test a compiler.

Compiling to categories

My favourite talk this year. What if you consider lambda and apply in the lambda calculus as 2 operations which could be interpreted differently? Conal Elliott developed a GHC plugin to transform Haskell expressions into objects in a category where function composition corresponds to the composition of morphisms in a category. What does that give us? Instead of applying functions and getting a result we can generate circuits computing them, differentiate them or do interval computations where the results are intervals instead of just being values.

Staged Generic Programming

Generic programming is cool. Write a few lines of code and let the compiler figure the right data structure and / or program. Unfortunately the performances might be very bad compared to hand-written code. With staging we can let the compiler be smarter and generate “unrolled” code which will perform as well as the hand-written one.

I was also glad to get to talk to Jeremy Yallop, the presenter, who is a very nice guy, now working at Docker on unikernels, something to keep an eye on!

Whip: higher-order contracts for modern services

This is looks more like a CUFP talk, with real applications in the industry :-). The idea is to install some “decorators” on webservices to check where contracts are being broken. They have a nice website and I am tempted to try this at work.

Haskell Symposium

Algebraic graphs with Class

A simple but very powerful to describe graphs “algebraically”. Some atoms: the empty graph, the singleton graph and 2 operations: overlay (sort of addition) and connect (sort of multiplication). Based on that you can represent any graph and encode any algorithm against that representation, which can then optimised for different purposes.

Quickspec / Speculate

Those 2 tools will generate the equations that you API is supposed to satisfy. For example, given [], ++ and reverse they should be able to generate reverse ([] ++ xs) == reverse xs. But even more, they can now generate implications and inequalities!

Composable networks and remote monads

The remote monad explores different ways of “grouping” operations when addressing remote services. After tried to do something similar for eff I appreciate a lot more the subject :-).

Testing with crowbar

Impressive example about finding bugs in a unicode library in 10 minutes which QuickCheck can not find in 1 week. The secret? Fuzzing and code coverage. Generate random sequence, turn them to test cases and mutate the sequences until you find some “interesting” part of the code. Those are likely to contain bugs.

ML Family workshop

State machines all the way down

My periodic reminder that I need to find some time to use Idris. Even more so after Edwin Brady said that the state machine programming with the type system tracking before/after state subsumes the effects library! He also told me that he was implementing the Idris compiler in Idris and this was a good example for him about “when not to go too far with types”.

Effects without monads: non determinism

Oleg Kiselyov is back, implementing non-determinism with TTFI (typed tagless final interpreters). This example is particularly interesting because he shows that monads are not necessary to interpret this mini-language. Not only that but that but they cannot even be used if your translation is about generating code. Because you cannot write code which will write code!

Bioinformatics: the typed tagless final way

A concrete example of people using TTFI in real life for large and extensible DSLs.

CUFP

Bonsai: DSL for serverless decisioning

This DSL is pretty simple, it is just decision trees (if / then / else) but compiled for maximum throughput for the ad industry.

Gens N’ Roses: appetite for reduction

Jacob Stanley talk on Hedgehog where shrinks come for free and failures are beautiful. A “must-use” for Haskell users.

Haskell implementors workshop

Syntactic musings

This was a lightning talk but it made me laugh. Those are proposals to improve haskell syntax with almost zero possibilities to be adopted :-):

How to write bracket without parenthesis:

Instead of

bracket
  (acquire some resources)
  (do your stuff which might be
   pretty long)
  (and clean up)

Go (yes, bullet points!)

bracket
  . acquire some resources
  . do your stuff which might be
     pretty long
  . and clean up

Then set a value for a bunch of functions

context fixed (value) where
  add a = value + a
  mul a = value * a

to avoid the repetition of value and to avoid accidentally messing with it. This screams like “module/functor envy” to me.

The last one is crazy but brilliant. Get away with grouping with parentheses by removing whitespace where you want to indicate higher precedence

a+b * c+d

Why not :-)?

An experiment in fragment-based code distribution

I thought this idea could really not roll but now I am intrigued. Following Joe Armstrong’s “Why do we need modules at all?” Philip Schuster implemented a build system for haskell where each program is decomposed into “slices” containing just one function and its dependencies. Everything identified by numbers. Advantages: just compile and package what you need, never care about version numbers. But the job of curating all of that to get a coherent snapshot, oh boy! As a data point, running anything on Scotty requires 12.000 slices.

Conclusion

A very intense week as in the previous editions I have been to. I don’t know yet what will stick with me and influence my future work but there’s plenty to think about. I also enjoyed very much the discussions I had with Jonathan and Philipp, the authors of scala-effekt.

Specs2 4.x

It has been now almost 3 years since the last major version of specs2. But why would a new major version of a testing library be even needed? Because the world doesn’t stop revolving!

In particular Scala can now be compiled to JavaScript and even natively. It is then only natural that specs2 users want to use specs2 on their Scala.js projects:

How hard can this be? First of all specs2 3.x relies on scalaz-stream which doesn’t have a Scala.js build, and more largely on scalaz (which has a Scala.js build).

This is where an issue becomes an opportunity! If I need to find an alternative to scalaz-stream why not remove scalaz altogether? Indeed it is very inconvenient for a test library to rely on third-party libraries which might conflict with users libraries. Because of this conflict I have to publish several versions of specs2 for different versions of scalaz (7.0.x, 7.1.x, 7.2.x and so on). This is pretty time-consuming, not only because of having to wait for more than 30 minutes for all the jars to be published but also because I have to adapt to some small differences between each scalaz version.

Removing scalaz and scalaz-stream is easier said than done:

1. specs2 relies heavily on functional programming so I need the usual FP suspects: Functor, Applicative, Monad and so on

2. specs2 interacts with the file system, writes out to the console, executes specifications concurrently so it needs a way to control effects

3. specs2 is structured around the idea of a “stream of specification fragments” so it needs a streaming abstraction

All of this before I can even consider moving to a Scala.js build!

specs2 own FP / streaming infrastructure

Fortunately I have developed on the side many of the pieces needed above:

1. eff is a library to handle effects, including concurrency and resources management

2. producer is a simple streaming library designed to work with any Monad, including the Eff monad

3. origami implements a notion of composable “folds” which can be applied to any stream, which is a pattern I (re)-discovered on earlier versions of specs2: reporting the results of an executed specification is just “folding” the specification, a bit like using foldLeft on a List

A copy of those libraries is now included in the specs2-common module.

The only thing missing was the FP abstractions. It turns out that it is not too hard to implement the whole menagerie:

• Functor, Applicative, Monad, Traverse, Foldable, Monoid. Those typeclasses are very tied together
• Tree and TreeLoc for manipulating trees

I am greatly indebted to the Scalaz and cats projects for this code, which I reduced to the only parts I needed for specs2. Those abstractions and datatypes are now available in the specs2-fp module (not intended for external use).

Execution model

At the heart of specs2 is a Fragment, which is a Description and an Execution yielding a Result. The Description holds an informal description of the specified system and the Execution runs some Scala code to verify that behaviour.

Using Scala.js imposes a big change is specs2. We can never await to the get the result of executing a Fragment. Also since scalaz.concurrent.Task is out of the equation, in specs2 4.x an Execution is implemented using a scala.concurrent.Future, like this:

case class Execution(
 run:       Option[Env => Future[() => Result]],
 executing: Option[Throwable Either Future[Result]])

This is a lot more complicated than just Future[Result]. Why is that?

run takes an action to execute () => Result, possibly concurrently Future[() => Result] and which might depend on the environment Env => Future[() => Result]. You might wonder why we don’t simply use Env => Future[Result]. This is to allow the user to add some behaviour around the result, for example with the AroundEach trait and the def around[R : AsResult](r: =>R): Result method.

We also want to “stream” results, which are executed concurrently and display them as soon as they are available. This is done by having executing holding either a Throwable if we could not even start the execution or the Result being currently computed.

Note that we can never call await in a Scala.js program so when we are executing a Fragment the best we can get is the equivalent of Future[Result] (in reality an Action[Result] which is a type possibly having more effects than just concurrency).

This triggers major changes in the implementation of specs2 and there are rippling effects up to the Runners used to run a specification. However most of the users of specs2 should not be concerned with these changes as the “user” DSL for specs2 isn’t modified. However if you are on the JVM and really need to await to get the result of a fragment execution you can do so by using one of the methods in the org.specs2.control.ExecuteActions object. For example result.runAction(executionEnv) will return an Error Either Result. Running the same method on Scala.js will throw an exception.

Scala.js modules

“specs2 runs on Scala.js” is ever going to be a partial truth. There are many limitations related to Scala.js and this means that, while specs2-core can be used on any Scala.js project, there are some specs2 features and modules which cannot be used with Scala.js. Here is a list (up to my current knowledge):

1. the isolate argument to run each example in its own copy of the specification cannot be used (because this uses reflection to instantiate classes)

2. no custom Selector/Executor/Printer/Notifier can be passed from the command line (for the same reason)

3. the RandomSequentialExecution trait cannot be used because the implementation uses a TrieMap not available on Scala.js. This could be fixed in the future.

4. running only examples which previously failed with was x on the command-line is not possible because this accesses file system

5. all the modules where scala-xml is used don’t have a Scala.js version because there’s no scala-xml for Scala.js: specs2-form, specs2-html

6. modules accessing the file system cannot be used: specs2-html, specs2-markdown

7. the specs2-gwt module defines fragments based on the result of previous fragments, so it currently needs to await. This could be fixed potentially with a smart enough implementation in the future.

Breaking changes

A major version number is also the opportunity to fix some of the API flaws. I tried to limit the changes in the low-level API to the strict necessary but I also wanted to clean-up one important aspect of the high-level API.

When specs2 3.x came out I envisioned that there could be ways to create a specification from arguments passed on the command line, or from an ExecutionContext (to execute futures). With the addition of some traits like CommandLineArguments you could declare:

class MySpec extends Specification with CommandLineArguments {
  def is(args: CommandLine) = s2"""
    try this $ok
"""
}

or

class MySpec extends Specification { def is = s2"""
  try this $ex1
"""

  def ex1 = { implicit ec: ExecutionContext => Future(1) must be_==(1).await }
}

I later realized that it would be a lot more convenient to directly “inject” the command line arguments or the execution context in the specification like so:

class MySpec(args: CommandLine) extends Specification { def is = s2"""
  try this $ex1
"""

 def ex1 = { (1 + 1) must be_==(2).when(args.contains("universe-is-sane"))}
}

class MySpec(implicit ec: ExecutionContext) extends Specification { def is = s2"""
  try this $ex1
"""

  def ex1 = { Future(1) must be_==(1).await }
}

Now there were 2 ways to do the same thing, the second one being a lot easier than the first one. In specs2 4.x the traits supporting the first behavior org.specs2.specification.Environment, org.specs2.specification.CommandLineArguments, org.specs2.specification.ExecutionEnvironment have been removed. Same thing for the implicit conversions for functions like implicit ee: ExecutionEnv => Result to create examples.

This will demand some migration efforts for some users but the result will be more concise specifications.

Another word on injected execution environments. From specs2 3.8.9, 2 distinct execution environments are being used in specs2. One for the execution of the specification itself, the other one for the execution of the user examples (to map futures or await for results for example). This is to make sure that some incorrect user code doesn’t prevent the full specification from being executed.

With specs2 4.x we can go one step further and allow each specification to get its own dedicated execution environment and make sure that one specification execution doesn’t impact another one:

import org.specs2.specification.core.{Env, OwnExecutionEnv}

class MySpec extends(val env: Env) extends Specification with OwnExecutionEnv { def is = s2"""
    ...
"""
}

The injected env is used to pass command-line argument to a copy of the user execution env, implicitly available as a member of the OwnExecutionEnv trait. That trait also takes care of shutting down the execution environment once the specification has finished executing so as not to lose resources.

Feedback is essential

In conclusion, specs2 4.x should be a smaller gap from specs2 3.x than specs2 2.x to specs2 3.x. With a lot more to offer!

1. no more dependencies for specs2-core

2. a Scala.js version for specs2-core

3. better control on execution environments

The release notes can be found here.

Please, please, please ask on gitter, on the mailing-list, on twitter if you have any issue. And a big thank you for reporting any mistake, big or small, that I may have made on this release!

A neat trick from ICFP 2016

ICFP is an awesome conference where I can’t pretend I understand every presentation I’m attending but there is definitely lots of food for thought and some gems which makes it immensely worthwhile.

The gem I want to share here is a paper called “All sorts of permutations”.

This paper shows that you can use the filterM function with a predicate like a -> [True, False] and generate all the sublists of a list.

Similarly, you can evolve the sort function into sortM, taking a monadic comparison function: a -> a -> m Bool. If the comparison function is a -> a -> [True, False] the sortM function will generate the permutations of a list.

The paper goes on exploring if sortM generates all the permutations of a list or exactly the permutations of a list depending on the implementation of the sortM function (insertion sort, merge sort, quicksort,…) and tries to set various restrictions on the comparison function to get there.

At the end of the talk Philip Wadler asked the following question (paraphrased):

This is a neat trick to generate permutations but why should I teach it to my students, considering that there are more ‘classical’ implementations for getting the permutations of a list?"

I think that it is worth showing students because this sortM function gives us a very neat way of writing a QuickCheck generator for distinct elements!

It is indeed not uncommon to have to generate lists of distinct elements for testing applications. For example if you want to test a function scheduling different tasks and these tasks are supposed to be unique, but arriving in any order.

There is unfortunately no permutations :: [a] -> Gen [a] combinator in QuickCheck. So how can you produce random Lists with distinct elements?

One way is to take an existing list of distinct elements and shuffle it to get a random permutation of those elements. This is exactly what sortM gives us, we just need to use the right monad, which happens to be Gen instead of []!

Let’s take an example. Say you want to generate all the permutations of ["Homer", "Marge", "Bart", "Lisa"]

import Test.QuickCheck

-- An implementation of sortM
sortM :: Monad m => (a -> a -> m Bool) -> [a] -> m [a]
sortM _ [] = return []
sortM cmp (x : xs) = do
  ys <- sortM cmp xs
  insertM cmp x ys

insertM :: Monad m => (a -> a -> m Bool) -> a -> [a] -> m [a]
  insertM _ x [] = return [x]
  insertM cmp x yys@(y : ys) = do
    b <- cmp x y
    if b then return (x : yys)
    else fmap (y:) (insertM cmp x ys)

-- A generator for lists of distincts "Simpsons"
type Simpson = String

simpsons = ["Homer", "Marge", "Bart", "Lisa"]

distinctSimpsons :: Gen [Simpson]
distinctSimpsons = sortM (\_ _ -> choose(True, False)) simpsons

Let’s test it:

λ> sample distinctSimpsons
["Homer","Lisa","Marge","Bart"]
["Homer","Marge","Bart","Lisa"]
["Homer","Marge","Lisa","Bart"]
["Homer","Marge","Bart","Lisa"]
["Bart","Homer","Marge","Lisa"]
["Marge","Homer","Bart","Lisa"]
["Marge","Homer","Lisa","Bart"]
["Marge","Homer","Lisa","Bart"]
["Marge","Bart","Lisa","Homer"]
["Homer","Marge","Bart","Lisa"]
["Bart","Homer","Lisa","Marge"]

I think this answers Philip’s question, this is a nice application of the sortM “trick” which can be taught to students learning functional programming.

And what looks really promising is that this idea extends to the generation of other data structures. For example we can generate the partitions of a given set with groupByM:

partitionM :: Monad m => (a -> m Bool) -> [a] -> m ([a], [a])
partitionM _ [] = pure ([], [])
partitionM p (x : rest) = do
  b        <- p x
  (xs, ys) <- partitionM p rest
  if b then pure (x:xs, ys)
  else      pure (xs, x:ys)

groupByM :: Monad m => (a -> a -> m Bool) -> [a] -> m [[a]]
groupByM _ [] = pure []
groupByM cmp (x : rest) = do
  (xs, ys) <- partitionM (cmp x) rest
  ps       <- groupByM cmp ys
  pure $ (x:xs):ps

Then we can get all the set partitions of ["Homer","Lisa","Marge","Bart"]

λ> pl $ groupByM (\_ _ -> [True, False]) simpsons
[["Homer","Marge","Bart","Lisa"]]
[["Homer","Marge","Bart"],["Lisa"]]
[["Homer","Marge","Lisa"],["Bart"]]
[["Homer","Marge"],["Bart","Lisa"]]
[["Homer","Marge"],["Bart"],["Lisa"]]
[["Homer","Bart","Lisa"],["Marge"]]
[["Homer","Bart"],["Marge","Lisa"]]
[["Homer","Bart"],["Marge"],["Lisa"]]
[["Homer","Lisa"],["Marge","Bart"]]
[["Homer","Lisa"],["Marge"],["Bart"]]
[["Homer"],["Marge","Bart","Lisa"]]
[["Homer"],["Marge","Bart"],["Lisa"]]
[["Homer"],["Marge","Lisa"],["Bart"]]
[["Homer"],["Marge"],["Bart","Lisa"]]
[["Homer"],["Marge"],["Bart"],["Lisa"]]

or get some random partitions of that set:

λ> sample $ groupByM (\_ _ -> choose(True, False)) simpsons
[["Homer","Marge","Bart"],["Lisa"]]
[["Homer","Lisa"],["Marge","Bart"]]
[["Homer","Bart"],["Marge"],["Lisa"]]
[["Homer","Marge","Bart","Lisa"]]
[["Homer","Marge","Bart","Lisa"]]
[["Homer","Lisa"],["Marge"],["Bart"]]
[["Homer","Marge"],["Bart"],["Lisa"]]
[["Homer"],["Marge"],["Bart","Lisa"]]
[["Homer","Marge","Bart","Lisa"]]
[["Homer","Marge","Bart","Lisa"]]
[["Homer","Marge","Lisa"],["Bart"]]

Some partitions can be drawn twice but the generation should be uniform over the set of all possible partitions (it would be better to not take my word for it and prove it though :-)).

I suspect that this is more than a useful trick and that this deserves a more systematic treatment to understand all the data structures which we can generate by using the appropriate functions taking predicates or relations as parameters. But for now it is already useful as it is!

Precise data types

Here is a new post about creating data types to encode different cases in applicationspp0pp, hosted on the Ambiata lab website: Precise data types.

specs2-three

This post presents the next major version of specs2, specs2 3.0:

• what are the motivations for a major version?
• what are the main benefits and changes?
• when will it be available?

The motivations

I started working on this new version a bit more than one year ago now. I had lots of different reasons for giving specs2 a good face-lift.

The Open Source reason

specs2 has largely been the effort of a single person. This has probably some advantages, like the possibility to maintain some kind of vision for the project, but also lots of drawbacks. Quality is one of them.

As a programmer I have all sorts of shortcomings. I have always been amazed by other people taking a look at my code and spotting obvious deficiencies either big or small (for example @jedws introduced named threads to me, much easier for debugging!). I want to maximize the possibility that other people will jump in to fix and extend the library as necessary (and be able to go on holidays for 3 weeks without a laptop :-)). Improving the code base can only help other people review my implementation.

The Design reason

In specs2 I’ve had this vision of a flow of “specification fragments” that would get created, executed, and then reported, possibly with different reporters for various output formats. This is not as easy at it seems:

• I want fragments to be executed concurrently while being printed in sequence
• the fragments should also be displayed as soon as executed
• I want to be able to recreate a view of the sequence of fragments as a tree to be displayed in IDEs like Eclipse and Intellij

This is all done and working in specs2 < 3.0 but in a very clumsy way, subverting Scalaz Reducers to maintain state and try to compose reporters.

One of these reporters is a HTML reporter and I’ve always wanted to improve it. This was not something I was eager to change given the situation 1 year ago. Luckily scalaz-stream version 0.2 came out in December 2013 and allowed me to try out new ideas.

The Functional Programming reason

The major difference between specs and specs2 was the use of immutable data structures and the avoidance of exceptions for control flow. Yet there was still lots of side-effects!

I hadn’t fully grasped how to use the IO monad to structure my program. Fortunately I happen to work with the terrific @markhibberd and he showed me how to use a proper monad stack to track IO effects but also how to thread in configuration data and track errors.

The main benefits and changes

First of all, a happy maintainer! That goes without saying but my ability to fix bugs and add features will be improved a lot if I can better reason about the code :-).

Now for users…

For casual users there should be no changes! If you just use org.specs2.Specification or org.specs2.mutable.Specification with no other traits, you should not see any change (except in the User Guide, see below). For “advanced” users there are new benefits and API changes (in no particular order).

Refactored user guide

The existing User Guide has been divided into a lot more pages (around 60) and follows a pedagogical progression:

• a Quick Start presenting a simple specification (and the mandatory link to the installation page)

• some links from the Quick Start to the most common concepts: what is the structure of a Specification? Which matchers are available? How to run a Specification?

• then, on each other page there is a presentation focusing on one topic plus additional links: "Now learn how to..." (what is the next thing you will probably need?) and "If you want to know more" (what is some more advanced topic that is related to this one?)

In addition to this refactoring there are some “tools” to help users find faster what they are looking for:

• a search box

• reference pages to summarize in one place some topics (matchers and run arguments for example)

• a Troubleshooting page with the most common issues

You can have a first look at it here.

Generalized Reader pattern

One consequence of the “functional” re-engineering is that the environment is now available at different levels. By “environment”, I mean the org.specs.specification.core.Env class which gives you access to all the components necessary to execute a specification, among which:

• the command line arguments
• the lineLogger used to log results to the console (from Sbt)
• the systemLogger used to log issues when instantiating the Specification for example
• the execution environment, containing a reference to the thread pool used to execute the examples
• the statsRepository to get and store execution statistics
• the fileSystem which mediates all interactions with the file system (to read and write files)

I doubt that you will ever need all of this, but parts of the environment can be useful. For example, you can define the structure of your Specification based on command line arguments:

class MySpec extends Specification with CommandLineArguments { def is(args: CommandLine) = s2"""
  Do something here with a command line parameter ${args.valueOr("parameter1", "not found")}
"""
}

The CommandLineArguments uses your definition of the def is(args: CommandLine): Fragments method to build a more general method Env => Fragments which is the internal representation of a Specification (fragments that depend on the environment). This means that now you don’t have to skip examples based on condition (isDatabaseAvailable for example), you can simply remove them!

You can also use the environment, or part of it, to define examples:

class MySpec extends Specification { def is = s2"""

 Here are some examples using the environment.
 You can access
   the full environment                            $e1
   the command line arguments                      $e2
   the execution context to create a Scala future  $e3
   the executor service to create a Scalaz future  $e4

"""

  def e1 = { env: Env =>
    env.statisticsRepository.getStatistics(getClass.getName).runOption.flatten.foreach { stats =>
      println("the previous results for this specification are "+stats)
    }
    ok
  }

  def e2 = { args: CommandLine =>
    if (args.boolOr("doit", false)) success
    else skipped
  }

  def e3 = { implicit executionContext: ExecutionContext =>
    scala.concurrent.Future(1); ok
  }

  def e4 = { implicit executorService: ExecutorService =>
    scalaz.concurrent.Future(1); ok
  }
}

Better reporting framework

This paragraph is mostly relevant to people who want to extend specs2 with additional outputs. The reporting framework has been refactored around 4 concepts:

A Runner (for example the SbtRunner)

• instantiates the specification class to execute
• creates the execution environment (arguments, thread pool)
• instantiates a Reporter
• instantiates Printers and starts the execution

A Reporter

• reads the previous execution statistics if necessary
• selects the fragments to execute
• executes the specification fragments
• calls the printers for printing out the results
• saves the execution statistics

A Printer

• prepares the environment for printing
• uses a Fold to print or to gather execution data. For example the TextPrinter prints results to the console as soon as they are available and the HtmlPrinter

A Fold

• has a Sink[Task, (T, S)] (see scalaz-stream for the definition of a Sink) to perform side-effects (like writing to a file)
• has a fold: (T, S) => S method to accumulate some state (to compute statistics for example, or create an index)
• has an init: S element to initialize the state
• has a last(s: S): Task[Unit] method to perform one last side-effect with the final state once all the fragments have been executed

It is unlikely that you will create a new Runner (except if you build an Eclipse plugin for example) but you can create custom reporters and printers by passing the reporter <classname> and printer <classname> options as arguments. Note also that Folds are composable so if you need 2 outputs you can create a Printer that will compose 2 folds into 1.

The Html printer

Pandoc

The Html printer has been reworked to use Pandoc as a templating system and Markdown engine. I decided this move to Pandoc for several reasons:

• Pandoc is one of the libraries that is officially endorsing the CommonMark format
• I’ve had less corner cases with rendering mixed html/markdown with Pandoc than previously
• Pandoc opens the possibility to render other markup languages than CommonMark, LaTeX for example

However this comes with a huge drawback, you need to have Pandoc installed as a command line tool on your machine. If Pandoc is not installed, specs2 will use a default template renderer, but won’t render CommonMark.

Templates

I’ve extracted a specs2.html template (and a corresponding specs2.css stylesheet) and it is possible for you to substitute another template (with the html.template option) if you want your html files to be displayed differently. This template is using the Pandoc template system so it is pretty primitive but should still cover most cases.

Better API

The specs2 API has been split into a lot more traits to support various objectives:

• support the new execution model with scalaz-stream

• make it possible to separate the DSL methods from the core ones (see Lightweight spec)

• offer a better Fragment API

Let’s start with the heart of specs2, the Fragment.

FragmentFactory methods

Advanced specs2 users need to tweak the creation of fragments. For example, when using a “template specification”:

abstract class DatabaseSpec extends Specification {
  override def map(fs: => Fragments): Fragments =
    step(startDb) ^ fs ^ step(closeDb)
}

In the DatabaseSpec you are using different methods to work with Fragments. The ^ method to append them, the step method to create a Step fragment. Those 2 methods are part of the Fragment API. Here is a list of the main changes, compared to specs2 < 3.0

• first of all there is only one Fragment type (instead of Text, Step, Example,…). This type contains a Description and an Execution. By combining different types of Descriptions and Executions it is possible to recreate all the previous specs2 < 3.0 types

• however you don’t need to create a Fragment by yourself, what you do is invoke the FragmentFactory methods: example, step, text,… This now unifies the notation between immutable and mutable specifications because in specs2 < 3.0 you would write step in a mutable specification and Step in an immutable one (Step is now deprecated)

• there is no ExampleFactory trait anymore since it has been subsumed by methods on the FragmentFactory trait (so this will break code for people who were intercepting Example creation to inject additional behaviour)

Finally those “core” objects have been moved under the org.specs2.specification.core package, in order to restructure the org.specs2.specification package into

• core: Fragment Description, SpecificationStructure…

• dsl: all the syntactic sugar FragmentsDsl, ExampleDsl, ActionDsl…

• process: the “processing” classes Selector, Executor, StatisticsRepository…

• create: traits to create the specification FragmentFactory, AutoExamples, S2StringContext (for s2 string interpolation)…

FragmentsDsl methods

When you want to assemble Fragments together you will need the FragmentsDsl trait to do so (mixed-in the Specification trait, you don’t have to add it).

The result of appending 2 Fragments is a Fragments object. The Fragments class has changed in specs2 3.0. It doesn’t hold a reference to the specification title and the specification arguments anymore, this is now the role of the SpecStructure. So in summary:

• a Specification is a function Env => SpecStructure

• a SpecStructure contains: a SpecHeader, some Arguments and Fragments

• Fragments is a sequence of Fragments (actually a scalaz-stream Process[Task, Fragment])

The FragmentsDsl api allows to combine almost everything into Fragments with the ^ operator:

• a String to a Seq[Fragment]
• 2 Fragments
• 1 Fragments and a String

One advantage of this fine-grained decomposition of the fragments API is that there is now a Spec lightweight trait.

Lightweight Spec trait

Compilation times can be a problem with Scala and specs2 makes it worse by providing lots of implicit methods in a standard Specification to provide various DSLs. In specs2 3.0 there is a Spec trait which contains a reduced number of implicits to:

• create a s2 string for an “Acceptance Specification”
• create should and in blocks in a “Unit Specification”
• to create expectations with must
• to add arguments to the specification (like sequential)

If you use that trait and you find yourself missing an implicit you will have to either:

• use the Specification class instead

• search specs2 for the trait or object providing the missing implicit. There is no magic recipe for this but the MustMatchers trait and the Spec2StringContext trait should bring most of the missing implicits in scope

It is possible that this trait will be adjusted to find the exact balance between expressivity and compile times but I hope it will remain pretty stable.

Durations

When specs2 started, the package scala.concurrent.duration didn’t exist. This is why there was a Duration type in specs2 < 3.0 and a TimeConversions trait. Of course this introduced annoying collisions with the implicits coming from scala.concurrent.duration when that one came around.

There is no reason to go on using specs2 Durations anymore now so you can use the standard Scala durations everywhere specs2 expects a Duration.

Contexts

Context management has been slowly evolving in specs2. In specs2 3.0 we end up with the following traits:

• BeforeAll do something before all the examples (you had to use a Step in specs2 < 3.0)
• BeforeEach do something before each example (was BeforeExample in specs2 < 3.0)
• AfterEach do something after each example (was AfterExample in specs2 < 3.0)
• BeforeAfterEach do something before/after each example (was BeforeAfterExample in specs2 < 3.0)
• ForEach[T] provide an element of type T (a “fixture”) to some each example (was FixtureExample[T] in specs2 < 3.0)
• AfterAll do something after all the examples examples (you had to use a Step in specs2 < 3.0)
• BeforeAfterAll do something before/after all the examples examples (you had to use a Step in specs2 < 3.0)

There are some other cool things you can do. For example set a time-out for all examples based on a command line parameter:

trait ExamplesTimeout extends EachContext with MustMatchers with TerminationMatchers {

  def context: Env => Context = { env: Env =>
    val timeout = env.arguments.commandLine.intOr("timeout", 1000 * 60).millis
    upTo(timeout)(env.executorService)
  }

  def upTo(to: Duration)(implicit es: ExecutorService) = new Around {
    def around[T : AsResult](t: =>T) = {
      lazy val result = t

      val termination =
        result must terminate(retries = 10,
                              sleep = (to.toMillis / 10).millis).orSkip((ko: String) => "TIMEOUT: "+to)

      if (!termination.toResult.isSkipped) AsResult(result)
      else termination.toResult
    }
  }
}

The ExamplesTimeout trait extends EachContext which is a generalization of the xxxEach traits. With the EachContext trait you get access to the environment to define the behaviour used to “decorate” each example. So, in that case, we use a timeout command line parameter to create an Around context that will timeout each example if necessary. You can also note that this Around context uses the executorService passed by the environment so you don’t have to worry about resources management for your Specification.

Included specifications

As I was reworking the implementation of specs2 I also looked for ways to simplify its internal model. In specs2 < 3.0 you can nest a specification inside another one. This adds some significant complexity because a nested specification have its own arguments, its own title. For example during the execution of the inner specification we need to be careful enough to override the outer specification arguments with the inner ones.

I decided to let go of this functionality in favor of a view of specifications as “referencing” each other, with 2 types of references:

• “link” reference
• “see” reference

The idea is to model dependency relationships with “link” and weaker relationships with “see” (when you just want to mention that some information is present in another specification).

Then there are 2 modes of execution:

• the default one
• the “all” mode

By default when a specification is executed, the Runner will try to display the status of “linked” specifications but not “see” specifications. If you use the all argument then we collect all the “linked” specifications transitively and run them respecting dependencies (if s1 has a link to s2, then s2 is executed first).

This is particularly important for HTML reporting when the structure of “link” references is used to produce a table of contents and “see” references are merely used to display HTML links.

Online specifications

I find this exciting even if I don’t know if I will ever use this feature! (it has been requested in the past though).

In specs2 < 3.0 there is a clear distinction between the “creation” time and the “execution” time for specification. Once you have defined your examples you can not add new ones based on your execution results. But wait! this is more or less the property of a Monad! “Produce an action based on the value returned by another action”. Since specs2 3.0 is using scalaz-stream Process under the covers which is a Monad, this means that it is now possible to do the following:

class WikipediaBddSpec extends Specification with Online { def is = s2"""

  All the pages mentioning the term BDD must contain a reference to specs2 $e1

"""

    def e1 = {
      val pages = Wikipedia.getPages("BDD")

      // if the page is about specs2, add more examples to check the links
      (pages must contain((_:Page) must mention("specs2"))) continueWith
       pagesSpec(pages)
    }

    /** create one example per linked page */
    def pagesSpec(pages: Seq[Page]): Fragments = {
      val specs2Links = pages.flatMap(_.getLinks).filter(_.contains("specs2"))

      s2"""
       The specs2 links must be active
       ${Fragments.foreach(specs2Links)(active)}
      """
    }

    def active(link: HtmlLink) =
      s2"""
      The page at ${link.url} must be active ${ link must beActive }"""
  }

The specification above is “dynamic” in the sense that it creates more examples based on the tested data. All Wikipedia pages for BDD must mention “specs2” and for each linked page (which we can’t know in advance) we create a new example specifying that the link must be active.

ScalaCheck

The ScalaCheck trait has been reworked and extended to provide the following features:

• you can specify Arbitrary[T], Gen[T], Shrink[T], T => Pretty instances at the property level (for any or all of the arguments)
• you can easily collect argument values by appending .collectXXX to the property (XXX depends on the argument you want to collect. collect1 for the first, collectAll for all)
• you can override default parameters from the command line. For example pass scalacheck.mintestsok 10000
• you can set individual before, after actions to be executed before and after the property executes to do some setup/teardown

Also, specs2 was previously doing some message reformatting on top of ScalaCheck but now the ScalaCheck original messages have been preserved to keep the consistency between the 2 libraries.

Note: the ScalaCheck trait stays in the org.specs2 package but all the traits it depends on now live in the org.specs2.scalacheck package.

Bits and pieces

This section is about various small things which have changed with specs2 3.0:

Implicit context

There is no more implicit context when you use the .await method to match futures. This means that you have to either import the scala.concurrent.ExecutionContext.global context or to use a function ExecutionContext => Result to define your examples:

s2"""
An example using an ExecutionContext $e1
"""

  def e1 = { implicit ec: ExecutionContext =>
    // use the context here
    ok
  }

Foreach methods

It is possible now to create several examples or results with a foreach method which will not return Unit:

// create several examples
Fragment.foreach(1 to 10)(i => "example "+i ! ok)

// create several examples with breaks in between
Fragments.foreach(1 to 10)(i => ("example "+i ! ok) ^ br)

// create several results for a sequence of numbers
Result.foreach(1 to 10)(i => i must_== i)

Removed syntax

• (action: Any).before to create a “before” context is removed (same thing for after)
• function.forAll to create a Prop from a function

Dependencies

• specs2 3.0 uses scalacheck 1.12.1
• you need to use a recent version of sbt, like 0.13.7
• you need to upgrade to scalaz-specs2 0.4.0-SNAPSHOT for compatibility

Can I use it?

specs2 3.0 is now available as specs2-core-3.0-M2 on Sonatype. I am making it available for early testing and feedback. Please use the mailing-list or the github issues to ask questions and tell me if there is anything going wrong with this new version. I will incorporate your comments in this blog post, serving as a migration guide.

Special thanks

• to Clinton Freeman who started the re-design of the specs2 home page more than one year ago and sparked this whole refactoring
• to Pavel Chlupacek and Frank Thomas for patiently answering many of my questions about scalaz-stream
• to Paul Chiusano for starting scalaz-stream in the first place!
• to Mark Hibberd for his guidance with functional programming

Streaming with previous and next

The Scalaz streams library is very attractive but it might feel unfamiliar because this is not your standard collection library.

This short post shows how to produce a stream of elements from another stream so that we get a triplet with: the previous element, the current element, the next element.

With Scala collections

With regular Scala collections, this is not too hard. We first create a list of all the previous elements. We create them as options because there will not be a previous element for the first element of the list. Then we create a list of next elements (also a list of options) and we zip everything with the input list:

def withPreviousAndNext[T] = (list: List[T]) => {
  val previousElements = None +: list.map(Some(_)).dropRight(1)
  val nextElements     = list.drop(1).map(Some(_)) :+ None

  // plus some flattening of the triplet
  (previousElements zip list zip nextElements) map { case ((a, b), c) => (a, b, c) }
}

withPreviousAndNext(List(1, 2, 3))

> List((None,1,Some(2)), (Some(1),2,Some(3)), (Some(2),3,None))

And streams

The code above can be translated pretty straightforwardly to scalaz processes:

def withPreviousAndNext[F[_], T] = (p: Process[F, T]) => {
  val previousElements = emit(None) fby p.map(Some(_))
  val nextElements     = p.drop(1).map(Some(_)) fby emit(None)

  (previousElements zip p zip nextElements).map { case ((a, b), c) => (a, b, c) }
}

val p1 = emitAll((1 to 3).toSeq).toSource
withPreviousAndNext(p1).runLog.run

> Vector((None,1,Some(2)), (Some(1),2,Some(3)), (Some(2),3,None))

However what we generally want with streams is combinators which you can pipe onto a given Process. We want to write

def withPreviousAndNext[T]: Process1[T, T] = ???

val p1 = emitAll((1 to 3).toSeq).toSource
// produces the stream of (previous, current, next)
p1 |> withPreviousAndNext

How can we write this?

As a combinator

The trick is to use recursion to keep state and this is actually how many of the process1 combinators in the library are written. Let's see how this works on a simpler example. What happens if we just want a stream where elements are zipped with their previous value? Here is what we can write:

def withPrevious[T]: Process1[T, (Option[T], T)] = {

  def go(previous: Option[T]): Process1[T, (Option[T], T)] =
    await1[T].flatMap { current =>
      emit((previous, current)) fby go(Some(current))
    }

  go(None)
}

val p1 = emitAll((1 to 3).toSeq).toSource
(p1 |> withPrevious).runLog.run

> Vector((None,1), (Some(1),2), (Some(2),3))

Inside the withPrevious method we recursively call go with the state we need to track. In this case we want to keep track of each previous element (and the first call is with None because there is no previous element for the first element of the stream). Then go awaits a new element. Each time there is a new element, we emit it, then call recursively go which is again going to wait for the next element, knowing that the new previous element is now current.

We can do something similar, but a bit more complex for withNext:

def withNext[T]: Process1[T, (T, Option[T])] = {
  def go(current: Option[T]): Process1[T, (T, Option[T])] =
    await1[T].flatMap { next =>
      current match {
        // accumulate the first element
        case None    => go(Some(next))
        // if we have a current element, emit it with the next
        // but when there's no more next, emit it with None
        case Some(c) => (emit((c, Some(next))) fby go(Some(next))).orElse(emit((c, None)))
      }
    }

  go(None)
}

val p1 = emitAll((1 to 3).toSeq).toSource
(p1 |> withNext).runLog.run

> Vector((1,Some(2)), (2,Some(3)), (2,None))

Here, we start by accumulating the first element of the stream, and then, when we get to the next, we emit both of them. And we make a recursive call remembering what is now the current element. But the process we return in flatMap has an orElse clause. It says "by the way, if you don't have anymore elements (no more next), just emit current and None".

Now with both withPrevious and withNext we can create a withPreviousAndNext process:

def withPreviousAndNext[T]: Process1[T, (Option[T], T, Option[T])] = {
  def go(previous: Option[T], current: Option[T]): Process1[T, (Option[T], T, Option[T])] =
    await1[T].flatMap { next =>
      current.map { c =>
        emit((previous, c, Some(next))) fby go(Some(c), Some(next))
      }.getOrElse(
          go(previous, Some(next))
        ).orElse(emit((current, next, None)))
    }
  go(None, None)
}

val p1 = emitAll((1 to 3).toSeq).toSource
(p1 |> withPreviousAndNext).runLog.run

> Vector((None,1,Some(2)), (Some(1),2,Some(3)), (Some(2),3,None))

The code is pretty similar but this time we keep track of both the "previous" element and the "current" one.

emit(last paragraph)

I hope this will help beginners like me to get started with scalaz-stream and I'd be happy if scalaz-stream experts out there leave comments if there's anything which can be improved (is there an effective way to combine withPrevious and withNext to get withPreviousAndNext?

I finally need to add that, in order to get proper performance/side-effect control for the withNext and withPreviousAndNext processes you need to use the lazy branch of scalaz-stream. It contains a fix for orElse which prevents it to be evaluated more than necessary.

The revenge of the chunks

This series of posts feels like a whole saga for something which should have a quick an easy way to demonstrate the obvious superiority of functional programming over a simple loop. In the first post. Then the second post was about defining proper scalaz-stream combinators to do the same thing, and particularly how to "chunk" the processing in order to get good performances.

However as I was writing unit tests for my requirements I realized that the problem was harder than I thought. In particular, the files I'm processing can have several sections made of HEADERs and TRAILERs. When you create chunks of lines to process this results in a number of combinations that need to be analysed. A chunk can:

• start with a HEADER but not finish with a TRAILER which is in another chunk
• contain lines only
• contains lines + a TRAILER + a new HEADER
• and so on...

For each of these cases it is necessary to use the current state and the contents of the lines to determine if the file is malformed or not. This is a lot less easy that previously.

All the combinations

This is what I came up with:

  def process(path: String, targetName: String, chunkSize: Int = 10000): String \/ File = {

    val targetPath = path.replace(".DAT", "")+targetName

    val read = 
      linesRChunk(path, chunkSize) |> 
      validateLines.map(lines => lines.mkString("\n"))

    val task = 
      ((read |> process1.intersperse("\n") |> 
      process1.utf8Encode) to io.fileChunkW(targetPath)).run

    task.attemptRun.leftMap(_.getMessage).map(_ => new File(targetPath))
  }

  /**
   * validate that the lines have the right sequence of HEADER/column names/lines/TRAILER
   * and the right number of lines
   */
  def validateLines: Process1[Vector[String], Vector[String]] = {

    // feed lines into the lines parser with a given state
    // when it's done, follow by parsing with a new state
    def parse(lines: Vector[String], state: LineState, newState: LineState) =
      emit(lines) |> linesParser(state) fby linesParser(newState)

    // parse chunks of lines
    def linesParser(state: LineState): Process1[Vector[String], Vector[String]] = {
      receive1[Vector[String], Vector[String]] { case lines =>
        lines match {
          case first +: rest if isHeader(first) =>
            if (state.openedSection) fail("A trailer is missing")
            else
              parse(lines.drop(2),
                    state.open,
                    LineState(lines.count(isHeader) > lines.count(isTrailer), 
                              lines.drop(2).size))

          case first +: rest if isTrailer(first) =>
            val expected = "\\d+".r.findFirstIn(first).map(_.toInt).getOrElse(0)

            if (!state.openedSection)             
              fail("A header is missing")
            else if (state.lineCount != expected) 
              fail(s"expected $expected lines, got ${state.lineCount}")
            else {
              val dropped = lines.drop(1)
              parse(dropped,
                    state.restart,
                    LineState(dropped.count(isHeader) > dropped.count(isTrailer), 
                              dropped.size))
            }

          case first +: rest =>
            if (!state.openedSection) fail("A header is missing")
            else {
              val (first, rest) = lines.span(line => !isTrailer(line))
              emit(first) fby
              parse(rest, state.addLines(first.size), state.addLines(lines.size))
            }

          case Vector() => halt
        }
      }
    }
    // initialise the parsing expecting a HEADER
    linesParser(LineState())
  }


  private def fail(message: String) = Halt(new Exception(message))
  private def isHeader(line: String) = line.startsWith("HEADER|")
  private def isTrailer(line: String) = line.startsWith("TRAILER|")

The bulk of the code is the validateLines process which verifies the file structure:

• if the first line of this chunk is a HEADER the next line needs to be skipped, we know we opened a new section, and we feed the rest to the lines parser again. However we fail the process if we were not expecting a HEADER there

• if the first line of this chunk is a TRAILER we do something similar but we also check the expected number of lines

• otherwise we try to emit as many lines as possible until the next HEADER or TRAILER and we recurse

This is a bit complex because we need to analyse the first element of the chunk, then emit the rest and calculate the new state we will have when this whole chunk is emitted. On the other hand the processor is easy to test because I don't have to read or write files to check it. This would be a bit more difficult to do with the loop version.

But unfortunately not all the tests are green. One is still not passing. What if there is no ending TRAILER in the file? How can I raise an exception? There's no process to run, because there are no more lines to process! My test is pending for now, and I'll post the solution once I have it (maybe there's a smarter way to rewrite all of this?).

Is it worth it?

This was definitely worth it for me in terms of learning the scalaz-stream library. However in terms of pure programmer "productivity", for this kind of requirements, it feels like an overkill. The imperative solution is very easy to come up with and there is no problems with performances. This should change once streaming parsing is available (see the roadmap). Probably this use case will just be expressed as a one-liner. In the light of this post I'm just curious how the implementation will deal with chunking.