Pages

12 October 2011

Counting words - part 2

In this small post, I'm going to show how I used Functional Reactive Programming (FRP) to improve the GUI part of the small application presented in the previous post.

The best article to read on the subject is Deprecating the Observer Pattern (DOP). This article explains what are the drawbacks of using a listener-based approach to components communication in a user interface and proposes FRP as an alternative.

Be reactive

What I've been using is a simplified yet powerful version of the code described in DOP, using the Reactive library by Naftoli Gugenheim. In this library you have 2 essential concepts: EventStreams and Signals:

  • EventStream[T] is a collection of values of type T happening at certain moments in time
  • Signal[T] is a value of type T which can change from time to time

They are actually 2 sides of the same coin (more or less) as explained in this video: from an EventStream you can get a Signal by holding the last emitted value and from a Signal you can get an EventStream by getting all the changes.

The great thing about an EventStream (let's focus on that one for now) is that you can manipulate it like a collection of objects:

  • you can filter it to get only the events you're interested in
  • you can map the events to other events
  • you can flatMap with a function returning another EventStream and so on,...

But how is this helpful for GUI programming?

With GUI components

The hard reality of Swing GUI components is that they are really mutable at heart. Once you compose a GUI with components (say a Frame) containing other components (say a TextField), then, when anything happens in your application, you mutate the innermost components heavily (by changing the text color for example). When you add publish/subscribe mechanisms on top of that, you add even more developer-managed mutability since you need to add-remove listeners to the whole graph of components. It is also not very easy to understand how events "flow" between components.

The way I see the usage of FRP with GUIs is:

  • components are seen as either creating values to propagate (like a button action) or consuming values (like a text field change with the new value). Hence they have a role of an EventStream source or of an EventStream sink (sometimes both)

  • those components explicitely declare the abstract type of streamed events they're expecting. For a given TextField this might be something like NewSelectedFile whether the selection comes from a FileChooser or from a simple TextField

  • the event streams can be merged, filtered, mapped functionally, with no side-effect so that the logic of events routing is very composable and explicit

Let's see that more precisely in the context of my simple application.

An example

In my WordCount application, the first thing the user does is to select a file to count. Then she is supposed to click on the "Count" button to start the count before the results are displayed. In terms of graphical components I have:

    val openFileMenuItem = OpenFileMenuItem("./src/test/resources")
    val countAction      = CountAction()

What you don't see above is that those 2 custom GUI components have been declared as extending EventStream[T] (simplified code for the explanation):

    OpenFileMenuItem ... with EventStream[File]
    CountAction      ... with EventStream[Unit]

This means that when you open a file using the OpenFileMenuItem you're providing new File events which other components can react on, and when you invoke the CountAction you, well,... you just pressed the button, there's no meaningful value to convey, so the Unit type is appropriate here (the clients just want to know that something happened).

Then we can compose those 2 EventStreams:

  val filePath: Signal[String]  = openFileMenuItem.map(_.getPath).hold("")
  val doCount: EventStream[Any] = filePath.distinct.change | countAction

First we do a bit of filtering, because we just need the file path as a Signal[String] (using hold to transform the stream to a signal, with an empty initial value).
Then we declare that we need to do a count whether there's a distinct change in the file path value, or (|), if the user pressed countAction button.

How do we "consume" this doCount stream of events? We flatMap it to another stream of events providing the results of the counting:

 val results: EventStream[Results] =
   doCount flatMap { doIt => WordCounter.count(filePath.now).inBackground }

For each doCount event we flatMap it (actually we forget about it,...) and we use the current value of the filePath to count the number of words.

The expression computeValue.inBackground computes a value using the SwingUtils.invokeLater method to avoid computations being done on the event dispatch thread (this might cause grey screens). The inBackground method returns an EventStream[T] to signal consumers that the value is ready.

Since the result of counting is an EventStream[Results] I can then "plug" it into the GUI component doing the display:

  val resultsPanel = ResultsPanel(results)

And that's it. The ResultsPanel component doesn't care where the values come from, who created them. It is also interesting to see how the ResultsPanel declares its sub-components:

  object ResultsPanel {
    def apply(results: EventStream[Results]) = {
      new ResultsPanel(TotalWordsNumbers(results),
                       ReferencesTable(results.map(_.references)),
                       ErrorMessageBox(results.map(_.message)))
    }
  }

There are 3 sub-components and they use different parts of the Results event stream, so we use the map function to restrict exactly the stream to what is needed:

  • TotalWordsNumbers uses the full Results object to display the total words count (the one we really want), the references word count and the full text word count (to check that the count is ok)

  • ReferencesTable just needs the references

  • ErrorMessageBox needs the error message so we just map that

Finally, how is the event stream used in the component itself? Let's look at the ErrorMessageBox component:

  case class ErrorMessageBox(message: EventStream[String]) extends TextField with Observing {
    foreground = Color.red
    font = new Font("Courier", Font.PLAIN, 15);
    editable = false

    message.foreach { m => text = m }
  }

The important line is the last one where we declare that for each message event, we change the text attribute of the TextField to the new value m.

Some implementation notes

If you read the actual code, you'll find quite a few differences with the real implementation:

  • the inBackground mechanism is enhanced with an additional actionProgress eventStream which fires events before and after the action to execute. This is used to change the main frame cursor to a waiting watch when the computation takes some time

  • I found useful to introduce a trait called Trigger for EventStream[()]

  • I also added an EventStreamSourceProxy[T] extends EventStream[T] trait which can be mixed in any GUI class. This trait uses an internal EventSource[T] which is the implementation of the EventStream[T] and can be used to fire events. For example:

    // When the action is executed (a file is selected) we fire an event and
    // the whole `OpenFileMenuItem` component acts as an `EventStream[File]`.
    case class OpenFileMenuItem(start: String) extends MenuItem("Open") with EventStreamSourceProxy[File] {
      ...
      action = new Action("Open") {
        def apply() {
          ...
          source.fire(fileChooser.selectedFile)
        }
      }
    }
    
  • I had some difficult debugging time with the Observing trait. If you place it on an object that's going to be garbage collected, your components will not be notified of new events. This happened to me as I placed it on a class used for an implicit conversion, in order to get a new shinyMethod (tm):

    implicit def toComponent(a: A): ImplicitComponent = new ImplicitComponent(a)
    class ImplicitComponent(a: A) extends Observing {
      def shinyMethod = a
    }
    

Features ideas

This is something which amazed me: just having to think about event streams made me rethink the application functionalities. How? When I started thinking about what would trigger a word count I realized that:

  • there is no reason why the user should have to click the "Count" button when the file is selected, we can do the count and display the results right away

  • thinking about event stream as a flux made me realize that the file can be polled regularly for changes and the results displayed whenever there's a change

Changing my program to incorporate those 2 ideas was soooo easy:

  1. the first idea is reflected by the doCount event stream definition given above, we just say that we want to count whenever there's a file selection or a count action

    val doCount: EventStream[Any] = filePath.distinct.change | countAction
    
  2. creating a file poller is easy using the Timer class from the reactive library

    class FilePoller(path: Signal[String], delay: Long = 500) extends Trigger {
    
      private var previousLastModified = new File(path.now).lastModified()
    
      val timer = new Timer(0, delay, {t => false}) foreach { tick =>
        def newLastModified = new File(path.now).lastModified()
        if (newLastModified > previousLastModified || newLastModified == 0) {
          previousLastModified = newLastModified
          source.fire(())
        }
      }
    }
    
    The `FilePoller` uses a `path` signal regularly. If the underlying file is modified, a notification event is triggered.
    
  3. then the final version of doCount becomes:

    lazy val filePoller                = new FilePoller(filePath)
    lazy val doCount: EventStream[Any] = filePath.distinct.change | countAction | filePoller
    

I don't know about you but I really find nice that the abstractions in my implementation give me hints about what the application could do! For me this is a good sign that FRP is really well-suited for the job of GUI programming.

11 October 2011

Counting words

In this 3 parts post I want to show:

  1. how standard, out-of-the-box, Scala helped me to code a small application
  2. how Functional Reactive Programming brings a real improvement on how the GUI is built
  3. how to replace Parser Combinators with the Parboiled library to enhance error reporting

You can access the application project via github.

I can do that in 2 hours!

That's more or less what I told my wife as she was explaining one small problem she had. My wife is studying psychology and she has lots of essays to write, week after week. One small burden she's facing is keeping track of the number of words she writes because each essay must fit in a specific number of words, say 4000 +/- 10%. The difficulty is that quotations and references must not be counted. So she cannot check the file properties in Word or Open Office and she has to keep track manually.

For example, she may write: "... as suggested by the Interpretation of Dreams (Freud, 1905, p.145) ...". The reference "(Freud, 1905, p.145)" must not be counted. Or, "Margaret Malher wrote: "if the infant has an optimal experience of the symbiotic union with the mother, then the infant can make a smooth psychological differentiation from the mother to a further psychological expansion beyond the symbiotic state." (Malher cited in St. Clair, 2004, p.92)" (good luck with that :-)). In that case the quotation is not counted either and we must only count 3 words.

Since this counting is a bit tedious and has to be adjusted each time she does a revision of her essay, I proposed to automate this check. I thought "a few lines of Parser Combinators should be able to do the trick, 2 hours max". It actually took me a bit more (tm) to:

  • write a parser robust enough to accommodate for all sorts of variations and irregularities. For example, pages can be written as "p.154" or "pp.154-155", years can also be written "[1905] 1962" where 1905 is the first edition, and so on

  • use scala-swing to display the results: number of words, references table, file selection

  • write readers to extract the text from .docx or .odt files

Let's see now how Scala helped me with those 3 tasks.

Parsing the text

The idea behind parser combinators is very powerful. Instead of building a monolithic parser with lots of sub-routines and error-prone tracking of character indices, you describe the grammar of the text to parse by combining smaller parsers in many different ways.

In Scala, to do this, you need to extend one of the Parsers traits. The one I've choosen is RegexParsers. This is a parser which is well suited for unstructured text. If you were to parse something more akin to a programming language you might prefer StdTokenParsers which already define keywords, numeric/string literals, identifiers,...

I'm now just going to comment on a few points regarding the TextParsing trait which is parsing the essay text. If you want to understand how parser combinators work in detail, please read the excellent blog post by Daniel Spiewak: The Magic behind Parser Combinators.

The main definition for this parser is:

  def referencedText: Parser[Results] =
    rep((references | noRefParenthesised | quotation | words | space) <~ opt(punctuation)) ^^ reduceResults

This means that I expect the text to be:

  • a repetition (the rep combinator) of a parser

  • the repeated parser is an alternation (written |) of references, parenthetised text, quotations, words or spaces. For each of these "blocks" I'm able to count the number of words. For example, a reference will be 0 and parenthetised text will be the number of words between the parentheses

  • there can be a following punctuation sign (optional, using the opt combinator), but we don't really care about it, so it can be discarded (hence the <~ combinator, instead of ~ which sequences 2 parsers)

Then I have a function called reduceResults taking the result of the parsing of each repeated parser, to create the final Result, which is a case class providing:

  • the number of counted words
  • the references in the text
  • the quotations in the text

Using the RegexParser trait is very convenient. For example, if I want to specify how to parse "Pages" in a reference: (Freud, 1905, p.154), I can sequence 2 parsers built from regular expressions:

  val page = "p\\.*\\s*".r ~ "\\d+".r
  • appending .r to a string returns a regular expression (of type Regex)
  • there is an implicit conversion method in the RegexParsers trait, called regex from a Regex to a Parser[String]
  • I can sequence 2 parsers using the ~ operator

The page parser above can recognize page numbers like p.134 or p.1 but it will also accept p134. You can argue that this is not very well formatted, and my wife will agree with you. However she certainly doesn't want to see the count of words being wrong or fail just because she forgot a dot! The plan here is to display what was parsed so that she can eventually fix some incorrect references, not written according to the academia standards. We'll see, in part 3 of this series how we can use another parsing library to manage those errors, without breaking the parsing.

One more important thing to mention about the use of the RegexParsers trait is the skipWhitespace method. If it returns true (the default), any regex parser will discard space before any string matching a regular expression. This is convenient most of the time but not here where I need to preserve spaces to be able to count words accurately.

To finish with the subject of Parsing you can have a look at the TextParsingSpec specification. This specification features a ParserMatchers trait to help with testing your parsers. It also uses the Auto-Examples feature of specs2 to use the text of the example directly as a description:

  "Pages"                                                                           ^
  { page must succeedOn("p.33") }                                                   ^
  { page must succeedOn("p33") }                                                    ^
  { pages must succeedOn("pp.215-324") }                                            ^
  { pages must succeedOn("pp.215/324") }                                            ^
  { pages("pp. 215/324") must haveSuccessResult(===(Pages("pp. 215/324"))) }        ^

Displaying the results

The next big chunk of this application is a Swing GUI. The Scala standard distribution provides a scala-swing library adding some syntactic sugar on top of regular Swing components. If you want to read more about Scala and Swing you can have a look at this presentation.

The main components of my application are:

  • a menu bar with 2 buttons to: select a file, do the count
  • a field to display the currently selected file
  • a results panel showing: the number of counted words and the document references

wordcount application
count example, note that the parsing is not perfect since the word counts do not add up!

If you have a look at the code you will see that this translates to:

  • an instance of SimpleSwingApplication defining a top method and including all the embedded components: a menu bar, a panel with the selected file and results
  • the subcomponents themselves: the menu items, the count action, the results panel
  • the "reactions" which is a PartialFunction listening to the events created by some components, SelectionChanged for example, and triggering the count or displaying the results

I was pretty happy to see that much of the verbosity of Swing programming is reduced with Scala:

  • you don't need to create xxxListeners for everything
  • there are components providing both a Panel and a LayoutManager with the appropriate syntax to display the components: FlowPanel, BoxPanel, BorderPanel
  • thanks to scala syntax you can write action = new Action(...) instead of setAction(new Action(...))

This is nice but I think that there is a some potential for pushing this way further and create more reusable out-of-the-box components. For example, I've created an OpenFileMenuItem which is a MenuItem with an Action to open a FileChooser. Also, something like a pervasive LabeledField with just a label and some text would very useful to have in a standard library.

I also added a bit of syntactic sugar to have actions executed on a worker thread, instead of the event dispatch thread (to avoid grey screens), using the SwingUtilities.invokeLater method. For example: myAction.inBackground will be executed on a separate thread.

Eventually, I was able to code up the GUI of the application pretty fast. The only thing which I didn't really like was the Publish/React pattern. It felt a bit messy. The next part of this series will show how Functional Reactive Programming with the reactive library helped me write cleaner code.

Reading the file

I anticipated this part to be a tad difficult. My first experiments of text parsing were using a simple text file and I knew that having the user (my wife, remember,...) copy and paste her text to another file just for counting would be a deal-breaker. So I tried to read .odt and .docx files directly. This was actually much easier than anything I expected!

Both formats are zipped xml files. Getting the content of those files is just a matter of:

  • reading the ZipFile entries and find the file containing the text

    val rootzip = new ZipFile(doc.path)
    rootzip.entries.find(_.getName.equals("word/document.xml"))
    
  • loading the xml as a NodeSeq

    XML.load(rootzip.getInputStream(f)))
    
  • find the nodes containing the actual text of the document and transform them to text

    // for a Word document text is under <p><t> tags
    (xml \\ "p") map (p => (p \\ "t").map(_.text) mkString "") mkString "\n"
    

For further details you can read the code here.

Recap

That's it, parser combinators + scala-swing + xml = a quick app solving a real-world problem. In the next posts we'll try to make this even better!