Less is more, slow is fast (expensive is cheap?)

Dear Joel,...

Here's a letter I just received from Joel Spolsky's company, FogCreek Software. One year and a half later, they've sent me, for the second time, a DVD I had ordered. Good customer service, you would say: my first package was lost in the wild so they graciously sent me another one. But I really didn't expect that they would resend me the initial DVD that was miraculously returned to them after more than a year! Is that exceptional service or what?

Why did they do that? I was already pretty much satisfied the first time when they quickly sent me another DVD after the first one was lost (I've even heard about some companies doing that on purpose, just to show how good their support was afterwards). But what could they gain by over-satisfying me? Well, I assume, they just didn't miss that wonderful opportunity to show me how much they actually care about me. Not that I felt so miserable and alone on earth, but I was really touched by the attention. Now I really can say that this company has something special.

Ok, ok. How can they afford that? How can they sustain a double-digit growth each year by wasting so much time? Well, they simply don't ("Fog Creek is focused on growing slowly and carefully and staying profitable"). However if they pay so much attention to each of their customer, which I think they do, they are certainly here to stay.

Less is more

There is some kind of "less is more" pattern here. Less market growth, conquest, expansion,... but also more satisfied customers, upgraded installations, license extensions (anyway, Joel thinks that it takes ten years). And guess what, I think that the people working at the customer service are happier too. They certainly prefer 1000 times receive congratulations than coping with angry clients.

In a sense, this is also a "political" decision. Deliberately choosing to do less but to do better shows how you care for others: your customers, your co-workers. It is some kind of "idealist" choice. But is it also a "pragmatic" choice? Is it also a good economic choice for your company, especially for a software company (yes, I don't speculate often about fashion,...)?

Just wondering

Unfortunately, I don't have an answer with elaborate studies. But I am wondering,...

1. Are there times when you should choose "speed" against "quality"?

From the moment you select a feature to implement in your next release what do you get from delivering it faster but buggy? One more customer, of course! The one who is craving for this new feature. The one who's considering that not having it is a deal-breaker, the one,... who will be so mad to discover that it doesn't work!

[Unless he's just a golf player, he's a close buddy of the CEO, and he doesn't care so much as long as your software is buzzword-compliant. But this is another discussion]

Even when prototyping a new idea, we should be careful with "speed", because the frontier between "rapid prototyping" and "semi-chaos" is pretty thin (not to talk about the frontier between "prototype done" and "let's call it 1.0"?)

2. Is it so expensive to focus on the user-interface?

Yes, this can be pretty expensive. You may have to hire a usability engineer for the sole purpose. And I say that because I know what's the big gap between the half-baked interfaces I will ever produce and the interfaces that are carefully crafted by professionals. But this is not only for the beauty of the game. How many hours are going to be lost by: your helpdesk, your consultants, your newly-hired engineers, your partners? Oh, yes, and your customers,...

3. Why not set-up an "aggressive plan"?

This one really makes me wonder,... I guess that for any project, if you could start it over a thousand times, you may discover the most efficient way to order and parallelize activities so that the least amount of time is lost. But you can't do better, can you?

What are the variables you can play with when starting a project?

1. deadlines
2. quantity of people
3. motivation
4. quality of people
5. customer expectations

So far, an "aggressive plan":

1. fixes deadlines using ballpark estimates of the project estimates and insists that the target can't be missed. Time lines being then the only success criteria, this usually compromises any chance of success before the project even begun.

2. throws in between a moderate amount of people ("we can do it") and a big crowd ("we'll do the maximum")

There are strong suspicions that throwing more people on a project can only make it worse (Brooke's law and this study). Anyway, I am not sure there's much "aggressiveness" to have here. The "right" number of people will do. The good question is not: "is it aggressive enough?" whatever it means, but "are we properly staffed for this?"

3. motivates highly everybody

By either: insisting on the "heroic" aspect of the venture or reminding that "aggressive" concerns each one personally,...

4. tries to gather talented individuals,..

As far as 2 or 3 weeks allow it (because the preparation phase was also "aggressive")

5. decides bravely to deliver only one single feature.

No, just kidding!

Quality, ergonomic user-interfaces, sustainable project pace and even slack, why not go slow?

"less is slow, slow is fast, fast is more" (or the other way around ;-) )

One place where it makes the most sense in certainly when coding:

• less is slow: you need time to find the right abstractions, the proper structure, the proper names. You need time to remove the unessential (the famous StExupery word about perfection)
  

• slow is fast: you worked hard to produce less, but you get more value because code is a liability, and less allow you to move things around faster
  

• fast is more: there is a tipping point when you can add much more ideas to your software because you just play with the right legos

Why does it sound so paradoxical? A is ContraryOf(A). I believe strongly this is a matter of perceptions.

Perceptions

Try this at home. As an experiment:

• crank out code (I mean "crank" if you don't religiously follow the red-green-refactor mantra)
• take time to refactor it
• add some meaningful comments (the why, not the how)
• add some user doc (it can be a document or a few comments in a header)
• take note of the additional time taken
• forget all that for a few months (2 will usually do)
• try to estimate how much time was saved -just for you- when you read that code again

Seems simple and obvious, so why don't we do that all the time? Because we're so focused on the here and now that we don't perceive the poor chap that will read the code in 2 months. He's not here now, to cry in anger [This is why pair programming helps. He's sitting next to you,...]

Same idea, different context: why don't we do TDD all the time? Because we don't perceive the difficulty of what we're doing. If you'd realize that each line of code can bear many different and subtle bugs, you would just want to secure it before coding the rest. [And there are plenty of other benefices too].

I am ashamed to confess this, but 2 days ago I wrote a simple "Table" class, taking a list of n row names, a list of m column names and an array of n x m values. I had to create a method that returned the value for a given row name and column name. Can you do it with you eyes closed? Good for you. I was inspired enough to write a few specs before using the class,...

Those are 2 examples showing that we should be careful with our perceptions and ask ourselves:

• Can we really "not afford" to refactor?
• Can we really "not afford" to give more time to a project?
• Can we really "not afford" to hire a usability engineer?
• Can we really "not afford" to look for better tools and practices?
  

My theory is that the reality of what we do is so complex that we can be easily fooled by our perceptions [Or we choose to see things this way!]

Higher-order ideas

In my next post, I'll go slow again. I'll have a look at a language I just started learning. Yes, there's going to be a substantial learning curve, desperate evenings just trying to understand why this f*****g line doesn't work. But I am pretty sure this will pay off, allowing me to get more with less, to go fast because I was slow.

I've chosen Scala as my "Programming language of the year". Scala is not even in the 50 most popular programming languages, but I think it really embodies higher-order ideas, ideas that let you express other ideas with less words, doing more with less.

Stay tuned for the "Scala to Heaven"!

Net pearls

Reading tons and tons of internet stuff, I often find some posts or some documents I find interesting or funny: some net pearls.

As usual I would like to share this with the rest of the world and present those collected pearls as a post.

Pearl n. 1

This historical document by Dr. Winston Royce presents the waterfall model. The interesting sentence in this document is:

I believe in this concept, but the implementation described above is risky and invites failure.

This line must not have been quoted very often by waterfall proponents,...

Pearl n. 2

The next pearl comes from a study of the productivity of several languages. This story is interesting because it compares the use of different types of languages, static or dynamic, through different axes: productivity, length, memory consumption, speed,... Whatever my love for programming languages, the pearl is in the following conclusion:

Interpersonal variability, that is the capability and
behavior differences between programmers using the
same language, tends to account for more differences
between programs than a change of the programming
language.

Pearl n. 3

This one is both funny and interesting (read the whole thread): do we need a "might-equal" operator?

How about adding a "might-equal" operator?  I recommend useng
 the characters "=?".  I hav implemented my hone programming 
language, incubating the might-equal operater.  It works 
somthing like this:

if (x =? y)
  print ("x might equal y");
else
  print ("x might not equal y");

Acording to my studys, the might-eqqul operatir is 299834%
 more useful than the standerd ekwal operator.  For
 testing porposes, I have usd the mighty-kwal operatyr 
in severul custom sofware packages, including spell 
chex and statistics soffy-soff.  I hope that you ull can 
make yous of it in yore own pergroomink lunkishes. 

Pearl n. 4

This presentation from Michael Feathers introduces some coaching patterns and suggests visualizing the group as an individual, "Pat", even if one-on-one action is the privileged interaction medium:

‘Pat’ does not embody an organizational goals, ‘Pat’ is an amalgam of the team

Pearl n. 5

A little bit of history here. Here's a snapshot of notes taken during the meeting that led to the Agile Manifesto.

Pearl n. 6

Acceptance tests is one area that is still not very well covered by software vendors, despite their great usefullness (in terms of executable specifications). This may not be true anymore with Greenpepper software.

Pearl n. 7

Do you like programming quizzes? Here's a nice puzzle from John McCarthy (Lisp's inventor), solved with Haskell:

We pick two numbers a and b, so that a>=b and both 
numbers are within the range [2,99]. We give Mr.P 
the product a*b and give Mr.S the sum a+b. The 
following dialog takes place:

Mr.P: I don't know the numbers
Mr.S: I knew you didn't know. I don't know either.
Mr.P: Now I know the numbers
Mr.S: Now I know them too

Can we find the numbers a and b?

This puzzle is solved by encoding the facts in a few statements.

Pearl n. 8

You can still do funny stuff with Java! Check this out and turn the page.

Pearl n. 9

What's hot, what are the next technologies you should keep an eye on? This conference program proposes some answers:

• Services (SOA, web services, composition, mashup)
  
• Web frameworks (Rails, Tapestry) and Ajax
• Application frameworks (Spring, OSGi)
• Build systems (Maven) and automation
• AOP
• Open-source (use, participation)
• Collaborative, open webapps (Web 2.0)
  

Pearl n. 10

I like concrete examples when it comes to describe concurrencies issues, the next big question in our industry (not in the previous list, which also shows that is it far from complete). But fortunately, humans do distributed computing!

Bob. Alice calls Bob: "Could you get me those numbers?"

Bob jots Alice's request on his to-do list. "Sure thing, Alice, I promise I'll get them for you after I solve this engineering problem."

Bob has handed Alice a promise for the answer. He has not handed her the answer. But neither Bob nor Alice sits on their hands, blocked, waiting for the resolution.

Rather, Bob continues to work his current problem. And Alice goes to Carol, the CFO: "Carol, when Bob gets those numbers, plug 'em into the spreadsheet and give me the new budget,okay?"

Carol: "No problem." Carol writes Alice's request on her own to-do list, but does not put it either first or last in the list. Rather, she puts it in the conditional part of the list, to be done when the condition is met--in this case, when Bob fulfills his promise.

Conceptually, Alice has handed to Carol a copy of Bob's promise for numbers, and Carol has handed to Alice a promise for a new integrated spreadsheet. Once again, no one waits around, blocked. Carol ambles down the hall for a contract negotiation, Alice goes back to preparing for the IPO.

When Bob finishes his calculations, he signals that his promise has been fulfilled; when Carol receives the signal, she uses Bob's fulfilled promise to fulfill her own promise; when Carol fulfills her promise, Alice gets her spreadsheet. A sophisticated distributed computation has been completed so simply that no one realizes an advanced degree in computer science should have been required.

Conclusion

Here we are. I didn't mean at all to reach 10 pearls exactly but this came to be precisely the number of pearls I wanted to share. This must be a proof of the profound wisdom they embody.

See you for Net pearls 2! (Sooooo handy when I have nothing to say ;-) )

Pattern matching with Ruby

How do you know you got it all in your toolbox? Short answer: you never have it all. Long answer: you may add more, a lot more,...

I was reading one of Steve Yegge's old posts lately: "Choosing languages". This article introduced me to the pattern matching capabilities of a language such as Haskell. Wow,... Yes we can't do that in Ruby. But can't we really?

Indeed, this triggered my memory like a mantra: "if it's not nailed down, steal it... if it's not nailed down, steal it,..." Here's the article: If it's not nailed down, steal it. The author presents a "theft" in Ruby: the implementation of pattern matching.

So using the 'multi' gem, you are able to express the Fibonacci function as:

multi (:fib, 0) { 1 }
multi (:fib, 1) { 1 }
multi (:fib, Integer) { |n| fib(n-1) + fib(n-2) }

This may not be very impressive here, but later on in Steve's article, Steve tells us that pattern matching can really be a huge win when dealing with complex structured data. Here's an example from a contest in the Amazon's developer journal.

=============================================================

bad [[_,fs],[_,ds],[_,cs],[_,gs]] = (cs /= fs) && (cs == ds || cs == gs)

Here, the bad function gets a list as its argument. But instead of naming the list, it specifies a pattern that it expects the list to match: a list of 2-item sublists. It ignores the first element of each sublist, and assigns the second element to fs (farmer side), ds (dog side), etc. The variables then become available for use in the body of the function, after the "=" sign, where we check if if the chicken's alone with the dog or the grain.

=============================================================

For sure, writing the same thing in Java, or even in plain Ruby would be a pain. So I rolled up my sleeves and tried to enhance the 'multi' gem with the ability to use Arrays and variables in Arrays. So I allowed to write things such as:

multi(:foo, [String, Integer]) {|x, y| x; y}
multi(:foo, [:p1, :p2])  { |symbols| symbols[:p1]; symbols[:p2]}
multi(:foo, [:p1, [Integer, :p3]]) { |symbols, x| symbols[:p1]; symbols[:p3]; x}
multi(:foo, [:p1, [:_, :p2]]) { |symbols| symbols[:p1]; symbols[:p2]}

I also added some support for a 'multi' block, which is closer to a Haskell definition (though the 'let' name will be certainly shock a lisper,...):

multi(:foo) do
let(String) {|x| x.should eql "hello"}
let(String, String) {|x, y| x.should eql "hello"; y.should eql "world"}
end

My conclusion on that little experiment was:

• The use of a "symbols" map is a bit awkward, even if it uses the symbols declared in the pattern. It looks like a trick compared to a Haskell definition
  
• It is possible to reproduce Steve's examples in an easy way (I didn't say efficient!)
• The limitations mentioned in the article regarding the use of 'multi' methods in classes still hold: they are not inherited. 'multi' is not the 'def' keyword
  

What I am really interested in is: will I use it? Steve says that once you've tasted Pattern matching, you start seeing it everywhere (design patterns, anyone?). I still don't see it in my daily java programming, nor in my nightly ruby programming.

Now, let's say I start seeing it ("Yes, I got it! I am in the Matrix!!"). Should I use it in my ruby code? And should I use the rest? Let's see:

• we can do some pattern matching in ruby
• we can compose functions
• we can have lazy lists (and list comprehension)
• we can use statemachines
• we can do high-order messaging: [0, 1, 2].where.nonzero? => [1, 2]


Is it still Ruby? Will other people still understand what I write?

Hey, why not? I am truly amazed by the Ruby's capacities to be benefit from ideas from other languages or paradigms. Indeed, I think it still feels quite Ruby-like with functions and blocks, and those features allow me to write more concise code, at the right abstraction level.

However, capturing an animal and putting him in a zoo is not the same as watching him in its environment. Haskell, Lisp, Erlang and others are quite unique and they each provide different ways to blow your mind.

By the way, is it possible to do concurrency programming "a la" Erlang in Ruby? The challenge is opened,...

StrengthFinder: discover your talents, turn them into strengths!

I was the great show opener with my Spiderman costume

When I was 10 years old, my class was organizing a small theater play, based on the novel "Colombine" which is a variation on the famous characters from the Comedia del' Arte.

I have always been fascinated by stage action but this time, this was special. Catherine, my secret-beloved-classmate was playing the role of Colombine and she was the most beautiful girl ever in her white dress.

The other central character of this comedy was Arlequin, a joyful, carefree, colorful character. Arlequin entertains Colombine, charming her with his tricks and pleasant stories. I would have loved to play this role, the role of my life for sure, but it had been given to Gilles,... my best friend.

Well, I had accepted my fate, especially since Gilles was 1. a very good Arlequin, 2. my best friend. Until the day Gilles was ill at home,...

He can't rehearse the role! "Pleaaaaaase, let me do it!!!!". Yes! Here I go, ready to seduce Catherine with all my acting talents. Helas, after a few tries, everybody sees that I am not at all the same little extravagant boy than Gilles and that my performance is, how would I say? Well, average,...

Realizing on stage that I will never be able to seduce Colombine/Catherine by being her beloved Arlequin, I burst into tears.

Then, my teacher has the kind of idea that makes a great teacher. She acknowledges that I love being on stage, that I am a serious boy, engaging to other people and that I prefer intimacy to extravagance. So she just decides to create the perfect role for me, building on my talents. I'll be presenting the play!

So, on the big night, I sat under the lights, opening the "Colombine" book, reading an introduction that created the warm feeling of entering a formidable romantic story. Even if having a Spiderman costume, the only one left, must have left some parents wondering,...

Do your best or do -your- best?

This teacher had the great instinct to let me do something I was good at and that motivated me a lot, creatively finding a way to use my talents. She turned something which could have been a traumatic experience into a success story.

This story is like the piece of a puzzle for me. Last year, I made the semi-conscious decision that I should definitely do what I was good at. Why would I try to be a good project manager, a salesperson, a coach, an entrepreneur, a PR, a professional services leader, a top-executive if I can't give -my- best at it. Not my best efforts, but my own best, the things which make me very different from anybody else. Deciding this is both relieving and very energizing. This is the second piece of the puzzle.

Rely on other's strengths, they love that!

I also remember 2 years ago how I realized that I was much more successful when letting others use their own specific talents without trying to do all by myself. Severine is good at organizing meetings, she can do it! Rely on other's talents when you lack some. They will even be grateful that you let them do what they love and what they're successful at!

No wonder why I am always asking candidates what they like to do,... This is the third piece of the puzzle.

The online test

Then, starting a new job this year, I met my project manager, freshly arrived in the company, too. I was very impressed by the way he had been handling things and connecting to people from day 1. Having read "Behind closed doors: the secrets of great managers" last year, I thought: "He's really acting by the book!" (Kaz, you are allowed to blush if you read this blog,...).

Kaz told me about an idea I had discovered last year, when working with an external HR cabinet in my previous company: improve people's strengths, not their weaknesses. This idea was really appealing but also very perturbing (more on that further down).

So Kaz wants people in his team to pass a "StrengthFinder" test. This test is provided by the Gallup institute and aims at finding the 5 top strengths amongst 34 possible strengths.

I passed the test and the first thing I noticed having read the results was: "Hey, that's the first test I cannot feel bad about!". The test is telling me: here, here,... and here, you're really great!

And what about the rest? Is there something I should improve, work on, be wary of? NO, forget it. Use all your energy maximizing your own existing talents. Not the ones you may possibly acquire on the edge of exhaustion.

This makes hell of a difference, creating a nice little psychological "click" in your head,...

Me, you, our team

I don't want to paraphrase the book, it says a lot more that I do on the subject but here are some personal and team consequences:

• It is more than ok to rely on others strengths. This has many positive side effects: work is done ok, people love doing that, you are relieved of what you don't like
  
• Ÿ It is very important to think about your team in terms of provided strengths because this is your only building material for your project. You're not going to build on their weaknesses!
• Ÿ You can also think about the strengths you're looking for when recruiting, to complement those of your team
• Ÿ There's no better way to set yearly objectives: people have more chances to be successful, they will bring more value to the company, they will enjoy doing their job
• Ÿ "StrengthsFinder" is in fact more like "TalentsFinder". Now I know what are my talents, I still need to grow them to turn them into real strengths. Even Mozart had to practice violin,...

Now, this little "click" is so important that I need to talk about the thoughts that can sabotage it.

Preserve the "click"

It is very interesting to start thinking about why we can hear some kind of big warning when thinking about this "strengths-only" approach. Very nice, but,... :

1. Your defaults can ruin your life/project if you forget about them

2. You can always improve

3. You should always improve (it's a matter of will)

4. Life cannot be a bed of roses where you just do what you like

5. Why would I forget about trying to be a great Commander?

Let's take these points one by one:

1. Well, you don't have to totally forget about them! You can for instance use personal strategies to mitigate most of the side-effects. You're very creative but incapable to think about paying your bills? Add an Outlook recurrent meeting to your calendar, buy a Palm pilot, arrange an automatic transfer, put a sticker on your fridge,... Whatever. Just do the little thing that will prevent disasters. Don't try to turn into a "Logistic man/woman". But wait do better than that: find someone that will be delighted to do that for you! (Is that what marriage is all about ;-) ?)

2. Yes, you -may- improve, but is it worth the energy and, most of the time, the unpleasant feeling worth it? Why don't you use this energy to be even better and successfull in what you're already great at? You prefer to hear: "s/he's incomparable" or "s/he's trying hard"?

3. This is more like a moral argument whose psychological roots I leave to the reader to find out,...

4. Why not? By carefully selecting who you work with, you may just find in others the strengths that makes your life great. Chasing receipts and checking amounts to the cent is not considered to be anyone's preferred activity. Yet, I've met people just loving to do that!
5. I like this one,... Well, because you can't have your cake and eat it. By focusing on your strengths, you're great but you give up being a super-heros capable of anything. I will never be Arlequin but I can be a super-presenter,...

My top 5

To finish with, here are my Top 5 strengths:

• RELATOR
• STRATEGIC
• IDEATION
• LEARNER
• ACHIEVER

From what I know, this is pretty accurate and provides a solid starting point for improvement.

Now, why don't you take the test and drop me a line about how it feel passing it!

Disclaimer: To do the test, you have to buy the book. I have no interests in Gallup and there must be numerous other resources with the same approach, I am only interested in -your- little "click",...

Is Project Manager the highest lifeform?

Back after a looonnng time

I haven't kept the sacred "blog at least once a month" pace since last December because of my move to Tokyo (And I still don't have internet at home, arghh). However, here's my comeback
with a classical post about the developer vs project manager divide. I decided that an interview would be a better form of post than a long monolog and I really enjoyed doing it (beside I don't have to be the one to be smart!). You can find many more posts on the subject, including this recent one that had lots of comments, even if it's more on the lines of "everything sucks".

But I have more posts coming up:

• StrengthFinder 2.0
• Pattern matching with Ruby
• Net pearls
  

So stay tuned!

An interview for a change

I have had comments by Teki on my "Interesting software vs plumbing" post. Those comments brought us to the subject of programmer's evolution: starting from being a self-satisfied geek, why would you become a project manager?

Wanting to write a few lines on the subject, I preferred to go for an interview with one of the best software professionals I've came in touch with, my friend Olivier. [some of my comment are in brackets]

me> Olivier, it is sometimes said that a programmer should evolve to a manager position to get a satisfying job in terms of career (recognition, money, autonomy, employment), what do you think about that?

him> I don't agree! A good programmer doesn't necessarily make a good project manager. Becoming a project manager is not the only possible evolution. Technical expertise is also amongst the possibilities. It is true that most of the time, the management positions are the only ones recognized socially and financially. But It doesn't have to be that way. In my previous job, I encouraged the HR department to setup 2 different "ladders" in the company.

By the way, as a recruiter, I have a very, very bad feeling, when a programmer comes to me and says: "Hey, I want to be a project manager in 2 or 3 years". Programmer is a real job, not a transition.

me> Do you think that there could be more than one "expertise ladder"?

him> I don't know, this "ladder" thing is essentially a social and monetary issue. Most people need social recognition and it is often correlated to salary (at least in France).

me> Is it possible to be a good programmer as well as a good project manager?

him> I hope so! [me> Olivier is both ;-)]

me> Is it seldom to be good at both? Can it be learned?

him> I think it can be learned. The real difference between Mr. Universe and me is not the amount of muscle, but essentially the time spent training. Training sufficiently may not make me Mr. Universe but it may still build lots of muscles. The only question is: will I enjoy the training, am I willing to do all necessary efforts?

me> Is it possible to be a good manager without having been/being a programmer?

him> Yes, I think so. I've worked with a very good project manager, who was not a programmer at all. However, he had a real respect for my expertise and trusted my judgement. Respect is a key point in our field. It is also true for programmers who are sometimes capable of utter pretention: try a newbie message on the linux.fr mailing list for instance,...
[me> I sometimes wonder if it is possible to really have a deep understanding of programmers and software without having being one yourself]

me> Will programmers take over the world :-) and be rewarded as they should? What should they do to achieve this?

him> I don't think that any "world movement" [me> or certification program,...] can achieve that. Local action is prevalent, relentless communication is my only tool. I try to communicate whenever possible: to programmers, QA, directors, sales, consultants. Communication has to be relentless to make people realize the value of programmers.

However, the recognition of the importance of programmers will continue to rise as the upper management realizes their creativity is needed and how much turnover affects the company.

me> Why is programmers perceived added value so low?

him> Their added value? Is it really so low? Yes, perceptions can be very far from reality: some people think that programming is an obscure task, reserved for some kind of mutants. Other ones (sometimes the same but at different moments!), think that programming is easy because they use Word everyday and find it intuitive, easy to use. Or, they have programmed a few Excel macros and feel it is not such a big deal. But how much complexity is hidden behind a good software? It is so hard to make it simple!

Anyway, this is unfortunately the reality for most engineers I know, from mechanical to electricians.

me> Do you think it is a Taylorist / scientist vs agile / lean issue ? What I mean is: do you think that a Taylorist view of software leads to a natural division of Commanders / Executers while an agile view of software makes it more like a collaborative / creative process ? How do you see the future to that respect?

him> Taylorism doesn't apply well to software programming. It is fundamentally a creative activity and an undeterministic process. The V-Cycle was such a big misunderstanding. It has unfortunately spread through most of development standards from administrations to banks. Driving a car is a much better analogy: you constantly have to adjust your direction, your speed and manage obstacles as they occur.

Anyway, I don't think that any agile / lean methodology will drastically change things. Whatever has been said on the subject, there are not enough total project failures. I have not seen one around me in my whole career. [me> Lucky guy, I have seen quite a few, from medium to big].

Still it is so common to expect obligation of results and not obligation of means. Can you write a best-seller in 3 months? Maybe you can write "something" in 3 months, maybe you can write a best-seller in some years, but it takes a genius to write a best-seller in 3 months. How many programming Mozarts can write a brilliant symphony for each project they start?

me> Is there a question that you would have liked to answer ;-) ?

him> The question you ask at the beginning leads us to: what does it take to be a good programmer, a good manager? Is there an universal definition of those 2 roles, or only local ones? Whatever the answer, having the 2 activities being in conflict is always a sign of a big underlying problem: lack of respect,...

Testing a complex API

We all know the value of testing, and especially the importance of unit testing. However, it is sometimes quite a challenge to properly test some pieces of code.

What about testing classes that access an API that uses complex data structures as input and output?

Testing the Eclipse Modeling Framework

One example of this is the Eclipse Modeling Framework, but I am sure you will find plenty of examples of your own.

This kind of API offers points of entry to modeling data structures such as "PackageRegistry". A PackageRegistry has "getAllPackages" method returning Package objects.

Then, each Package object can be browsed for its content: Class objects, Diagram objects, nested Packages,...

Package, Class, Diagram,... are all interfaces to a complex data structure that can be navigated in order, for instance, to generate code or to create HTML reports. But how do you test this transformation code?

An "out-of-the-box" answer is to use mock objects to create the objects accessed through the API. A fine Mock object framework such as jmock is one of the best for that purpose.

However, setting up the mock structure is:

1. extremely painful (such as in "extracting-my-teeth-one-by-one")
2. near to unreadable

Here's what it looks like:

Mock packagesRegistry = mock(PackagesRegistry.class);
List packages = new ArrayList();
packagesRegistry.expects(once()).method("getAllPackages").will(returnValue(packages));

// now create one mock package
Mock package = mock(Package.class);
List classes = new ArrayList();
package.stubs().method("getName").will(returnValue("pack1"));
package.stubs().method("getClasses").will(returnValue(classes));

// and so on,...

This code is so awful that it is almost useless. We need a fresh approach for this kind of unit testing: a mocking DSL!

A mocking DSL

Well, it may be far fetched to call that a DSL, but the idea is to use operation calls to represent the mocks structure and the expected stubs (with real-life code sample):

topPackage = (Package) create(
mock(Package.class),
stub("getFullName", "model"),
stub("getOwnedMembers",
elist(createClassOne())),
stub("getNestedPackages",
elist(create(
mock(Package.class),
stub("getFullName", "model.pack1"),
stub("getOwnedDiagrams", elist(createDiagram())),
stub("getOwnedMembers", elist()),
stub("getNestedPackages",
elist(create(
mock(Package.class),
stub("getFullName", "model.pack1.pack2"),
stub("getNestedPackages", elist())))))))));I let you imagine the same mock structure with "classical" declarations,...

Even here, this is not a revolutionary improvement:

1. it is still quite verbose: more complex data structure may not be readable at first glance (but do you need them for unit testing?)
2. instead of having "create", "stub", "mock" methods, "package", "klass", "method"may be closer to the domain. If you are to test heavily the complex API, I would suggest to do so.
   

The MockBuilderTestCase class

The "create", "mock" and "stub" operations are offered by a MockBuilderTestCase class subclassing the MockObjectTestCase (see the jmock doc).

This is fairly simple code:
public class MockBuilderTestCase extends MockObjectTestCase {
private Stack mockStack = new Stack();

public Mock mock(Class klass) {
mockStack.push(super.mock(klass));
return currentMock();
}

private Mock currentMock() {return mockStack.peek();}
protected Mock callOnce(String methodName, Object value) {
currentMock().expects(once()).method(methodName).will(returnValue(value));
return currentMock();
}

protected Object create(Mock mock) {
mockStack.pop();
return mock.proxy();
}

protected Object create(Mock mock, Mock...) {
return create(mock);
}

protected Mock stub(String methodName, Object value) {
currentMock().stubs().method(methodName).will(returnValue(value));
return currentMock();
}
}
As "mock" operations are encountered when java executes the topmost "create" function, a new mock object is created and pushed on a Stack. Then all subsequent "stub" operations are applied to that mock. Eventually, when the "create" operation has all its arguments evaluated, the mock is ready and can be removed from the stack.

I wish I could turn the "eager/lazy" knob for evaluating arguments in java. I often find that using nested operation calls in java can provide an easy internal DSL, however, having the operation arguments evaluated first is not always practical.

Don't forget the Law of Demeter!

A last tip for an efficient and easy testing of classes that uses a complex API: don't let them access a too deep level in the API hierarchy.

That is, instead of allowing modelBrowser.getOperationNames() be like

List result = new ArrayList();
for (klass: getAllClasses()) {
for (operation: klass.getOperations()) {
result.add(operation.getName);
}
}
aim for delegation to classes dedicated to lower levels:

List result = new ArrayList();
for (klass: getAllClasses()) {
result.addAll(classBrowser.getOperationNames(klass));
}

Thinking twice about it

We are conducing an interesting exploration project these days.

The product I am working on uses UML models as a primary input. Those models are used for test generation, after setting various coverage criteria on the model elements.

Conceptually, the picture is quite clear: data in, data out. However, the devil is in the details,...

Having the same UML model referenced in 2 different tools leads to pretty complex synchronisation issues between the 2 data repositories:

• when an operation is removed, you have to remove the corresponding configuration
• when an operation is added, you have to set some kind of defaults for it
• when an operation is renamed, well,... you're stucked! You can only treat it as a removal + addition

So working on the third generation of the product, I was challenged pretty hard by the development manager to find anything that would allow us to get rid of those synchronisation mechanisms. I was quite reluctant at first but decided to have a go at it.

Thinking twice about it

I started this exploration and it appears that I may be quite possible:

• using the UML profile capabilities of the UML tool, it is possible to add more semantics to model elements and let user define properties on them
  

• using UML2.0 interactions allow us to represent quite nicely the operations that are targeted for test generation
  

Yes, this could work. But wait! Look at the pay-offs, we wouldn't hope they were so great:

• the configuration is always coherent with the model
  
• there's not a line of code to edit, visualize or store this configuration
• you have a natural way to define initial states for test generation and to deal with UML elements
  
• this approach is reproducible to other UML modeling tools in the category we're targeting
  
• you don't have anymore test generation projects to manage, because all the information is efficiently stored along with the modeling project!

Okay, okay the "No silver bullet" theorem still applies there. There are some obvious drawbacks:

• we are constrained by the UML tool. The properties that are vital for -us- are displayed by -them-

• we are, of course, a lot more dependent on the tool bugs (and of course I spotted some, even if I could find turnarounds)

• there is no easy way to tell the user what to do to configure the test generation. It doesn't come right into his face (that's the biggest drawback to me - a UI is supposed to provide the maximum guidance to the next thing to do)

I don't know where this experiment will lead us but the lesson is nonetheless interesting:

Think about it twice: can you state your software requirements so that the solution is not included in the question?

"We need an application to configure the parameters on the model elements"

Otherwise this may lead to other requirements,...

"Please, synchronize the data for me,..."

Writing on the walls

We had this wonderful session last week about "writing on the walls". This tutorial was held by Laurent Bossavit (read here for his contribution to the Agile movement).

The purpose of this tutorial was to teach the use of 3 simple agile tools:

• the tasks board
• the release board
• the burn-down chart

As far as pedagogy isinvolved, the tutorial was really great. In 90 minutes, we had planned 4 or 5 iterations, developed 3 (using dices to decrease the points on the cards!) and really experienced what if feels like to communicate the project status on the walls.

The tasks board

I have been practising the tasks board for some time now and already had this kind of revelation: holding the cards, displaying them, ripping them off, replacing them, writing your names (the pair) on it,...

This "physical", "concrete" approach to the work to be done makes it very real. It makes the people focus on something concrete, not on a line in an Excel spreadsheet or in an html page. Hence, when a card is moved from the "TODO" to the "DONE" column, you can see it's finished.

We found a common pattern for the 3 tools, during the tutorial. They deliver information in a very progressive way. The tasks board answers the questions:

1. "how far are we done?". Glancing at the board gives you the info right away: GOOD -> many cards in the DONE column, BAD -> many cards in the TODO column (unless at the very beginning of the iteration of course,...)
2. "how much exactly has been done?". Look at the expected/actual velocity
   
3. "what has been done? / what remains to do". Reading the cards tells you what the developped features are

By the way, the exercise is a lot less futile when you have acceptance tests corresponding to each card and if they are green when the card is said to be done. But this is another story,...

The release board

This one is fairly simple. You just display the remaining cards of the milestone, sorted along the remaining iterations. In additions to the cards, you add the expected velocity of each iteration (and make it match to what you know you can do!).

The interesting part about the release board is that it tells you a story. It is like a storyboard of what's going to happen in the project. If done properly, you should understand the plot, the tension between the characters, an overall sense of coherence, and... the happy ending, of course!

I also found that this is a very good tool for the client to communicate with the developers. Most of the time, developers are so much engaged into their code that they end up lacking a vision for the overall product. The release board helps them getting the broader picture and to understand the customer priorities.

The burn-down (or burn-up!) chart

This one is the typical tool of the manager and illustrates a very fundamental point about "writing on the walls". On this chart (in a "burn-down" fashion), you display the stock of points to be developed at the beginning of the milestone, and after each iteration, you add a new point on the chart, showing that this stock is decreasing. So you have a more or less declining slope, made of several segments.

Then, by interpolating roughly, you draw a line that crosses the X axis, giving you a bold estimate of the release date for the whole product.

Now, what is a burn-up chart? Well, reality is not very linear. Every once and then new features or weird discoveries pop out, and your stock will be increasing. You can display things the other way around, showing that each iteration adds features and try to reach a "ceiling". Then, you can easily raise the ceiling bar if necessary. You will also be able to get an estimate of how much the ceiling is raised along a typical milestone and use this data for your next planning game (thanks Olivier for convincing me on that point!).

To sum it up, the burn charts give a graphical representation of the velocity of the team for a whole milestone and helps conveying the "are we done yet?" data on a large scale of time.

Oh, by the way, what was the fundamental point?

The fundamental point

The fundamental point is not "cool, everyone sees the project progress".

The fundamental point is "cool, everyone cannot help but see the project progress". Developers are facing their responsabilities of trying to maintain the best velocity (mark this word: maintain,...). And managers have to acknowledge that it takes time to deliver software.

Information is not only displayed but it imposes itself to everybody.

Interesting software vs Plumbing

What makes interesting software? What makes hard software? What makes fun software?

Customer affinity

It all started with Martin Fowler post: CustomerAffinity. In this blog entry, Martin makes the point that "enterprise software" can be much more than "boring" and "shuffling data around". If you have enough interest for your customer (CustomerLove?), you can be pretty much delighted by finding the best ways to solve his business challenges.

So, as a programmer, even if you should be interested in frameworks (he calls that "plumbing"), you should be even more interested in the business that's supported.

Well, this has been more or less my opinion for years. I love to formalize, structure the fuzzy concepts that business people are toying with.

It is fascinating to see how human creativity can elaborate very complex things from very simple ideas. For instance, Thales is seemingly credited with the idea of buying the option to use olive presses during the next harvest. He bought the right to use something from sometime in the future. Now, play with the words in italics and you will get all the subtle ways to define options.

Is it boring?

Fine, fine, but Ravi Mohan disagrees with force (But Martin, entreprise software IS boring). He advocates that the talentest programmers would rather code a compiler than a loan-disbursing process.

So, "Computer science" and mathematics could be much more interesting than "enterprise software"? Reginald Braithwaite entry about business programming provides some answers.

Not only, getting the concepts right can be interesting (how do you define a "fraud suspicion"?), but this can raise really interesting computing issues. He takes the example of a system for routing concrete trucks: what are the optimal paths if you take the traffic into consideration? What if an accident occurs?

And Reginald raises the bar higher: can you do it in real-time? For thousands of trucks?

So, what's interesting in software? We can get some hints from those blogs:

• customer business (Martin's interest)
  
• mathematical / algorithmic challenges (from Reginald's blog)
  
• scalability / performance issues (Reginald again)
• computer-related stuff : languages, compilers, protocols, graphics-intensive software,... (Ravi )

I would also add one source of interest that is rarely given much interest as such: maintainance. Bleech! bad word, how can it be desirable? Well, when you code and design software day after day, you can also meet pretty tough challenges around maintainance. Is your code readable enough? How is conveyed your design? Can the software be plugged, unplugged gracefully? All these questions sometimes require specific talents that may be not found in the "hardcore programers" set. So, let's add one more point to the list of what makes software interesting:

• software maintainance 'art'

Fun happens

So far, so good. There's even a good news: you may have to deal with all of these in your next software project! The chances are very thin but software is such a continuity that you may have to implement a software for hospital management:

• what's the content and life of a medical file?
• how do you allocate efficiently resources such as rooms, surgery equipment? (what if a big -think 'terrorist'- accident occurs?)
• how do you do that for thousands of patients, in real time?
• how do you get x-rays pictures from the equipement to a secured medical file?
• how do you support a domain-specific language for an expert database?

Anyway, you will be bored somehow

Fun stuff, no doubt. So, what's REALLY boring? I'll tell you. This is what -I- call "plumbing":

• software compatibility: "oh, version X.Y.Z is needed for this dll!"
• environment variables: "BLOB_DIR was not set. How was I supposed to know that?!"
• defining paths, installing stuff: "a new Windows update is there, you should restart your computer"
  
• interoperability: "this very useful piece of java code is supposed to communicate with this VB crap,..."
• fighting with your programming language to express simple ideas: "look! 6 lines of java vs 1 ruby line of code"
• lousy operating systems: "what?! cmd.exe can not display ansi codes? What year are we?"
• fiddling with your UI framework to get what you want: "why on earth is this not resized?"
  
• and yes: business software that's repeating itself, either in the concepts or in the implementation,...
  

In the end, I feel it is very important to track the "boring" parts of software. Either make them disappear or become very good at dealing with them efficiently or hire someone to do that for you (why would "boring" be an universal definition?)!

And,... allow yourself to play from time to time.

So relax and enjoy

This is why I appreciate the RubyQuiz. This week's quiz is about a compression algorithm. I had a chance to learn a completly new field for me, without pressure nor any of the 'boring' constraints. I had everything at hand with a ruby interpreter (ok, not entirely true: I installed Radrails,...).

Another grid of evaluation for recruitement

Concluding thought. All those points can be considered when hiring someone:

• comfortable with concepts?
• has knowledge of the mathematics, algorithmics involved in programming?
• knows how to deal with scalability/performance issues (both in software and hardware?)
• is confident with parsing, ASTs, GUI, disk accesses?
• can produce maintainable code?
• can minimize the effects of boring activities? (knows how to install a full production system without spending hours on the web?)

Wow, this looks like a whole program that I can apply to myself! And I didn't even talked about project management and people,...

Ruby Quizzzzzzz

After 3 weeks of vacations (a wonderful place in Greece), I felt I needed some kind of warm-up,...

So I undertook the idea to answer the week's quiz from the RubyQuiz site every week.

This week quiz was especially fine since it was also simple enough to be used as an introduction to Ruby for our weekly Dojo at work (see here and here).

The subject of the quiz is to have a class that's able to transform days of the week into a human compact format:

• [1, 4, 6] should be displayed 'Mon, Thu, Sat'
• ['Monday', 'Tuesday', 'Wednesday', 'Thrusday', 'Sunday'] should be displayed 'Mon-Thu, Sun'

This looks simple but may be tricky to write properly,...

It was very interesting to take a look at the other submissions after coming up with my own solution:

• I discovered there was a way to use the Date class to get standard day names
• You can use infernals text manipulation ideas to get the job done (short but unreadable to me)
• No one used Sets or extended the Range class like I did
• There were only few submissions creating ranges from an array with successive integers in a readable way (this is were I struggled the most)
• Not everybody submitted unit tests
  
• I was the only one to submit specifications using rspec
• After writing down my own solution, I just modified it to take advantage from the "inject" method of Enumerable to make my code more concise

Well, I look forward for doing the next quiz, or even best,... submit a quiz myself!

Agility and legacy code refactoring

Let's say you have some bad legacy code (I have yet to see a codebase without any,...). When, how should you refactor/reengineer it?

In an agile context, there are conflicting forces related to that matter:

• deliver as much value as possible
• have a unit test for each line of code
• refactor mercilessly
• raise your velocity

Real cases from real life

Ok, this is very abstract, let's take some real cases:

1. You need to implement a cache to lower memory consumption (this is the consequence of a customer request) and you realize that the objects you are building are not properly build:

instance = new Model(xmlContent);

Creating a model doesn't have to be dependent on a specification representation format. You'd better use a factory for that. This code should be refactored,... But is it the right time to do so?

2. You found the fix for a nasty bug. Very very nasty, ... But deeply buried in a horrible mess of interactions. You have to write a unit test to correct the guilty class. Easier said than done. A real unit test may require a lot of refactoring, just to untangle the class dependencies.

What should you do? Write an "integration" test? Take time to write a unit test, refactoring everything that's necessary?

3. You have to add a new import/export functionality. However, your persistence design is so bad that implementing the function looks like hell,...

Should you refactor your persistence design? This may halt the team for many iterations. Should implement the function, going through so many loops and hooks, but doing it in a realistic time?

The books/consulting answer

I would like to point out an easy answer from a well known book or website, but I haven't found any. No, wait, there's one! The famous "consultant answer": "it depends,..."

Whose call is it?

Well, before emitting an opinion on the subject, I would like to add some more questions: whose responsibility is it to answer? Is it the client? Is it the team? Should they bargain refactoring actions around the velocity issue?

I think,... (drum roll,...) that this is,... (drum roll again,...) a development team issue. Let me list several reasons for that:

• the team has the responsibility to improve its own velocity
  
• the team has the best knowledge on how to do it
  
• the team has to live with the code day after day and suffer from its sluggishness
• really empowering the team also means letting it decide on how to improve its velocity
  

Now, what can be the policy for the 3 above cases?

Refactorings in the 3 cases

1. The "oh, this should be refactored" case

The pair-programmers should not refactor any code that doesn't get in their way. 3 reasons for that:

• keep the velocity up,
  
• don't risk to break existing working code,
  
• work on that code when really exercising it (so they don't take the risk to over-refactor)
  

By the way, this is such a real case that we incurred a several days development penalty lately for not following this rule (hence the idea for this blog entry,...)

2. The "nasty bug" case

This one is a "must do". You have to write a unit test for your fix, even if it is difficult. If you don't do that:

• you take to risk to face the bug again
• you may write an "integration test" that will slow your test suite and may be very fragile
• you open the door for your QA team to write more fragile validation scripts
• you don't show to path to more testability and better design for the guilty class
  

Having said that, the refactoring can range from a complete reengineering of the guilty class, to the sole extraction of the guilty behavior in a special-purpose class, passing by the only isolation of the bad class.

On that real case, we chose to reduce as much as possible the class dependencies, just to be able to test the fix, and only that.

2. The "new functionality/bad velocity" case

This one calls for a trade-off (the famous "it depends").

On one hand, you cannot spend several iterations on your brand new this-time-I-will-get-it-right-design. On the other hand, adding functionalities on existing debt adds more debt (and we all know about interest rate composition).

The only duty here is to do at least something. You have to set the path for the reengineering of the offensive design: implement a tiny part of the solution and communicate a new vision to the rest of the team.

Then, on any of the 3 refactoring cases, another pair can pick up the new design and push the codebase towards the new path. The net result of this strategy is:

• a velocity that will better reflect the codebase state
• an investment against more debt if not an investment towards a better velocity

This is also a real case,... we chose here to prepare the path for persistency isolation by introducing several abstract classes upon the base class of domain entities to isolate the persistence behavior. Then we have to move some entities to an abstract class with less dependencies.

Let the team clean the house

I think this is very important for the team to communicate a lot here. The first case calls for information: "refactor that, next time you have develop/fix that code".

The second case needs communication around the new concepts introduced to enhance testability or if the extraction that has been done is a bit clumsy.

The third case, presenting a compromise and beginning some kind of pharaonic work around the existing code warrants obviously for a lot of discussion.

It's like house-cleaning,...

• if you don't do it regularly, it can only get worse
• you can fix a broken bulb more easily if you have ready-to-use tools and spare parts
  
• building a new ground-heat heating system is a long project but this will be a nice replacement to your ever-broken old one. In the meantime, you can still add a new remote control, even if it would be easier on the new system.
  
• you have more fun when you appreciate where you live!

[Update 03-08-2006]

Ian Roughley has written a very good article on a non-intrusive technique to write tests for legacy code. Add some logging traces and a Junit appender!

This is a really nice technique to use when you want to fix a bug and don't have time to break all dependencies to write a unit test (or you can't take the risk: see case n. 2).

You can also use it before refactoring some code, but in any case you'll have to get rid of those tests sooner or later (see my comments on the article) to replace them with "real" unit tests.

Unit testing with style

Writing unit tests looks like an art sometimes. You want to:

• test your class functionalities
  
• test it properly in isolation
• convey the specification of the class as clearly as possible

Following those objectives, we encountered today some issues that are most certainly classical but not necessarily handled properly.

Setting-up of an input object hierarchy

Our tested class is a factory that takes an object hierarchy build from a parser and creates another hierarchy of objects.

To make an analogy with an xml configuration file, the first hierarchy is a set of xml nodes and attributes, the second one is a configuration object with specific composed options. The issue is: how do you build the input object hierarchy in a way that's readable and really conveys its expected structure?

The unreadable way would go like that (in java):


AtomParameter attribute1 = new AtomParameter("attribute1");
AtomParameter attribute2 = new AtomParameter("attribute2");
ListParameter classList = new ListParameter();
classList.add(attribute1);
classList.add(attribute2);

AtomParameter class = new AtomParameter("class");
ListParameter modelList = new ListParameter();
modelList.add(class);
modelList.add(classList);


And the readable way goes like:

Parameter parameter = list(atom("class"), list(atom("attribute1), atom("attribute2"));

By implementing the proper 'list' and 'atom' functions (returning List/Atom Parameter) the resulting code conveys the expected structure much more clearly. Unit tests as a software specification gets more real!

Testing in isolation

Not a new subject here, we use jMock to isolate our class. However, here's a pattern we've used some times. Our factory class declares 2 constructor methods:


public static ModelFactory createFactory(Project project);
public static ModelFactory createFactory(Project project, Translator translator);

In that case, the first method uses a default Translator implementation and the second one allows for another Translator, which is going to be our mock object of course. The second method may look like a pure test-only method but if you come about it, it is just extending the Factory API towards more flexibility.

By the way, I would also like to point out in this entry, one of the difficulties with mock objects in general. There is no way to specify the protocol for using an interface in a central place. If operation1() must always be called before operation2() on an interface, the mock expectations won't really declare that. Moreover, the expected protocol cannot be enforced from any unit test where the interface is called.

Protocol errors will then be catched by integration tests (acceptance tests for instance). The creator of jMock, Nat Pryce, has attempted to get around this, but he ultimately thinks this is too complex to do in Java (see the pragprog mailing list archive).

Expressive assertions

JUnit has blessed us with quite a few assertXXX methods but I have recently followed the advice found in this article, and it really changed the way we use assertions.

For instance, testing our factory goes like this:


assertThat(aFactoryUsing(model1).createInstance(parameters), containsSlots("a", "b"));

assertThat is an assertion method from MockObjectTestCase
aFactoryUsing() is a method creating a properly setup class to test
createInstance() is the tested method
containSlots() is a method returning a custom Constraint object (from the jMock framework) and doing the necessary checks for the test

The nice thing that comes for free from jMock is then the ability to compose custom and standard constraints. For instance:


and(containSlots("a", "b"), not(containSlots("c", "d")))

First-class code

At the end of the day, unit testing with style is not wasted time. If you consider your unit test code as first-class code, that must be as factored and expressive as possible, you will really facilitate communication, design and maintenance (especially test code maintainance, which can be otherwise very, very, very expensive,...).

Feedback from another Dojo

Today's Dojo was interesting, at least interesting enough to keep a trace of it.

The subject

I wrote a small Dojo subject a week ago that was selected by the group. The subject was to have a DSL to describe Recipes and to be able to compute some data about it.

Let's take the easy-child-access yogurt recipe:

• Pour one yogurt
• Add 3 eggs
• Add 100 ml of vegetal oil
• Add 200g of sugar
• Mix all with 300g of flour
• Cook during 30' at 180° celsius
  

The idea is to be able to compute:

• the list of ingredients with their quantities
  
• the cooking time
• the preparation time

Of course, you can start with simple calculation rules and add more complex ones such as:

• "Melting 50g of chocolate takes 30'' in the microwave"

You can also play with some grammar aspects:

• synonyms: Add, Pour
• complements: in a jar, with a mixer
• ...
  

I had several objectives with this subject:

• Make people collaborate on a quick design session
• Implement a parser for a DSL

So far, people decided to vote for this subject and we chose to focus on the quick design session then on some coding.

The Quick Design Session

This session was elaborated with somebody at the board during 5 mn, sketching a domain model. We quickly converged on the main classes: Recipe, Action, Ingredient, Quantity and thought about some kind of PreparationTimeEngine that was responsible for calculating preparation time depending on an action and its ingredients.

The acceptance test

We started coding an acceptance test for a very simple recipe: a yogurt with sugar!

• Take 1 yogurt
• Pour 10g of sugar
  

And programming the acceptance test took us the remaining 35' we had on the session,...

The retrospective

Yes, the result was not tremendous, and I am still thinking: "wow, 5 poor classes, 1 acceptance test = 1,5 hour, not a great productivity".

Anyway, I liked this Dojo for what we learned:

• :-) The quick design session pleased everybody because they really had the opportunity to share a common design. Something to do again "live", because this really accelerates the feeling of common ownership
  

• :-) I liked the emphasis on getting the acceptance test as readable as possible and the hunt for local variables that were not necessary

• :-) I liked the discussion around the necessity to have factory methods: createAction,...

• :-) There's no better place to jellify the group around fun and quality code
  

• :-\ Some pair co-pilot were relatively silent. Their role was taken up by the group

• :-\ Yes, I still think the productivity was poor and I think this shows plenty of improvement room for the group

• :-? We should try to enforce the rule that the group is silent and only the pair should talk. The other ones could take notes, observing the functionning of the pair

• :-? I wondered about the use of Java for this kind of Dojo. Python or Ruby would be more appropriate and productive

• :-? It is certainly better to prepare the Dojo subjects, so that the participants can get up to speed in a limited time. I had prepared a written spec with an acceptance test, but written acceptance tests would have been better. I think this is even more true with some of the subjects we wanted to do: Bayesian Filter for spam, Java Plugin Framework,...

Now, let's see how goes the next one.

Responsabilities dilemna

When can you say "too much is too much"?

Let's say you have an object: Customer. Wow, how original! Anyway, please bear with me.

So, your Customer class has some "customer info" responsabilities, but also it may be a "billable customer" or a "prospective customer" (for cross-selling). Hey wait, it can also be a "persistent customer". And so on.

How do you manage all these responsabilities with a classical object-oriented language?

The Horror show

First of all, you could put all the code in the Customer class. Boo-oo! "OO for beginners" tells us its bad, you end up with a 2000 lines class.

or,... you could have only a Customer class containing the customer info and a BillManager that bills a customer, a SellManager that tries to sell him stuff,... No, I'm kidding, let's forget that one too.

or,... you could inherit from a PersistentObject class, derive BillableCustomer from Customer,...

Time to stop the horror show.

Delegation forever

Then you can put everything in the Customer, but it can delegate to other objects. For instance, customer.save() would delegate the real action to a CustomerRepository. This would be the same for other responsabilities.

What about testing? Oh yes, testing,... Ok, you can use your best-of-breed dependency injection framework to inject the proper objects behing finely crafted interfaces.

Right, but what about dependencies anyway !? Your Customer class is still not reusable without a whole bunch of interfaces! What if you don't want your customer to be billable at all?

Ok, you can set the persistence aside by having the Customer clients make calls to an appropriate factory after dealing with the customer. But what can you do with the billing responsability (No, not the BillManager again,...)?

Object inheritance

Another way, suggested by the Streamlined Object Modeling book is to use "Object Inheritance". You use the Actor-Role pattern and define 2 classes:

• the actor: Customer class
• the role: Billable customer class

A billable Customer class will have the same interface as the Customer class and will delegate all customer info requests. We could say this is a kind of "external delegation" in contrast to the "internal delegation" mentioned before.

When discussing this pattern with a friend recently, I told him that I felt uneasy with one thing (beside the heavy use of delegation which results in silly code in Java). It was strange for me to have 2 objects representing the same logical entity.

He told me to consider another way to see things: BillableCustomer would be a "decorator" of Customer. If I need a BillableCustomer, I write: new BillableCustomer(customer) and do my job with it. I really like this way of seeing things. There's still one drawback in my mind: if BillableCustomer needs to be persisted, you still have to consider 2 entities for the same logical object in the system.

Dynamic languages to the rescue

In the end, what I would like to do is to mix behaviour into an object. This is exactly what Ruby allow me to do: I have a Customer class and I can extend it Billable responsabilities at runtime:

customer.extend(Billable)

This is the idea of "Duck typing". You cannot rely on the object type alone to know its capacities. If it walks like a duck and quack like a duck, then it is a duck (even if it is really a swan,...)

There is certainly closer to "object-orientation" than any class-interface-inheritance pattern. If we think of it, object are often thought as human when doing object analyzing. And we, as humans, often learn and forget, get abilities and loose abilities. Objects should be able to do the same.

A new persistence challenge

But "when is easy, too easy?".

Yes, when it comes to persistence, how do you do in Ruby with something like that? ActiveRecord was not thought with that kind of use in mind.

;-) I have no doubt that Rails fanatics could come with a fairly elegant solution to that problem.

Enlightment, planning and convictions

Sometimes, learning new things is scary.

You've just learned something you find very important, acquired a new skill and you ask yourself:

"How can I have worked without understanding something so crucial during all this time?"

Well, another more useful question usually pops up in my head:

"Why was I not able to pick this up before?". This is certainly more useful because it opens up the possibility to pick it up faster next time!

Recently, I got the feeling that I learned very important lessons about the raison d'être of agile methodologies.

The cards array enlightment

I remember standing in front of our cards array contemplating the next iteration planning and the result of our last planning game. I was thinking: "now I know why I was not so effective as a project manager for planning the activity of the project" (better late than never!)

Planning, yes but why?

I always had difficulties for planning software activity, and was even beginning to think that development was something like the butterfly story introducing any chaos theory presentation: a beat of wings and milestones go away.

Many of my attempts to get more insightful plannings failed. Here's a set of recurring issues:

• how can the development planning stay in line with the global roadmap (especially if it is very "ambitious" but also a moving target)?
• how do you plan bugs?
• how can you make accurate predictions regarding the delivery date?
• how do you fight developers natural optimism ("well, 2 days should be ok for the parser")?
• how do you plan non-development time (meetings, holidays, illness,...)?

Moreover, every "aggressive planning" I saw was the recipe for repeated failure. And the effects on the team are disastrous: "Whatever we do, we are late". But "late" relative to what? A release date set by a manager and approved by a developer muttering to himself "I'll do what I can". So what's the real use for a planning?

A manager answer included: "setting a challenge to motivate the developers". This is the "aggressive planning" theory. A really counter-productive one, since it leads to the "We're always late" syndrom, which is soon followed by the "We're poor developers" consequence.

I had my own variant on aggressive planning: "Watch what you do and register time passed". I thought that doing this will drive individuals to be more conscious of "unproductive" time and get rid of it. I was wrong. A developer can chase "unproductive" time only if he has decided to, not if he has to fill a time report precise to the hour (I have even read about minute-precise time report, where bathroom time is to be precised!). And then, as a mean, he can watch his time spent.

So I figured out there must be better reasons for doing plannings (instead of the "I develop to my best abilities, you pick the product when ready" approach).

One of the reasons is resources management. If time-to-market is a crucial issue to the Product owner, then he may wonder "how many people can I add to the project to get it faster?". Once you keep in mind the Mythical man-month essay by Fred Brooks (and this funny anecdot), you can still try to add people to reach the project optimal size.

One other reason, is some kind of control over what is developed. Plan your activity, do the job. If along the way, the plan is not respected, this may be interpreted as a warning. However, if the meaning of the warning is that your plan was not accurate, you have not gained much.

And maybe the most evident reason is the need to communicate dates to people outside of the project: customers, sales representatives, consultants, investors,...

Planning revisited

After our 6 months effort to implement the full agile practises stack (we already had continous integration, unit testing, iterations), I now understand why this works:

• you deliver software in a time-box manner and focus on priorizing customer value, having both short-term control (the iteration) and mid-term control (the milestone)
  

• you have a very simple measure for productivity: velocity. You globally measure the capacity of the team to deliver useful functionality. Meetings occur, bugs occur, obscure technical work occurs but it is not counted as such

• estimations are more reliable because of the planning game. Indeed, estimating a task is better done by a bunch of people having different experience and knowledge about it (I used to do that on one-to-one with the "concerned" developer - which is not a relevant concept in eXtremeProgramming anymore)

• estimations are also more reliable because of the frequency of the planning games (every 3 months)

By the way, I want to emphasize all the benefits of the planning game:

• developers have the opportunity to understand precisely the customer objectives
• the customer can get a better grip on development risks and difficulties
• the team is really empowered and feels responsible for the overall project

Let the team manage itself

The last point is what stroke me in front of the cards array: many brains are better than one.

• the team is much more efficient at organizing the project than the project manager
• the team is much more efficient at defining and estimating the tasks
• the only way to get the maximum productivity is to trust the team members and let them set high-productivity standards. No management indicator can ever imply high productivity. You cannot measure commitment, smartness, collaboration

The consequences for a project manager seem pretty clear to me: you are only here to help the team manage itself.

How to do better next time?

Thinking about that, I concluded 3 things:

• I should have experimented more. Some agile practises seems evident. Continuous integration is one of them (spend machine time instead of developer time). Some other are not so intuitive: why use physical cards to track development tasks? It is not until you hold one in hand that you can feel why

• Empower the team. This is the brain number effect. I also find some relations with Michel Crozier's theory regarding the fundamental liberty of any worker to gain power around him. This also means that team management can never be an easy path

• I should live up to my convictions. For instance, I was not strong enough on emphazising why continuous integration should always be green, no matter what (this one I experimented)

Don't give up on your conviction

We all make mistakes, learn more or less painful lessons. However, when we learn something the hard way, we'd better not forget about it. What we learn become our convictions and the remaining questions have to be experimented.

I recently read a blog entry of an experienced developer analyzing the reasons behind his software projects failures. The whole entry is worth reading and wondering about. But one specific thing was worth remembering for me right now:

And if you decide to make changes, have the courage to go 100% with your gut. I've failed more than once when I watered down my convictions in order to appease dissenters. The only thing worse than evangelizing change and failing is looking back and realized you might have succeeded if you'd held firm on your convictions. What a waste!

I won't be caught twice on that one.