Humilty 

So - last monday (it is now Saturday), at around 8PM - I found a troubling bug in the software I was about to go to QA with. I worked on it till midnight. I worked on it with an excellent collegue the next day from around 830 AM till around 130 AM (the next day…late)…. same thing the next day, and the next… and the next… We thought we had fixed it!! We stayed till around 10pm packaging a build for QA … we were wrong, before we sent it to QA, we found the bug elsewhere…. bad. sad. un-good. did i mention bad?  Today - saturday, I found the bug. It was in an area of code I had written some hundreds of unit tests for. This is code we have been using happily, without issue for several months. In fact - this framework was basically textbook developed according to TDD. 

Unit tests give you confidence - in my case - Hubris 

So, I am a bit of a TDD zealot. I have been pushing TDD in the worlds large investment banks for a few years now. I am responsible in my current firm for rolling out a full TDD practice. I am confident in myself in general, especially when it comes to software. Having tests, even lots of them, even 110% coverage does not mean it will work right 100% of the time. Lesson: At no point in your software development career are you too good for a death bug. It has been many, many years since I have had a bug I could not work out in a few hours. I guess I really spent some 40-50 hours personally on this one, and my collegue about the same. That is some 100 hours lost. .Granted, it was a complex, difficult issue in truth - but still.

.. Ouch … Humbling, a little humiliating 

Scrum - velocity and the “fat tail”  Financial models are good for the average cases, but the ‘fat tail’ failing is a common failing for financial models. We assume normal distribution often - and often (very) this is not the case. Financial models fail (current CDO/MBS issues for example). Being not only a TDD zealot - I am also a scrum pusher… I have generally been running scrum based projects (more or less) since ~2004. I have also managed to get several teams near me in my current position - running scrum projects. So - what happens when you have a really fat-tail point in time and it kills your velocity - by a long shot… As someone said to me this week “remember - this stuff is not linear”. … Ouch … 

The self image  All of this really violated my self-image. I really did not believe i was going to have a bug like this. I really did not believe I was going to be the guy that was approaching lateness. I really did not believe I was the guy that was going to have to make compromises. I really did not think I was going to be the guy to potentially impact the YEARS MOST IMPORTANT deliverable for many, many people. Guess what - I was close, it was closer to the wire than I have ever been. … Ouch … 

Yet still I believe  So - even thought I see these failings in myself, in TDD, in scrum - nothing and nobody is perfect. Scrum and TDD are better than not-scrum and not-TDD. As to myself - I simply need to remember that while it is OK to hold high standards for yourself - you will fail. I am not afraid to fail. I want challenges. TINSTAAFL. No guts no glory. No risk no reward. When you are outside your zone of comfort - you are not learning (therefore, consider the opposite position). 

Agile development does not get rid of the risks - it just helps minimize them. VELOCITY is not 100% accurate. Estimation is still hard.Re-learning humility is never fun. 

Did I fail?  Well – I believe work ethic is important. I worked many, many hours, till very, very late as hard as I possibly could till I closed the issue. Closure is king.  I often say “closing is the mark of the professional”. I believe in the persistence and commit departments I won. I failed inasmuch as I will have to deliver less scope. I believe we will make it through QA on time. I know we can work this week to make what has to happen happen. Again- I believe I re-learned humility. Not fun - but good nonetheless. 

-Chris  PS - read this - not at all related, but excellent. It is a fantastic history of functional and OO programming languages - the interplay between both, the drama etc. 

No Responses to “Humilty, Life as a programmer, TDD, Scrum, (and a little F#)”

1. Luke Melia » Chris Donnan on Agile and Humilty, on November 12th, 2007 at 3:24 pm, said:
   

   [...] Part of agile is acknowledging that building software is hard and human beings (including yourself) are fallible. Chris Donnan has a blog post about his recent run-in with this truth: So - last monday (it is now Saturday), at around 8PM - I found a troubling bug in the software I was about to go to QA with. I worked on it till midnight. I worked on it with an excellent collegue the next day from around 830 AM till around 130 AM (the next day…late)…. same thing the next day, and the next… and the next… We thought we had fixed it!! We stayed till around 10pm packaging a build for QA … we were wrong, before we sent it to QA, we found the bug elsewhere…. bad. sad. un-good. did i mention bad? Today - saturday, I found the bug. It was in an area of code I had written some hundreds of unit tests for. [...]

2. michel, on November 12th, 2007 at 6:01 pm, said:
   

   Was this bug concurrency related? Those sometimes take some time to repro and activate. And TDD isnt really apt to find those bugs, at least thats what I have found out.

3. chrisdrop, on November 13th, 2007 at 1:44 am, said:
   

   Michael;

   No - the bug was not concurrency related. It was related to a deep clone implementation. Essentially, we have a layer that does deepcloning of some object graphs. Part of the graph was pointing, to something else’s internals. It was a very C++ throwback set of circumstances.

   We actually have lots of tests for this particular app that deal with its multi-threaded aspects. These sort of fall into the ‘integration tests in a unit test harness family’ moreover, but they were helpful and I believe effective. I would assert that writing good concurrent software is hard, but doing so with tests certainly makes it all a lot easier.

   -Chris

4. chrisdrop, on November 13th, 2007 at 1:50 am, said:
   

   Luke;

   Yes - I can appreciate your comments. I also agree moreover that we “went out too long” with this release. While I have been successful with implementing scrum in several teams in my current position, the higher up you go in the organization, the harder it is to convince people to change. I actually had a number of conversations that are not to dissimilar from your comments.

   The salient bits to me are:

   - humility is important.
   - unit tests are imperative, but will not prevent all bugs.
   -your velocity can be killed by the ‘outlier’ event

   As your comments help illuminate; N smaller projects running under a non-agile umbrella - is not agile.

   I am making headway still - and I think I have managed to compel people throught this particular project that you MUST be more iterative, must be getting end user feedback earlier and must simply disperse the risks of project delivery into smaller bites.

   All great food for thought - thanks again for the comments.

   -Chris

Comment on this post below

You must be logged in to post a comment.

You can leave a response, or trackback from your own site.