How Do You Handle Failure?

Posted: February 10, 2014 in Random Ramblings, Testing Lessons

Note: this is a somewhat unpolished post. There are a lot of ideas around this topic, and I’m just scratching the surface. However, in the interests of following the 80/20 rule, here are a few semi-organised thoughts on the matter…

It’s a moment that chills me to the bone. A manager wanders over to say “Oh, you know that software you tested last month. We found a bug in production – it’s currently affecting EVERYONE”. Or a couple of UAT testers running basic test cases find a couple of bugs your vaunted exploratory testing failed to detect. To misquote Darth Vader, my failure is now complete.

A missed bug is only one form of failure, and a test project provides dozens of different ways to “screw up”. The moment of failure is rarely a positive experience, but as Dan Ashby wrote not too long ago, it can be a very useful opportunity to reflect, learn and improve. Failure can also provide us with an excellent example or story to share and even teach others with (which is a handy way of remembering….).

However, it’s not enough to swear a silent oath that you’ll never make that mistake, or to shrug and say “We’re only human” (which could be offensive to those who believe “they” live amongst us, incidentally). But there are a cornucopia of ways to turn the bitter taste of harsh lessons into sweet nectar of experience – for example, James Bach mentions several in his book “Secrets of a Buccaneer Scholar”. These are three steps I try to use to help me get a better outcome…

Review & Record
Software development teams, in my experience, are big fans of reviews. Be it a peer review, a sprint retrospective, or a post-project review, they permeate software projects like the scent of dog pee soaked long and hard into beige carpet*. And they don’t have to be team reviews – some team members will conduct retrospectives of their own activities. I find it very instructive to examine my own activities through the life of a project to pinpoint the good, the bad and the ugly at a personal level.

A review of any kind is almost always accompanied by some sort of recording process, be it the “volunteered” note-taker scribbling the discussion down furiously, the ubiquitous whiteboard or the A3 sheets of butcher paper taped to the wall. Regardless of how it’s done, recording is an important part of capturing what went wrong and plans on how to tackle them in future. I record lessons and notes in a small notebook, re-reading my hastily scribbled “litany of instruction” at regular intervals. I may make every mistake in the book, but I’m determined to do so only once.

A final note on this aspect: there seems to be a common attitude that reviews need to occur at the end of a stage or milestone. This is far from the case. Mini-reviews can be done at any time, over an entire phase of development or as small as a single task or incident.

One problem I’ve found with reviews is the tendency for the results to fade away like the light from a late autumnal twilight – that is, very quickly. The process usually goes something like this…

1. Team arrives at review (biscuits & confectionery available to deliver sugar high for extra energy)
2. Team gives great feedback (possibly on the back of sugar high) on the highs and lows of the project
3. Feedback is diligently recorded, re-formatted and distributed
4. Report is filed away by all and sundry, never to be referred to again

All of those excellent, timely and relevant lessons are lost… until the forgotten mistakes resurface in the next project, where they’re captured in yet another review, codified in a report for the second time and lost once more in the mists of an Outlook archive folder.

A review is an excellent start down the road of improvement, but it’s only the first step. Lessons learned or mistakes made need to stay in the collective and individual minds that haunt the team if they are to provide any future benefit.

A story associated with a particular lesson can be an ideal way of remembering, and even better if it’s a memorable story for much of the team. Even if the incident only occurred to one or two team member, sharing the story amongst the team can provide impetus in etching into the memory.

Keeping reports and notes from reviews in an accessible, communal location is another method to retain the lessons learned. Having newcomers to the team read through notes, perhaps with an experienced team member, may allow the lessons to “live” beyond their natural life.

If a particular failure has been dissected, distributed and dug up at the appropriate moment, it’s on the verge of transforming from painful memory into useful experience. But for it to really help carry the day, that memory (and the lessons learned, or the plans made) needs to be acted upon.

I’ve found that if I can remember, I can react. The memory/story/instruction may not provide a “five star, guaranteed to work 100% of the time, all the time” solution, but at the least establishes a solid starting point for dealing with the problem.

However, it’s important to note that the right focus is critical for effectively reacting to a re-visited situation. For example…

“Oh, this bad thing is happening again. Remember how bad it was last time?” = not very useful
“Oh, this bad thing is happening again. Remember how we said we would handle this last time?” = much more useful

When the knowledge of tackling a previous problem that’s been scrupulously recorded and embedded in the team’s memory combines with the right attitude, there are few obstacles that can stand in the way.

After all, when all is said and done, victory has a thousand fathers and failure is an orphan. But that street-wise orphan can teach you more than any of the crowing pack of parents.

*Analogy prompted by recent experience

  1. Anne-Marie says:

    Hi Dean

    When I miss a bug, that’s found later in production, I don’t like it. It irks my professional pride that I would miss it regardless of wether I “should* or *should* not have found it. Recently, it struck me that this is how a lot of developers must feel when we rock up and gleefully tell them about a bug we found it our code.

    So even thought it still irks me, the benefit I get is that I get to empathise a little with the developers I work with, and hopefully that will in some small way, help me to appreciate the work they do and to be more humble when I find a bug.

    • Great attitude Anne-Marie,

      I’ve had problems with my own reactions to missed bugs, resulting in some semi-unprofessional behaviour (the worst thankfully long ago). It never occurred to me that missing bugs could actually improve our empathy (and hopefully relations) with developers.

      I can easily imagine making such a change of (inwards to outwards) perspective can be difficult at times, though.

  2. […] How Do You Handle Failure? Written by: Dean Mackenzie […]

  3. Michael says:

    Hi Dean,

    nice post! I recognize your story as I have attended many sessions in which lessons learned were discussed. After these sessions they were indeed added to some kind of report without any priority given to them.

    As testers we mentioned our concerns about this behaviour during every meeting of this nature. The funny part was we always were able to repeat most of the lessons learned time and time again without having to think about it. Bringing the previous report was always enough.

    Nowadays we do a lot better. During our review/retrospective sessions we usually create a Product Backlog Item (PBI) of a lesson learned so we will not forget about it. During these sessions we discuss how to deal with this PBI and are, most of the times, keen to add it to the next iteration to start improving right away.

    Hope this helps. Good Luck!


    • Hi Michael,

      It’s good to see testers leading the way on this! I feel that as testers, we naturally focus more on problems as that is our mind-set, and that makes us ideal candidates to lead this kind of initiative (well, that and the fact that nobody else usually bothers with retaining this info).

      Using PBIs to capture lessons is a great idea, especially in the environment I’m currently working (where all the attention is on PBIs). It captures what we need to in a format that’s relevant to the team.

      Thanks for your comment,

  4. Michael says:

    Hi Dean,

    glad to hear that you like my comment. I hope you can help out to incorporate this way of working within your environment as well. I would like to read about your findings in another blog post over time 😉



    • Thanks Michael,

      I’ll be speaking to my manager next week about this, and I’m interested to see what kind of reception it might get. I hope I get the chance to revisit this topic, especially if the idea can be given a fair go where I work.


      • Michael says:

        Hi Dean,

        I was wondering how your talk with your manager went.
        Hope he was – and is – enthusiastic about the idea.


      • Hi Michael,

        He was receptive to the idea, but events have over-taken us somewhat. There is a lot of process review going on at the moment, and using backlog items for lessons will be one of the many ideas the team will trawl through over the coming weeks. That’s not to say the testers can’t still do it informally in the meantime, but not being involved in that project, I’m not sure if they’ve started doing it as yet. I hope it gets adopted more widely, along with a few other ideas that have been recently floated.


Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s