Sybase iAnywhere SQL AAnywhere Mobile and Embedded Database

I'd rather play golf

Thoughts on data management, autonomic computing, and self-managing database systems.

header image

Why don’t we document failure?

September 17th, 2008 · 3 Comments

I quite clearly recall a conversation with my PhD co-supervisor, Paul Larson (now at Microsoft Research), when I was a graduate student about documenting failures and not only successes in Computer Science. Paul felt quite strongly about this, and I agreed with him. A significant reason behind the need to document failure is that Computer Science, in ways similar to Engineering, is a field in which the same ideas are re-used and re-invented over and over again.

Documenting failure is also a significant recommendation of Donald G. Reinertsen in his book Managing the Design Factory, a book that I have mentioned previously in this blog. Here’s an excerpt from Reinertsen’s book (page 79):

The fallacy in thinking that high first-pass success optimizes the design process lies in underestimating the important information generation that occurs with failure. This tendency to treat failure as the enemy is relatively new; people have not always considered failures to be negative. For example, consider the statement of Robert Stephenson, one of the great engineers of the early industrial revolution. As Henry Petroski points out in his book “Design Paradigms” [nb. a followup book is now available, Success Through Failure: The Paradox of Design], Stephenson strongly advocated discussing failure in engineering literature. In 1856 he wrote: “A faithful account of those accidents, and of the means by which the consequences were met, was really more valuable than a description of the most successful work”.

Why the need to document failures? Here is a subsequent excerpt from the same chapter:

This [handling of failure] is difficult for us to do well because we have strong human bias to value successes more than we value failures. In most organizations failure is stigmatized and nobody wants to be associated with it…..Unfortunately this produces some dangerous side-effects. Since improbable failures have high information content, it is important to communicate information about failure quickly and widely throughout the organization. To the extent that we hinder the flow of this information, we will force people to reinvent failures that we have already experienced, and that generates no useful new information.

Precisely this question – why don’t we as a scientific community document failure – was the subject of a fascinating program on CBC Radio’s Ideas that was broadcast this past Monday, 15 September. On this program, entitled Science at the Summit, noted researcher and University of Toronto president David Naylor interviews a panel of leading scientists about this question, and the many implications that lie behind the stigma of research failure. It is a compelling program, and I recommend listening to the show’s podcast which will likely be available on the CBC Ideas website at the end of September.

Tags: Product development

3 responses so far ↓

  • 1 tom s. // Sep 17, 2008 at 8:42 pm

    Two suggestions.
    1. Many failures are documented – they are just not called failures. At some level, many published results are judged not worth following up. They may have merited publication, but they lead nowhere. Witness several papers by T Slee et al.
    2. Some failures are renamed something else. In synthetic chemistry (not my field) attempts to synthesize complex molecule A that instead yield complex molecule B are not described as “Our failure to make A” but as “A synthetic route to B”.

  • 2 Glenn Paulley // Sep 17, 2008 at 8:49 pm

    A point that perhaps I didn’t make as clearly as I wanted is this: I certainly agree with your point (1) above: lots of work is published and never followed up (and NOT just yours!!!). However, very rarely do you see an ANALYSIS of why the research failed in its expectations, or why a particular algorithm may be good for some particular application but is unsuitable for others.

  • 3 Learning lessons from projects // Jun 10, 2009 at 10:54 am

    [...] September I penned some thoughts on documenting project failure and its importance in the product development process, prompted by reading similar views held by [...]