Software failures. Be afraid, be very afraid.
I used to love software.
Since 1989, I have been in the business of developing custom software applications. If you had a problem, all you would need, in my view at least, was to find the right software “solution”. And if you could not find it on the shelf of your local software supermarket, I could tailor one for you that was “just right”.
But over time I have developed somewhat of a love/hate relationship with software. When it works, it’s great. I have seen many cases (at least with my systems) where it not only makes businesses more productive and more profitable, but also where it has vastly improved the lives of those people “manning the cubicles” for eight or more hours per day.
On the other hand I have had my share of sleepless nights wondering if what I was building was ever going to work, and many uncomfortable meetings to “discuss” cost overruns and missed deadlines. Not to mention those “calls” I get every now and then describing undesired system behavior. No “bugs” in my systems!
Stop, listen, what’s that sound? Everybody looks what’s going round...
So I started to look around and what I saw was a long history of software projects gone awry, leaving behind not only billions of dollars in losses, but also losses of life. In fact it is more common that I ever expected, and the scale of damage that it causes is even more I ever imagined.
The Standish Group did a study of software projects and found a failure rate of 31.1%, with over half of the projects go ing over budget by almost 90%. With $250 billion is spent on application development annually, the cost of failed software in the US was estimated to be $60 billion per year.
Here are just a few examples from the “Software Hall of Shame”:
Therac-25 –Cancer Cure turned killer
In the late 80’s a programming error with the Therac-25, a machine used in the treatment of cancer, resulted in six cases where patients were give massive doses of radiation by mistake. In 2000, another machine used in the treatment of cancer machine ended up killing 8 people and injured 20 others; again the result of a programming error.
Oh Canada – we are on the list!
In fact, software failures hit close to home. Just last year the Hudson Bay Company experienced problems with their inventory control system which contributed to a $33 million dollar loss, a loss that helped pave the way for the takeover by Jerry Zucker. In 1983, Donald Hudson (no relation), the embarrassed president of the Vancouver Stock Exchange, had to admit they had goofed at calculating their own index for the past 22 months; a team of people took two weeks to do the recalculation. (The correct index was 574 points above what was reported.) And today there is the infamous Gun Registry. First budgeted at $2 million, it is going to cost $2 billion to complete. What were they thinking?
You would think the big guys could get it right.
Having lots of resources is not guarantee for success; in fact the failure rate goes up with large projects. In 2004 Ford Motor Co. scrapped a $400 million purchasing system after deployment. 160,000 Toyota Prius’s were recalled because of a software error in an embedded system. The new Denver airport is in a state of chaos due to the failing is its new systems. In 1997, the US Internal Revenue Service cancelled a tax modernization effort after spending $4 billion. In the UK (2004) a the grocery chain J Sainsbury PLC had to abandon a $527 million supply-chain purchasing system after deployment and hire 3000 people to stock the shelves manually.
Who cares about software? I have a meeting to catch and my flight is delayed.
The FAA spent $2.6 billion on a replacement for their “massively complex” air-traffic control system. Riddled with problems the project was cancelled, and they had to revert to the antiquated system. Today you can thank the gridlocked skyways in part for the failure of that project. The economic impact of these delays is estimated to be $50 billion per year.
One Small Misstep for Mankind
In 1999 NASA's tried to launch a spacecraft into the orbit of Mars. Instead of reaching a cruising altitude around the planet, the spacecraft crashed into Mars, destroying itself and kissing $125 million goodbye. Investigators blamed the crash on “the failed translation of English units into metric units in a segment of ground-based, navigation-related mission software”.
After spending $7 billion unmanned Ariane 5 rocket the European Space Agency watched in horror as the $500 million rocket exploded just forty seconds after lift-off. It turned out that the cause of the failure complete loss of guidance and attitude information due to specification and design errors in the software of the inertial reference system.
Software and the Big Bang
In the Soviet Union bad software there led to the largest non-nuclear explosion out planet’s history! Operatives working for the CIA allegedly planted a bug in Canadian software purchased to control the trans-Siberian pipeline. This technology was consider sensitive US technology, and the Soviets could only obtain it “covertly”. The bug told the software to give positive readings to bad equipment. While this case is a little different, it does demonstrate the potential consequences of putting too much trust in bad software.
Who needs the National Enquirer?
You may rather want to read one of the over 75,000 cases documented at the The Risks Digest going back to 1985.
OK, what can we learn from these mistakes?
Of course there is no shortage of research into software failure and how to avoid it. Surely an awareness of these factors will guarantee your project’s success. On the other hand, just because you read the South Beach Diet doesn’t mean you are going to loose weight. So, get your pencil handy. Here are the top factors contributing to software project failure:
• Unrealistic or unarticulated project goals
• Inaccurate estimates of needed resources
• Badly defined system requirements
• Poor reporting of the project's status
• Unmanaged risks
• Poor communication among customers, developers, and users
• Use of immature technology
• Inability to handle the project's complexity
• Sloppy development practices
• Poor project management
• Stakeholder politics
• Commercial pressures
Is your project on the “Death March”?
Ed Yourdon, a major thinker in the field of software development, wrote in his book “Death March” that any software project is an exercise in futility if one of these key factors is 50% off a “reasonable norm”;
• Is the schedule 50% too short?
• If the staff 50% too small?
• Is the budget 50% too low?
• Is the scope of features 50% too large?
I suppose this principle could be applied many other fields of endeavour as well.
So did they all live happily every after?
On a more positive note there have been great strides made in the way that software is constructed. “Best Practices” and “Object Oriented” development models have evolved which are much better suited to handling complex systems. What I have yet to see however, is much change in the way we interact as people in organizations. Lets just hope that with better tools and methodologies at least we can look forward a software landscape with a better view.