Are there any statistics on the average number of deaths per year due to software bugs?
— Jeff (@jeffr0) May 20, 2015
The answer, Jeff, is yes. But they’re weird and hard to find.
Back around 2008 or so, my father published a paper on the death of Lisa Norris.
Don’t know who she is? Don’t worry. Most people don’t. Norris died of radiation overexposure that was determined to be caused by a software issue. People hadn’t fully tested software changes. I only know who she is because I maintain my father’s website and converted all his PDF articles to posts. In doing so, I read them all.
I’m not a mathematician like my father, and luckily he and I share a fantastic trait. We’re both used to explaining technical things to non-technical people. Or rather, we can explain the technical things to people who are cleverly technical in other arenas. Yes, that’s where I learned it.
When I saw Jeff’s tweet, I asked if he was serious (as opposed to just ruminating on Twitter) and then directed him to two of my father’s articles. First I pulled up the one about Lisa Norris, since that stuck in my memory. But then I remembered he’d written an article for the Nikkei Asian Review that was more non-mathematician readable. Knowing that he’s written the first paper gives a little more credence to the statements he makes when he talks about Death by Software.
The part that has always stuck in my mind is this quote:
As Dr. Nancy Leveson wrote in her Therac-25 investigation report: “Most accidents are system accidents; that is, they stem from complex interactions between various components and activities. To attribute a single cause to an accident is usually a serious mistake. We want to emphasize the complex nature of accidents and the need to investigate all aspects of system development and operation to understand what has happened and to prevent future accidents.”
When we talk about how software can (and will continue to) kill people, we get stymied by the considerably complexity of the question. Did Lisa Norris die because no human thought “This looks weird?” That is also why it’s hard to say “Give me the statistics on all people who died because of software failure.” We have to define what, specifically, is a software failure.
In the book (and the movies) “Fail-Safe,” we face nuclear war because a light burns out causing a false-positive alert resulting in American bombers heading to the USSR. It’s very similar to the boom “Red Alert” (and of course the movie “Dr. Stranglove”). The failure is that our fail-safe measures, the steps we take to make sure that a machine (or computer) cannot make the situation worse is nothing more than a pipe dream.
At its crux, the deaths by software are often the result of failure of imagination. I first learned of the phrase when reading about the Apollo 1 fire back in 1967. Astronaut Frank Borman spoke at the post-mortem hearings of how the deaths, the fire was born from our failure to imagine how things could go wrong.
Why don’t we have statistics? We would have to be very specific in what we ask for. How many people have died because their GPS was wrong? How many people died because the software to handle Anti-Lock Brakes failed? We do not lump all software failures together, making the research mystifying and bewildering. This is not meant as an excuse, though even to me it feels like one. We should be more transparent in how our software kills.
And rest assured. Software kills.
