I'm a broken record on how important I think this document is, and that it's hard to appreciate it until you've had extended experience with complex systems actually failing.
The most commonly cited subtext or thrust of it is that "root cause analysis", at least on complex systems, is a fools errand. Something goes wrong, say, in a distributed lock system, and your whole deployment system enters a metastable failure state. Naturally, the "root cause" seems like lock system resiliency. But definitionally a metastable failure is one that persists after the inciting condition is resolved. Now you have two "root causes", the lock failure and the metastability of the deployment system fault. Keep looking and you'll find more.
But to me the biggest brick to the forehead in this piece is further observation that random things are failing all the time in any complex system. "Complex systems run in degraded mode". Resilient components are good, but it's the resiliency of the overall process that orchestrates the whole system that determines whether things are going to blow up.
All practitioner actions are gambles. I should have that inked somewhere.
I'm a broken record on how important I think this document is, and that it's hard to appreciate it until you've had extended experience with complex systems actually failing.
If you want a bit of cheese to go with that wine, this article pairs nicely with The Grug-Brained Developer: https://grugbrain.dev/
The most commonly cited subtext or thrust of it is that "root cause analysis", at least on complex systems, is a fools errand.
It's only an issue when people take that phrase very literally. People have common sense to understand that things have multiple causes and a chain of events. NASA has "Root Cause Analysis" (singular) all over various official documentation and it doesn't stop them from understanding that the failed O-rings were not the single root cause of the Challenger explosion. Another cause was management normalizing the deviations of previous unsafe datapoints of prior launches which let them greenlight the launch in freezing temperatures. Another cause was the unrealistic flight schedules which can subconsciously pressure management into normalizing dangerous deviations. It wasn't The Rogers Commission that found the multiple causes; it was NASA engineers and management themselves explaining the multiple causes as they were interviewed by the Rogers Commission members.
For whatever reason, alternative jargon such as "Root Causes Analysis" (plural) or "Proximate and Distal Causes Analysis" isn't as widely used.
Another cause was the unrealistic flight schedules which can subconsciously pressure management into normalizing dangerous deviations.
In the case of Challenger, I think it's pretty clear this was the "root cause":
According to testimony by Kilminster and Boisjoly, Mason finally turned to Bob Lund and said, "Take off your engineering hat and put on your management hat." Joe Kilminster wrote out the new recommendation and went back on line with the teleconference.
The new recommendation stated that the cold was still a safety concern, but their people had found that the original data was indeed inconclusive and their "engineering assessment" was that launch was recommended, even though the engineers had no part in writing the new recommendation and refused to sign it.
If you want to take the "system" view here, as is often the case, it is the organizational power structure and incentives therein that comprise the dangerous system. You had engineering experts easily predicting the disaster, but they had no decision making power. That was the problem. But if you set up an organization like that, where the egos of "get it done" managers are allowed to gamble with other people's lives to win their own accolades, the system is doomed from the start.
If you’re doing “five* whys” analyses correctly, you don’t stop at the technical causes. You continue further to analyze the causes that precipitated the technical errors, too. This exposes the business reasons behind them and forces management to face them.
* the number five isn’t magical here. You don’t have to stop at five, and you often shouldn’t.
A deeper cause there, as I understand it, is that the statistical analysis was incomplete/flawed but in a way that none of the technical practitioners caught and called out at the time. The way the data was presented was to the effect of "half of the o-ring damage incidents were in cold temp launches", which made things seem less bad. The way it could have been presented was more like "nearly every cold temp launch has resulted in damage, only like 10% of others do" (I forget the exact numbers) which is far more effective highlighting the impact of the temperature and the magnitude of the increased risk. And that would've let people make a stronger case "hey, we know temps in the 50s cause damage almost every time, and it's way colder today."
There's a "system" aspect (outside of power structures and incentives) which is that prob/stats knowledge among almost all engineering disciplines (industrial engineering waves from the sidelines) is exceedingly poor and often viewed as "soft" and "less important" than calculus, linear algebra, etc. And so practitioners are ill-equipped at spotting things like that ("wait, is this the right denominator? what about frequency of incident?").
“Root cause analysis”, like “single point of failure”, is often misunderstood. A root cause analysis often finds multiple root causes. And a system can have multiple single points of failure.
"The system continues to function because it contains so many redundancies and because people can make it function, despite the presence of many flaws. After accident reviews nearly always note that the system has a history of prior ‘proto-accidents’ that nearly generated catastrophe. Arguments that these degraded conditions should have been recognized before the overt accident are usually predicated on naïve notions of system performance. System operations are dynamic, with components (organizational, human, technical) failing and being replaced continuously."
This very much resembles Admiral Cloudberg's write-up of the National Airport collision:
"No human being can look at a complex system and predict with any degree of accuracy how exactly it will fail. But with sufficient data, prediction becomes possible because of something known in the occupational safety field as Heinrich’s Law, which states that there are approximately 300 “near misses” for every serious accident; or as the latest version of the law puts it, each fatal accident is accompanied by about 3,000 near misses and about 30,000 “at-risk behaviors.”[39] Statistically, hundreds or thousands of at-risk behaviors and near misses are likely to occur prior to the first fatal accident, providing an opportunity to identify the risk before lives are lost."
This is very important work. If it interests you, and you want to dig further, it's broadly called "Safety II". You probably want to read Erik Hollnagel [1] or Sydney Dekker. I put together a presentation once about how this translates to IT security [2]. But broadly:
Safety I Learn from Errors, Safety defined by absence,Reactive approach, Understanding what goes wrong, Accident causation models, Avoidance of errors, Reducing losses
Safety II Learn from successes, Safety defined by presence, Proactive approach, Understanding what goes right, Repeat what goes right, Enforce successful behaviours, Create new processes based on successful behaviour
If anything, think long and hard about this: "Post-accident attribution to a ‘root cause’ is fundamentally wrong." It's a true statement. If you truly understand this sentence, you have mastered Safety II to the level that a light bulb will come on, and you won't be able to un-see it. This light bulb is a curse, a beautiful curse.
"Catastrophe requires multiple failures – single point failures are not enough."
I disagree with this point. From what I have seen and experienced, there are so many SPOFs in the real-world complex systems that do not have adequate defense or even any defense at all. The only reason catastrophe didn't occur (often) is only because the natural failure rate is extremely low. Like the probability of air traffic controller making a critical mistake is very low, maybe once every few years, but when it happens, the SPOF will almost certainly cause a disaster, and the system (ie. the way air traffic is controlled today) has been around for decades and is so outdated but the cost of changing the entire process around the world is simply way too high so we just take this as "normal".
The operators know complex systems work. Words I've spoken, heard, or both:
"Just rerun it, works the second time."
"That's been failing for a while now. Just ignore the alert."
"This procedure is old. We've added a step 64b in pencil. I can't remember why."
"Is it meant to sound like that?"
"What's the tape for?"
"Careful you don't bump this button."
"The database name has changed, the post-it has the new one. That's a '1' by the way."
"Accept the failure, keep going."
"Because you're the person touching the system the most, you're the most likely person to break it."
"I know it says call on failure, but they never answer."
All of this sounds just like any air crash investigation I ever read
The definitive work on this topic is Normal Accidents, with a modern retelling in Meltdown.
https://en.wikipedia.org/wiki/Normal_Accidents
https://en.wikipedia.org/wiki/Meltdown_(Clearfield_and_Tilcs...
I'm a broken record on how important I think this document is, and that it's hard to appreciate it until you've had extended experience with complex systems actually failing.
The most commonly cited subtext or thrust of it is that "root cause analysis", at least on complex systems, is a fools errand. Something goes wrong, say, in a distributed lock system, and your whole deployment system enters a metastable failure state. Naturally, the "root cause" seems like lock system resiliency. But definitionally a metastable failure is one that persists after the inciting condition is resolved. Now you have two "root causes", the lock failure and the metastability of the deployment system fault. Keep looking and you'll find more.
But to me the biggest brick to the forehead in this piece is further observation that random things are failing all the time in any complex system. "Complex systems run in degraded mode". Resilient components are good, but it's the resiliency of the overall process that orchestrates the whole system that determines whether things are going to blow up.
All practitioner actions are gambles. I should have that inked somewhere.
If you want a bit of cheese to go with that wine, this article pairs nicely with The Grug-Brained Developer: https://grugbrain.dev/
It's only an issue when people take that phrase very literally. People have common sense to understand that things have multiple causes and a chain of events. NASA has "Root Cause Analysis" (singular) all over various official documentation and it doesn't stop them from understanding that the failed O-rings were not the single root cause of the Challenger explosion. Another cause was management normalizing the deviations of previous unsafe datapoints of prior launches which let them greenlight the launch in freezing temperatures. Another cause was the unrealistic flight schedules which can subconsciously pressure management into normalizing dangerous deviations. It wasn't The Rogers Commission that found the multiple causes; it was NASA engineers and management themselves explaining the multiple causes as they were interviewed by the Rogers Commission members.
For whatever reason, alternative jargon such as "Root Causes Analysis" (plural) or "Proximate and Distal Causes Analysis" isn't as widely used.
In the case of Challenger, I think it's pretty clear this was the "root cause":
-- https://onlineethics.virginia.edu/cases/engineering-ethics-c...
If you want to take the "system" view here, as is often the case, it is the organizational power structure and incentives therein that comprise the dangerous system. You had engineering experts easily predicting the disaster, but they had no decision making power. That was the problem. But if you set up an organization like that, where the egos of "get it done" managers are allowed to gamble with other people's lives to win their own accolades, the system is doomed from the start.
If you’re doing “five* whys” analyses correctly, you don’t stop at the technical causes. You continue further to analyze the causes that precipitated the technical errors, too. This exposes the business reasons behind them and forces management to face them.
* the number five isn’t magical here. You don’t have to stop at five, and you often shouldn’t.
A deeper cause there, as I understand it, is that the statistical analysis was incomplete/flawed but in a way that none of the technical practitioners caught and called out at the time. The way the data was presented was to the effect of "half of the o-ring damage incidents were in cold temp launches", which made things seem less bad. The way it could have been presented was more like "nearly every cold temp launch has resulted in damage, only like 10% of others do" (I forget the exact numbers) which is far more effective highlighting the impact of the temperature and the magnitude of the increased risk. And that would've let people make a stronger case "hey, we know temps in the 50s cause damage almost every time, and it's way colder today."
There's a "system" aspect (outside of power structures and incentives) which is that prob/stats knowledge among almost all engineering disciplines (industrial engineering waves from the sidelines) is exceedingly poor and often viewed as "soft" and "less important" than calculus, linear algebra, etc. And so practitioners are ill-equipped at spotting things like that ("wait, is this the right denominator? what about frequency of incident?").
“Root cause analysis”, like “single point of failure”, is often misunderstood. A root cause analysis often finds multiple root causes. And a system can have multiple single points of failure.
"The system continues to function because it contains so many redundancies and because people can make it function, despite the presence of many flaws. After accident reviews nearly always note that the system has a history of prior ‘proto-accidents’ that nearly generated catastrophe. Arguments that these degraded conditions should have been recognized before the overt accident are usually predicated on naïve notions of system performance. System operations are dynamic, with components (organizational, human, technical) failing and being replaced continuously."
This very much resembles Admiral Cloudberg's write-up of the National Airport collision:
"No human being can look at a complex system and predict with any degree of accuracy how exactly it will fail. But with sufficient data, prediction becomes possible because of something known in the occupational safety field as Heinrich’s Law, which states that there are approximately 300 “near misses” for every serious accident; or as the latest version of the law puts it, each fatal accident is accompanied by about 3,000 near misses and about 30,000 “at-risk behaviors.”[39] Statistically, hundreds or thousands of at-risk behaviors and near misses are likely to occur prior to the first fatal accident, providing an opportunity to identify the risk before lives are lost."
https://admiralcloudberg.medium.com/reaping-the-whirlwind-in...
This is very important work. If it interests you, and you want to dig further, it's broadly called "Safety II". You probably want to read Erik Hollnagel [1] or Sydney Dekker. I put together a presentation once about how this translates to IT security [2]. But broadly:
Safety I Learn from Errors, Safety defined by absence,Reactive approach, Understanding what goes wrong, Accident causation models, Avoidance of errors, Reducing losses
Safety II Learn from successes, Safety defined by presence, Proactive approach, Understanding what goes right, Repeat what goes right, Enforce successful behaviours, Create new processes based on successful behaviour
If anything, think long and hard about this: "Post-accident attribution to a ‘root cause’ is fundamentally wrong." It's a true statement. If you truly understand this sentence, you have mastered Safety II to the level that a light bulb will come on, and you won't be able to un-see it. This light bulb is a curse, a beautiful curse.
[1] https://safety4sea.com/cm-safety-i-vs-safety-ii-an-overview [2] https://www.msoos.org/largefiles/safety2.pdf
Whenever I see this [excellent] paper, I am always struck by the first sentence, in the first section:
(Emphasis mine)
I am not sure if that's a typo, or some writing convention that my uneducated mind can't grok.
Is it not merely an example of such failure?
"Catastrophe requires multiple failures – single point failures are not enough."
I disagree with this point. From what I have seen and experienced, there are so many SPOFs in the real-world complex systems that do not have adequate defense or even any defense at all. The only reason catastrophe didn't occur (often) is only because the natural failure rate is extremely low. Like the probability of air traffic controller making a critical mistake is very low, maybe once every few years, but when it happens, the SPOF will almost certainly cause a disaster, and the system (ie. the way air traffic is controlled today) has been around for decades and is so outdated but the cost of changing the entire process around the world is simply way too high so we just take this as "normal".
The operators know complex systems work. Words I've spoken, heard, or both: "Just rerun it, works the second time." "That's been failing for a while now. Just ignore the alert." "This procedure is old. We've added a step 64b in pencil. I can't remember why." "Is it meant to sound like that?" "What's the tape for?" "Careful you don't bump this button." "The database name has changed, the post-it has the new one. That's a '1' by the way." "Accept the failure, keep going." "Because you're the person touching the system the most, you're the most likely person to break it." "I know it says call on failure, but they never answer."
a nice compliment is The Myth of Progress by Tom Wessels. Preview here: https://www.perlego.com/book/2236739/the-myth-of-progress-to...