been reading "how complex systems fail" by richard cook, 18 short points from a doctor who studied why things break in hospitals. it was written in the late 90s and it reads like a postmortem of every outage i've ever been part of.
the one that hit me: there's no such thing as a root cause. catastrophe needs several small failures lining up, each harmless on its own. "root cause" is mostly our social need to point at someone. yesterday the us grounded every flight for hours and the official answer was "old equipment." sure. and also everything else that quietly had to go wrong at the same time.
my other favorite: complex systems always run in degraded mode. everything is a little broken all the time, and it works because people keep patching it on the fly. that's not a bug report, that's my homelab's biography.
the uncomfortable takeaway is that safety isn't a part you buy. it's something people create every day, and the "fix" after an accident usually just adds more complexity for the next one.
❤ 101 likes · 🔁 79 reposts · 💬 8 replies
-
Hi Pablo, Please share or doante to help us.
-
channeling your inner @norootcause ?
-
Reminds me of this talk on how the confluence of factors caused a global outage of Indeed back in 2010. https://www.youtube.com/watch?v=cuq-oDt4Hfc
-
I like this thread, thanks. I was thinking that small-c communities keep systems going all the time, and breakdowns might occur where communities and trust fail.
-
It is such a great treatise. I had a boss of mine - in adtech no less - who gave me a copy in 2013 and I was never the same.
-
we probably should use "last straw" instead...
-
Have you read "Breaking the mishap chain". ?
-
cybernetics
