What Breaks Our Systems: A Taxonomy of Black Swans
Central idea
Catastrophic outages are hard to predict in detail but often share recurring shapes that can be tested, monitored, and contained in advance.
Summary
Black-swan incidents are not ordinary failures with familiar recovery paths. They emerge from hidden correlations and can keep an entire service down. Nolan's taxonomy replaces prediction with preparation. Hidden logical or storage limits require growth-oriented tests and early monitoring. Slow dependencies spread through saturated pools and queues, so interactions need useful timeouts, bounded retries, and load shedding. Coordinated demand should be assumed. Dangerous automation needs independent constraints and a tested stop control. Segmentation limits blast radius, while cold-start drills expose dependency loops before a real recovery depends on them.
Key ideas
Test the operational system
Capacity tests must include data growth, writes, startup, backup, restart, and control-plane behavior—not only request throughput.
Contain correlation
Timeouts, retry budgets, circuit breakers, load shedding, and segmentation stop one stressed component from consuming the rest.
Constrain automation independently
A controller capable of changing production needs limits and a shutdown path that do not depend on its own judgment remaining correct.
Six recurring outage shapes
Hidden limits are reached.
Slowness saturates pools and queues.
Demand arrives in a coordinated herd.
Automation amplifies a bad signal.
One failure crosses a large blast radius.
Dependency loops prevent cold recovery.
Why it matters now
Modern systems contain more services, automation, and AI-driven actions. Reliability increasingly depends on containing correlated failure rather than preventing every local mistake.
Continue with the original
This short version preserves the main argument. Follow the source for the complete talk, article, or book.