Failure Analysis
Failure analysis is the discipline of asking "what will break?" before something does. It identifies single points of failure (SPOFs), cascading failure paths, and bottlenecks under load. Techniques: component-by-component review ("what if this dies?"), dependency mapping, capacity analysis ("what if traffic doubles?"), and game days (deliberate failures in production). The output isn't just a list of risks — it's a prioritized plan for what to harden first.
How it works
Failure analysis answers three questions about a system:
- What can fail? — enumerate every component and ask what happens when it dies.
- How does the failure propagate? — does it take down other components? Does it cascade?
- How do we mitigate? — redundancy, circuit breakers, fallbacks, failover.
The systematic technique is the component failure review: list every component (load balancer, web server, database, cache, queue, downstream API), then for each, ask: "If this dies, what happens to the user? What happens to the rest of the system? How long does it take to recover?"
A system that survives the component failure review is robust. A system that hasn't been through one will fail at 3am in surprising ways.
A single point of failure (SPOF) is a component whose failure takes down the whole system. The classic example: a single database with no replica. When it dies, the service is down until you restore from backup.
Identify SPOFs by tracing the request path and asking "is there redundancy here?" For each component:
- Multiple instances? (load balancer, web servers)
- Replicas? (database, cache)
- Failover plan? (DNS, region)
- Health-checked and rerouted? (every component)
Eliminate SPOFs by adding redundancy. But beware: redundancy that's not tested is theoretical. A replica that's never been promoted will fail when you need it (configuration drift, broken replication, missing credentials). Test failover regularly — chaos engineering (Game Days) exists for this reason.
Common SPOFs that get missed:
- DNS: a single DNS provider is a SPOF. Use multiple providers or an authoritative anycast service.
- TLS cert management: an expired cert takes down the service. Automate renewal and alert on expiry.
- Single cloud region: even multi-AZ doesn't survive a region outage. Multi-region for true resilience.
- Shared dependencies: if every service depends on the same Redis cluster, that cluster is a SPOF.
A cascading failure is when a failure in one component triggers failures in others, spreading outward. The classic pattern:
- A downstream service (DB, third-party API) gets slow.
- Callers hold threads/connections waiting for it.
- Caller thread pools exhaust; callers become unresponsive.
- Services that depend on the callers hold their threads.
- The cascade spreads outward through the dependency graph.
Defenses:
- Timeouts: every call has a deadline. Slow dependencies fail fast, freeing resources.
- Circuit breakers: sustained failures trip the breaker; subsequent calls fail immediately instead of waiting.
- Bulkheads: separate thread pools per downstream service so a slow one doesn't consume all threads.
- Back-pressure: when a queue grows, signal upstream to slow down rather than pile up work.
- Graceful degradation: when a dependency fails, serve a fallback instead of crashing.
The combination matters. Timeouts alone cause retry storms. Circuit breakers alone fail too aggressively. The pattern is: timeout on every call, circuit breaker on sustained failure, fallback when the breaker is open. This is the resilience stack.
A well-designed system limits the impact of any single failure. The technique is bulkheading — partitioning resources so a failure in one partition doesn't consume resources in another. Examples: separate thread pools per downstream service; separate database instances per tenant (or per shard); separate clusters per region. The goal is that the failure of any single component takes down only that component's users, not everyone. Netflix's chaos engineering tests this: they deliberately kill instances and verify the system degrades gracefully rather than cascading. Smaller blast radius = smaller incident = faster recovery.
Failure analysis also asks: "what happens when traffic doubles?" — capacity analysis.
For each component:
- Current utilization: CPU, memory, disk, network, queue depth.
- Headroom: how much can it grow before saturating?
- Saturation behavior: when it saturates, does it degrade gracefully or crash?
- Scaling plan: can it scale horizontally? How fast? Automatically?
Common findings:
- The database is the bottleneck (single primary, can't scale writes horizontally without sharding).
- A cache is undersized (hit rate drops, DB load spikes).
- A connection pool is too small (requests queue when traffic spikes).
- A single hot shard takes disproportionate load (key distribution problem).
- Network egress is the limit (a CDN would help).
The output: a capacity plan — what to scale, when, and how. "At 2x current traffic, we'd need to add 5 web servers and shard the database. At 5x, multi-region."
Your service has a single Redis cache (no replica) holding session data. Failure analysis reveals this is a SPOF. What's the risk and the mitigation?
Pick one answer.
A downstream API gets slow. Your service has no timeouts and no circuit breaker. What's the cascade?
Pick one answer.
What's the difference between a single point of failure (SPOF) and a bottleneck?
Pick one answer.
Engineering mental model
Mental model. Think of Failure Analysis as a deliberate boundary in a system. The boundary exists because something becomes harder to manage when everything is done in one place: latency, scale, failure isolation, consistency, cost, or team ownership. The useful question is not “what does Failure Analysis mean?” but “what pressure makes this boundary worth introducing, and what new failure mode does it create?”
Before choosing Failure Analysis, name the workload, the critical user path, the dominant bottleneck, the failure you are trying to absorb, and the trade-off you are willing to accept. If you cannot name those five things, the design is probably premature.
// Pseudocode
request = receive()
result = failure_analysis(request)
return result
// Production questions:
// 1. What happens on timeout?
// 2. Can this operation be retried safely?
// 3. What is the bottleneck?Back-of-the-envelope reasoning
Numerical lens: write down traffic, payload size, read/write ratio, peak multiplier and durability target before choosing a component. The numbers should justify the architecture.
Interactive thought experiment: Failure Analysis
Change the variables below and predict what breaks first in Failure Analysis. The production lab can later reuse these same inputs.
Change one variable at a time. Predict the failure mode first, then move the slider and see whether your mental model matches the simplified system response.
If you are stuck on Failure Analysis, start by drawing the request path and marking every network hop, stateful component, queue, cache and failure boundary. Then estimate where the system will saturate.
You increase traffic by 10× in a system using Failure Analysis. What should you inspect first?
Pick one answer.
Which statement is the safest engineering habit when using Failure Analysis?
Pick one answer.
You have dashboards for traffic, latency, errors and saturation. You can change the architecture, but every change has operational cost.
Production scenario: your system uses Failure Analysis, traffic suddenly spikes, and p99 latency doubles. What is your first move?
Interview drill
Answer this without notes: When would you choose Failure Analysis, and when would you intentionally avoid it? Mention at least one bottleneck it addresses, one failure mode it introduces, and one alternative. Then quantify the workload you are designing for.
A useful engineering lens for Failure Analysis: define the problem it solves, the simpler design that fails first, the constraint that forces you to introduce this concept, and the new failure modes the concept creates.
Numerical sanity check
Back-of-the-envelope reasoning beats fake precision. State your traffic, payload, concurrency and growth assumptions explicitly, then calculate enough to know whether the current architecture is orders of magnitude away from the target.
Do not optimize for a memorized definition. Reason from the workload and failure mode.
Imagine the simplest version of a system using Failure Analysis. What breaks first as traffic grows by 10×, and what would you change before reaching 100×?
Pick one answer.
What you gain, what you pay
- +Finds failure modes before they happen in production.
- +Forces explicit decisions about redundancy, failover, and graceful degradation.
- +Produces a prioritized hardening plan.
- +Essential interview skill — every system design interview touches it.
- −Time-consuming — full analysis takes days for a complex system.
- −Findings need follow-through — analysis without fixes is documentation.
- −Untested mitigations are theoretical — failover drills are essential.
- −Doesn't catch novel failure modes (the ones no one predicted).
How this breaks in production
- Analysis without action — risks documented but never fixed.
- Mitigations never tested — failover fails when needed.
- Missing dependencies — a shared component (single Redis) treated as not-a-SPOF.
- Novel failure modes not anticipated by the analysis.
Don't fall into these traps
- •Reviewing only the happy path, not failure scenarios.
- •Forgetting shared dependencies (single DNS provider, single Redis).
- •Not testing failover — assuming replicas work without promoting them.
- •Treating cascading failures as independent outages.
Real systems using this
How real systems implement this
- Netflix Chaos Monkey — Deliberately kills production instances to test that the system survives. Failure analysis is continuous and tested, not theoretical. Surfaces SPOFs and missing failover paths before users notice.
- AWS Well-Architected Framework Reliability Pillar — Structured failure analysis checklist used in design reviews. Covers SPOFs, cascades, capacity, disaster recovery. AWS's playbook for ensuring services meet reliability expectations.
Practice saying it out loud
- Q1What's a single point of failure? Walk through identifying them in a system.
- Q2Describe a cascading failure. How do you prevent it?
- Q3What's the difference between a SPOF and a bottleneck?
- Q4How would you do a failure analysis for an e-commerce checkout system?
Further reading & references
Core explanations are original NO CAP material. External references are provided for deeper study and standards.
What next?
Mark as understood once the mental model clicks.
Next recommended
Single Points of Failure