Single Points of Failure
A Single Point of Failure (SPOF) is any component whose failure takes down the whole system. Identifying SPOFs means walking every request path and asking 'if this one thing dies, does the system die with it?' Elimination is achieved through redundancy (N+1, active-active), graceful degradation, and failover design — but every redundancy has a cost, so you eliminate the SPOFs that matter and consciously accept the rest.
How it works
A Single Point of Failure (SPOF) is any component in a system whose failure causes the entire system (or a critical user journey) to fail. The textbook metaphor is a chain: a chain is only as strong as its weakest link, and a system is only as available as its least-redundant component.
To find SPOFs you walk every request path from client to database and back, asking at each hop: 'If this dies, does the request fail? And does it fail for everyone, or only for some users?' Components with no fallback, no replica, and no failover are SPOFs.
Common SPOFs to look for:
- Single database instance (no read replica, no standby) — the database dies, the system dies.
- Single load balancer (no failover VIP, no DNS failover).
- Single availability zone deployment — an AZ outage takes you fully offline.
- Single Redis cache with no cluster/failover — cache dies, every request hits the database.
- Single region for a global service — a region outage is a global outage.
- A required downstream service with no circuit breaker — its failure cascades.
- A shared dependency like a TLS certificate authority, DNS provider, or identity provider.
- A single hot shard — the customer whose data lives on shard 7 takes an outage if shard 7 dies.
- A single on-call engineer — bus factor of one is a human SPOF.
Eliminating a SPOF means introducing redundancy, failover, or graceful degradation. The techniques stack:
- N+1 redundancy — run N instances to handle load, plus 1 (or more) to survive one dying. For stateful components (databases, caches), this means active-passive or active-active replication with automatic failover.
- Multi-AZ deployment — every tier runs in at least two AZs. AZ outages become a non-event.
- Multi-region deployment — for global services, run active-active in two regions behind a global load balancer (Route 53, Cloudflare, GSLB). One region's failure just shifts traffic.
- Read replicas — read traffic can survive primary failure by failing over to a replica (with brief write unavailability, or via promotion).
- Circuit breakers and fallbacks — if you cannot make a dependency redundant, make its failure survivable: serve stale cache, return a default, or degrade gracefully.
- Health checks + auto-replacement — if a component dies, replace it automatically (k8s restarts, autoscaling groups).
- Decoupling with queues — if a downstream is slow, a queue between producer and consumer keeps the producer alive.
The discipline is to not stop at the first layer. People add three API replicas and feel safe, but if all three talk to one database, the database is still a SPOF. Walk every hop, every time.
A SPOF is the extreme case of large blast radius: the blast radius is the whole system. The general principle is to minimize blast radius — partition by tenant, by region, by shard — so that any single failure affects only a small slice of users. Sharded databases, cell-based architectures, and per-tenant deployments are all 'shrink the SPOF' moves. A failure that affects 1% of users for 5 minutes is far cheaper than one that affects 100% for 1 minute.
The sneakiest SPOFs are shared dependencies — services you don't own but rely on. Examples:
- DNS provider: if your DNS goes down, nobody can find you. Use a secondary DNS provider or multi-vendor DNS.
- TLS certificate authority / OCSP responder: if cert validation fails, every browser refuses to connect. Pin a backup CA.
- Identity provider (Okta, Auth0): if SSO is down, nobody can log in. Have a break-glass local account or cached tokens.
- Cloud provider itself: a single cloud is a SPOF. Multi-cloud is rarely worth it for normal services, but regulated or hyperscale services do it.
- A shared internal library: if every service uses one auth library and that library has a bug, every service is down.
- A shared on-call rotation: bus factor of one is a human SPOF.
In interviews, candidates who mention shared dependencies and break-glass procedures demonstrate production maturity. 'Multi-AZ' is the floor; 'what if AWS us-east-1 dies?' is the next level.
A service runs 10 stateless API replicas across 3 AZs, behind a multi-AZ load balancer, talking to a single PostgreSQL primary with no read replicas. Which is the strongest true statement?
Pick one answer.
Your global service runs only in us-east-1. After an AWS us-east-1 outage takes you fully offline for 4 hours, what is the most important SPOF to address first?
Pick one answer.
Engineering mental model
Mental model. Think of Single Points of Failure as a deliberate boundary in a system. The boundary exists because something becomes harder to manage when everything is done in one place: latency, scale, failure isolation, consistency, cost, or team ownership. The useful question is not “what does Single Points of Failure mean?” but “what pressure makes this boundary worth introducing, and what new failure mode does it create?”
Before choosing Single Points of Failure, name the workload, the critical user path, the dominant bottleneck, the failure you are trying to absorb, and the trade-off you are willing to accept. If you cannot name those five things, the design is probably premature.
// Pseudocode
request = receive()
result = single_points_of_failure(request)
return result
// Production questions:
// 1. What happens on timeout?
// 2. Can this operation be retried safely?
// 3. What is the bottleneck?Back-of-the-envelope reasoning
Numerical lens: write down traffic, payload size, read/write ratio, peak multiplier and durability target before choosing a component. The numbers should justify the architecture.
Interactive thought experiment: Single Points of Failure
Change the variables below and predict what breaks first in Single Points of Failure. The production lab can later reuse these same inputs.
Change one variable at a time. Predict the failure mode first, then move the slider and see whether your mental model matches the simplified system response.
If you are stuck on Single Points of Failure, start by drawing the request path and marking every network hop, stateful component, queue, cache and failure boundary. Then estimate where the system will saturate.
You increase traffic by 10× in a system using Single Points of Failure. What should you inspect first?
Pick one answer.
Which statement is the safest engineering habit when using Single Points of Failure?
Pick one answer.
You have dashboards for traffic, latency, errors and saturation. You can change the architecture, but every change has operational cost.
Production scenario: your system uses Single Points of Failure, traffic suddenly spikes, and p99 latency doubles. What is your first move?
Interview drill
Answer this without notes: When would you choose Single Points of Failure, and when would you intentionally avoid it? Mention at least one bottleneck it addresses, one failure mode it introduces, and one alternative. Then quantify the workload you are designing for.
A useful engineering lens for Single Points of Failure: define the problem it solves, the simpler design that fails first, the constraint that forces you to introduce this concept, and the new failure modes the concept creates.
Numerical sanity check
Back-of-the-envelope reasoning beats fake precision. State your traffic, payload, concurrency and growth assumptions explicitly, then calculate enough to know whether the current architecture is orders of magnitude away from the target.
Do not optimize for a memorized definition. Reason from the workload and failure mode.
Imagine the simplest version of a system using Single Points of Failure. What breaks first as traffic grows by 10×, and what would you change before reaching 100×?
Pick one answer.
What you gain, what you pay
- +Eliminating SPOFs directly increases availability — every SPOF you remove raises the ceiling on uptime.
- +Forces you to understand every request path, which improves overall system understanding.
- +Redundancy often comes with scalability as a bonus (read replicas offload reads, multi-AZ adds capacity).
- +Smaller blast radius means incidents are less severe and easier to recover from.
- −Redundancy costs money — every additional replica, AZ, or region multiplies the bill.
- −Redundancy adds operational complexity — failover logic, replication lag, split-brain scenarios.
- −Active-active multi-region is hard: data consistency, conflict resolution, write routing.
- −You can never eliminate all SPOFs — at some point you accept residual risk (e.g., single cloud, single planet).
How this breaks in production
- Replica exists but failover doesn't work — a 'redundant' database with no tested failover procedure is a SPOF you falsely believe is safe.
- Replica exists but is too far behind — failover to a stale replica loses committed data.
- Multi-AZ but shared control plane — all replicas die together when the control plane dies (e.g., shared k8s control plane, shared DNS).
- Split-brain — both nodes think they're primary, causing data divergence.
- Failover storm — primary flaps, failover triggers repeatedly, replicas can't catch up.
- Hidden shared dependency — three 'independent' services all depend on the same Redis cluster, so it's actually a SPOF for all three.
Don't fall into these traps
- •Stopping at the obvious layer — 'we have 3 replicas' is meaningless if they all depend on one database.
- •Treating 'multi-AZ' as a guarantee — multi-AZ protects against AZ failure, not against region or control-plane failure.
- •Not testing failover — untested failover is no failover. Chaos engineering (GameDays, fault injection) exists for this reason.
- •Forgetting about stateless vs stateful — stateless services are trivially redundant; stateful ones (databases, caches) require careful replication.
- •Confusing N replicas with N+1 redundancy — N replicas handling N×load has zero redundancy; you need capacity for N+1.
- •Ignoring human SPOFs — single on-call, single maintainer, runbook only in someone's head.
Real systems using this
How real systems implement this
- AWS Multi-AZ RDS — Amazon RDS Multi-AZ maintains a synchronous standby in another AZ and automatically fails over to it if the primary dies — eliminating the database as a SPOF for the most common failure mode.
- Netflix Chaos Monkey / Simian Army — Netflix engineers random instance termination in production to verify that no single instance is a SPOF. If termination causes user impact, the SPOF is real and must be fixed.
- Cloudflare's multi-vendor DNS — Cloudflare uses multiple DNS providers and anycast routing so that no single DNS provider's outage takes them offline — an explicit defense against DNS-as-SPOF.
Practice saying it out loud
- Q1Walk me through a system you designed and identify every single point of failure. What would you do to eliminate each one?
- Q2Your service is deployed in a single region with multi-AZ. What outages can still take you fully offline, and what would you do about them?
- Q3How do you decide when to go from single-AZ to multi-AZ, and from multi-AZ to multi-region? What's the cost-benefit?
- Q4What's the difference between a SPOF and a bottleneck? Give an example of each.
- Q5Your service depends on a third-party API (e.g., Stripe). They have an outage. What can you do to not be a SPOF victim?
Further reading & references
Core explanations are original NO CAP material. External references are provided for deeper study and standards.
What next?
Mark as understood once the mental model clicks.
Next recommended
Failure Analysis