Failover
Failover is the process of switching to a redundant standby when the primary fails — promoting a replica database, redirecting traffic to a healthy region, or routing around a failed instance. The hard part isn't the switch itself; it's deciding when to fail over, doing it without split-brain, and recovering when the original primary comes back. Automatic failover trades speed for risk of false positives; manual failover trades speed for safety.
How it works
Failover is the act of switching to a standby when the primary fails. The setup is always: one primary (or active) serving traffic, and one or more standbys (or replicas) waiting. When the primary is detected as failed, a standby is promoted and traffic is redirected.
The same pattern applies at every layer:
- Database failover: a read replica is promoted to primary.
- Instance failover: a load balancer routes traffic away from a dead instance to healthy ones.
- Region failover: DNS or anycast routing redirects traffic to a healthy region.
- VIP failover: a virtual IP moves from one host to another (VRRP, Keepalived).
The mechanics differ but the hard parts are universal: detection, split-brain prevention, and recovery.
The nightmare scenario in failover is split-brain: two nodes both believe they're the primary. This happens when the network partitions and the standby can't tell whether the primary is down or whether it's just unreachable from the standby's vantage point. It promotes itself. Now you have two primaries — both accepting writes — and reconciling them later is impossible without data loss.
The defense is a quorum: a consensus protocol (Raft, Paxos) requires a majority of nodes to agree on who's primary. If a node can't reach a majority, it doesn't promote itself. This is why etcd, ZooKeeper, and modern databases (CockroachDB, Spanner, YugabyteDB) use consensus for leader election.
For systems that don't use consensus (classic Postgres streaming replication), the standard trick is a witness server in a third location that breaks ties. Without it, a partition between primary and standby with no third party can cause split-brain.
Stateful failover risks data loss. If the primary accepted writes that haven't yet reached the replica, promoting the replica means those writes are lost. This is the RPO (recovery point objective) — the amount of data you can tolerate losing.
- Synchronous replication: the primary waits for the replica to acknowledge before committing. Zero data loss, but slower writes and the primary fails if the replica is slow.
- Asynchronous replication: the primary commits immediately, the replica catches up later. Faster writes, but if the primary dies before the replica catches up, data is lost.
PostgreSQL supports both: synchronous_commit=on for zero-loss, off for speed. Most production systems use a hybrid: synchronous to a local replica (within AZ), asynchronous to a remote replica (cross-region). The local replica gives fast failover with no data loss; the remote replica gives region-level disaster recovery with some RPO.
Before promoting a standby, you must guarantee the old primary is no longer accepting writes. This is called fencing or STONITH (Shoot The Other Node In The Head). Techniques include: power-cycling the old primary via a smart PDU, revoking its credentials, marking its storage read-only, or removing it from the cluster's quorum. Without fencing, a 'failed' primary that's actually still alive can keep writing and split-brain results. The classic horror story: old primary wakes up, accepts writes, then those writes are lost when the new primary catches up — and the user thinks their write was persisted.
Failover at the server side is only half the story. The client must also handle it:
- DNS failover: clients cache DNS, so they keep hitting the old primary for the TTL duration (often minutes). Set short TTLs (60s) on critical records, or use a client that respects DNS TTLs (most don't).
- Connection rerouting: when a connection breaks, the client must reconnect to the new primary. Smart drivers (PgJDBC, MySQL Connector/J) do this automatically; dumb drivers require a restart.
- Retry on failover: when a write fails because the primary changed, the client should retry against the new primary. This requires the driver to refresh its endpoint list.
The Amazon RDS pattern is illustrative: the database has a single DNS endpoint. On failover, RDS atomically updates the DNS to point at the new primary. Smart clients reconnect; dumb clients need a restart. The failover itself takes 30-60 seconds; the DNS propagation adds up to the TTL.
Your primary database is in AZ-A, with an async replica in AZ-B. The link between AZs fails. AZ-B can't reach AZ-A. What's the risk if AZ-B auto-promotes?
Pick one answer.
Why is fencing necessary before promoting a standby?
Pick one answer.
You're choosing between synchronous and asynchronous replication for a database. Which trade-off are you making?
Pick one answer.
Engineering mental model
Mental model. Think of Failover as a deliberate boundary in a system. The boundary exists because something becomes harder to manage when everything is done in one place: latency, scale, failure isolation, consistency, cost, or team ownership. The useful question is not “what does Failover mean?” but “what pressure makes this boundary worth introducing, and what new failure mode does it create?”
Before choosing Failover, name the workload, the critical user path, the dominant bottleneck, the failure you are trying to absorb, and the trade-off you are willing to accept. If you cannot name those five things, the design is probably premature.
// Pseudocode
request = receive()
result = failover(request)
return result
// Production questions:
// 1. What happens on timeout?
// 2. Can this operation be retried safely?
// 3. What is the bottleneck?Back-of-the-envelope reasoning
Numerical lens: write down traffic, payload size, read/write ratio, peak multiplier and durability target before choosing a component. The numbers should justify the architecture.
Interactive thought experiment: Failover
Change the variables below and predict what breaks first in Failover. The production lab can later reuse these same inputs.
Change one variable at a time. Predict the failure mode first, then move the slider and see whether your mental model matches the simplified system response.
If you are stuck on Failover, start by drawing the request path and marking every network hop, stateful component, queue, cache and failure boundary. Then estimate where the system will saturate.
You increase traffic by 10× in a system using Failover. What should you inspect first?
Pick one answer.
Which statement is the safest engineering habit when using Failover?
Pick one answer.
You have dashboards for traffic, latency, errors and saturation. You can change the architecture, but every change has operational cost.
Production scenario: your system uses Failover, traffic suddenly spikes, and p99 latency doubles. What is your first move?
Interview drill
Answer this without notes: When would you choose Failover, and when would you intentionally avoid it? Mention at least one bottleneck it addresses, one failure mode it introduces, and one alternative. Then quantify the workload you are designing for.
For Failover, treat the system as a control loop: observe load and failure, choose a bounded response, and measure whether the response stabilizes the system instead of simply moving the bottleneck somewhere else.
Numerical sanity check
When estimating capacity, distinguish average from peak. If average traffic is 4,000 RPS and the observed peak-to-average factor is 3×, design the first pass around roughly 12,000 RPS, then leave headroom for failure and growth.
Do not optimize for a memorized definition. Reason from the workload and failure mode.
What is the earliest signal that Failover is becoming the bottleneck: latency, saturation, errors, queue depth, or something else? Why?
Pick one answer.
What you gain, what you pay
- +Turns redundancy into actual availability — primary dies, traffic keeps flowing.
- +Automatic failover can recover from outages in seconds, before users notice.
- +Enables multi-AZ and multi-region deployments.
- +Decouples availability of any single node from the system.
- −Risk of split-brain if detection is wrong (network partitions look like failures).
- −Data loss on async replication if the replica is behind.
- −Automatic failover can cause cascading failures if it triggers too aggressively.
- −Recovery (failing back to the original primary) is often harder than the failover itself.
How this breaks in production
- Split-brain: two primaries both accepting writes after a partition.
- Data loss when the promoted replica was behind the primary.
- Failover storms: oscillating between primaries due to flapping health checks.
- Dumb clients that don't reconnect to the new primary after failover.
Don't fall into these traps
- •No fencing — old primary keeps accepting writes.
- •No quorum — a single witness can't break ties, leading to split-brain.
- •Long DNS TTLs — clients keep hitting the dead primary for minutes.
- •Never testing failover — when you need it, the runbook is wrong.
Real systems using this
How real systems implement this
- Amazon RDS Multi-AZ — Maintains a synchronous standby in a second AZ. On failure detection, RDS promotes the standby and atomically updates the DNS endpoint. Typical failover: 60-120 seconds. Application reconnects via the same DNS name.
- Patroni (PostgreSQL HA) — Uses etcd or ZooKeeper for leader election via consensus. A postgres primary is elected, others follow. Fencing is performed by removing the old primary's leader key, preventing split-brain.
Practice saying it out loud
- Q1What is split-brain, and how do you prevent it?
- Q2Compare synchronous and asynchronous replication for failover. When would you use each?
- Q3What is fencing, and why is it necessary before promoting a standby?
- Q4How does DNS-based failover work, and what's its main weakness?
Further reading & references
Core explanations are original NO CAP material. External references are provided for deeper study and standards.
What next?
Mark as understood once the mental model clicks.
Next recommended
Circuit Breaker