Availability Patterns — Failover, Replication, Redundancy
Availability patterns are the architectural moves you make so that the failure of any single component does not take down the system. The core ideas are redundancy (multiple copies of everything), replication (kept in sync), failover (automatic switch when one copy dies), and graceful degradation (serving partial results when dependencies degrade). High availability is not a property you add at the end — it is a property you design in from the first box.
Foundational.
How it works
Availability is the probability a request succeeds. It is measured in 'nines': 99% uptime (two nines) allows 7.2 hours of downtime per month; 99.99% (four nines) allows 4.3 minutes; 99.999% (five nines) allows 26 seconds. Each additional nine is exponentially harder to earn because you have to eliminate every plausible failure mode, including ones that happen simultaneously.
The patterns that get you there are built on three primitives:
- Redundancy: deploy multiple copies of every component (servers, AZs, regions, network paths, power feeds).
- Replication: keep the stateful copies synchronized (sync for strong consistency, async for low latency).
- Failover: automatically detect a failed component and reroute traffic to a healthy one.
Layered on top are two deployment topologies: active-passive (one serves, the other waits) and active-active (both serve, doubling capacity and removing the failover step). The choice is the single biggest architectural decision after the consistency model.
Failover is the act of switching traffic from a failed component to a healthy one. The simplest case is a single load balancer in front of N app servers; if one server fails its health check, the LB stops sending it traffic and the system keeps running. This is N+1 redundancy — you can lose one server and still serve full load.
Stateful failover is harder. For a database, you need a clear promotion path: the primary dies, a replica is promoted, the load balancer points clients at the new primary. The four things that can go wrong:
- Split-brain: the network partition is partial; the old primary is still alive but unreachable from the new one. Two primaries accept writes; data diverges; on heal, one side loses writes. Mitigation: a fencing mechanism (STONITH — shoot the other node in the head) or quorum-based leader election (Raft, Paxos).
- Data loss: if replication was asynchronous, the replica may not have the latest writes. Promoting it loses them. Mitigation: synchronous replication for the data that matters (at the cost of write latency).
- Failover storm: a flapping primary causes repeated failovers. Mitigation: hysteresis — require N consecutive failed health checks before failing over, and a cooldown before failing back.
- Failover that nobody tested: the failover configuration works in theory but has never been run. Mitigation: chaos engineering — kill a primary in production regularly (Netflix's Chaos Monkey).
The metrics that matter for failover are RTO (Recovery Time Objective — how long until service is restored) and RPO (Recovery Point Objective — how much data you can lose). Active-passive synchronous replication gets you RTO seconds, RPO zero. Asynchronous replication gets you RTO seconds, RPO minutes-to-hours. Active-active gets you RTO zero, RPO zero — if you can solve the conflict problem.
Replication is how you keep redundant copies in sync. Three topologies cover most production systems:
-
Single-leader (primary-secondary): one node accepts writes, replicates to followers. Reads can hit followers. Failover promotes a follower. Simple, predictable, but the leader is a write bottleneck and a single point of write failure. Used by PostgreSQL streaming replication, MySQL, MongoDB replica sets.
-
Multi-leader: multiple nodes accept writes and replicate to each other. Removes the bottleneck, allows writes during partitions, but requires conflict resolution (last-write-wins, vector clocks, application merge). Risk of split-brain and write conflicts. Used by CouchDB, DynamoDB global tables, Cassandra.
-
Quorum-based: no single leader; writes go to a quorum of N nodes. Reads can use a quorum too. Linearizable if R + W > N. Used by Dynamo-style systems and consensus systems (etcd, Spanner).
Replication synchronicity is the second axis. Synchronous replication waits for the replica to ack before returning success — zero data loss on failover, but write latency = slowest replica. Asynchronous replication returns success after the local write — low latency, but the replica may lag, and a failover loses unreplicated writes. Most production databases (Postgres, MySQL) use a hybrid: synchronous to one local replica for durability, asynchronous to remote replicas for disaster recovery.
The realistic pattern for high availability is: synchronous replication within a region (multi-AZ, RPO zero, RTO seconds), asynchronous replication across regions (multi-region, RPO minutes-to-hours, RTO near-zero by promoting the remote). This is what AWS Multi-AZ RDS does, what Cloudflare does for its data plane, and what most SaaS databases do.
High availability is multiplicative: if any single layer has only one copy, the whole system has one copy. Production HA designs add redundancy at every layer — multiple load balancers (with VRRP/keepalived), multiple AZs for app servers, multi-AZ synchronous database replicas, multiple DNS providers, multiple ISPs for the datacenter, dual power feeds, generator backup. The weakest layer determines your real availability.
The named patterns you should be able to identify in an interview:
- Hot standby: a replica is running and up-to-date, takes over in seconds. Cost: the standby is mostly idle.
- Warm standby: a replica is running but lagging; takes over in seconds-to-minutes after catching up. Used for disaster recovery.
- Cold standby: a replica is configured but not running; takes minutes-to-hours to start. Cheapest, slowest.
- Active-active: both replicas serve traffic. Highest availability, hardest to make consistent.
- Multi-AZ: deploy in multiple availability zones of one cloud region. Survives a zone failure (the most common failure). Latency cost: minimal (single-digit ms between AZs).
- Multi-region: deploy in multiple regions. Survives a region failure (the catastrophic case). Latency cost: significant for synchronous replication (cross-region is 30-100ms+); usually asynchronous.
- Pilot light: a minimal version runs in the second region; scale up on failover. Middle ground between warm standby and full multi-region.
- Graceful degradation: when a dependency fails, the system returns partial results (e.g., hide recommendations if the rec service is down, but still serve the page). Better than a hard error.
The realistic answer is almost always: multi-AZ active-passive for the database (sync within region), multi-AZ active-active for the stateless app tier (just deploy N copies), and async multi-region replication for disaster recovery. Reserve full multi-region active-active for systems that genuinely need it — global write-anywhere products like Notion or Google Docs — because the conflict resolution complexity is high.
Your database uses synchronous replication to one local replica in another AZ. You need to survive a full region outage without losing data. What is the smallest change that gets you there?
Pick one answer.
What is the primary risk of an active-active multi-region database using last-write-wins conflict resolution?
Pick one answer.
You want 99.99% availability (four nines, ~4.3 minutes downtime per month). Which combination is necessary but not sufficient?
Pick one answer.
Engineering mental model
Mental model. Think of Availability Patterns — Failover, Replication, Redundancy as a deliberate boundary in a system. The boundary exists because something becomes harder to manage when everything is done in one place: latency, scale, failure isolation, consistency, cost, or team ownership. The useful question is not “what does Availability Patterns — Failover, Replication, Redundancy mean?” but “what pressure makes this boundary worth introducing, and what new failure mode does it create?”
Before choosing Availability Patterns — Failover, Replication, Redundancy, name the workload, the critical user path, the dominant bottleneck, the failure you are trying to absorb, and the trade-off you are willing to accept. If you cannot name those five things, the design is probably premature.
// Pseudocode
request = receive()
result = availability_patterns(request)
return result
// Production questions:
// 1. What happens on timeout?
// 2. Can this operation be retried safely?
// 3. What is the bottleneck?Back-of-the-envelope reasoning
Example: with 5 replicas, a majority quorum is 3. Losing 2 replicas still leaves a majority available for a quorum-based protocol.
Interactive thought experiment: Availability Patterns — Failover, Replication, Redundancy
Change the variables below and predict what breaks first in Availability Patterns — Failover, Replication, Redundancy. The production lab can later reuse these same inputs.
Change one variable at a time. Predict the failure mode first, then move the slider and see whether your mental model matches the simplified system response.
If you are stuck on Availability Patterns — Failover, Replication, Redundancy, start by drawing the request path and marking every network hop, stateful component, queue, cache and failure boundary. Then estimate where the system will saturate.
You increase traffic by 10× in a system using Availability Patterns — Failover, Replication, Redundancy. What should you inspect first?
Pick one answer.
Which statement is the safest engineering habit when using Availability Patterns — Failover, Replication, Redundancy?
Pick one answer.
You have dashboards for traffic, latency, errors and saturation. You can change the architecture, but every change has operational cost.
Production scenario: your system uses Availability Patterns — Failover, Replication, Redundancy, traffic suddenly spikes, and p99 latency doubles. What is your first move?
Interview drill
Answer this without notes: When would you choose Availability Patterns — Failover, Replication, Redundancy, and when would you intentionally avoid it? Mention at least one bottleneck it addresses, one failure mode it introduces, and one alternative. Then quantify the workload you are designing for.
For Availability Patterns - Failover, Replication, Redundancy, start with access patterns rather than brand names. Identify the dominant reads/writes, data relationships, consistency requirements, partition key, hot keys and failure behavior before choosing a storage strategy.
Numerical sanity check
A rough capacity check: required write throughput ≈ peak writes/s × average record size. At 5,000 writes/s and 2 KB average payloads, the raw incoming data stream is about 10 MB/s before indexes, replication and overhead.
Do not optimize for a memorized definition. Reason from the workload and failure mode.
A team proposes Availability Patterns - Failover, Replication, Redundancy because it 'scales'. What workload characteristic would make that choice a poor fit?
Pick one answer.
What you gain, what you pay
- +Redundancy eliminates single points of failure — the most common cause of outages.
- +Active-active topologies double capacity while improving availability.
- +Multi-AZ replication is cheap insurance for the most common (zone) failures.
- +Tested failover lets you recover from incidents in seconds instead of hours.
- −Every redundant copy costs money; HA is not free.
- −Synchronous replication adds write latency proportional to the slowest replica.
- −Active-active requires conflict resolution, which is hard to get right and hard to debug.
- −More components = more failure modes; the system that 'never fails' often fails in novel ways.
How this breaks in production
- Split-brain during a network partition — two primaries accept writes, diverge, and lose data on heal.
- Failover to a stale replica — asynchronous replication lag causes data loss on promotion.
- Failover storm — a flapping primary triggers repeated promotions, each losing in-flight writes.
- Correlated failure — 'redundant' AZs share a power feed or network spine; one event takes both.
- Untested failover — the failover config has rotted; the first real incident is when you discover it.
- DNS propagation delay — failover succeeds but clients keep hitting the old IP for the TTL window.
Don't fall into these traps
- •Counting 'redundant' components that share a hidden dependency (power, network, DNS provider, vendor).
- •Treating the load balancer itself as infallible — LBs need redundancy too.
- •Running active-passive but never failing over to the passive — it accumulates rot and breaks when needed.
- •Asynchronous replication without monitoring lag — you do not know your RPO until you measure it.
- •Synchronous replication across regions for everything — kills write latency for no HA benefit.
- •Forgetting the data plane (stateless app tier) is easy HA, but the control plane (DNS, IAM, secrets) also needs redundancy.
Real systems using this
How real systems implement this
- AWS Aurora — Multi-AZ synchronous storage replication with automated failover to a standby instance in under 30 seconds. Survives AZ failure with RPO zero.
- Cloudflare — Anycast network of 300+ PoPs; every PoP serves traffic, so a PoP failure means traffic is automatically routed to the nearest healthy one with no failover step.
- Netflix Chaos Monkey — Production fault injection that randomly kills instances during business hours, forcing teams to ensure their failover paths actually work.
Practice saying it out loud
- Q1Design a web service that survives an AWS region failure. What is the RTO and RPO you can promise?
- Q2Active-active vs active-passive — when would you choose each? What are the trade-offs?
- Q3Your database uses asynchronous replication to a standby in another region. Replication lag is currently 30 seconds. What is your actual RPO, and what happens if you fail over right now?
- Q4How do you prevent split-brain in a multi-leader database?
- Q5What does 'N+1 redundancy' mean, and what is the hidden assumption that can break it?
Further reading & references
Core explanations are original NO CAP material. External references are provided for deeper study and standards.
What next?
Mark as understood once the mental model clicks.
Next recommended
Consistency Patterns — Weak, Eventual, Strong