Replication
Replication copies data from a primary database to one or more replicas. It improves read availability, read throughput, and fault tolerance. The main trade-offs are replication lag (eventual consistency) and write amplification.
How it works
Replication copies data from a primary (master) database to one or more replicas (slaves). Writes go to the primary; the primary replicates them to replicas. Reads can go to either — but reads from replicas may be stale (eventual consistency).
This gives you: read scaling (add replicas for more read capacity), high availability (if the primary dies, promote a replica), and geographic distribution (place replicas near users).
Synchronous replication: the primary waits for all (or a quorum of) replicas to acknowledge the write before returning success to the client. Strong consistency, but higher write latency and reduced availability if a replica is down.
Asynchronous replication: the primary returns success immediately and replicates in the background. Lower latency, higher availability, but replicas may lag (eventual consistency). If the primary dies before replicating, committed writes can be lost.
Most production systems use semi-synchronous: at least one replica acknowledges before the primary returns, but not all. This balances latency, consistency, and availability.
In async replication, replicas are behind the primary by some amount of time (milliseconds to seconds). This is replication lag. If you read from a replica, you might see stale data. For a social feed, this is fine. For 'did the payment go through?', it's not. Always read from the primary for critical reads.
Replication topologies:
- Single-primary (most common): one primary, N replicas. Simple, but the primary is a write bottleneck.
- Multi-primary (multi-master): any node accepts writes and replicates to others. No write bottleneck, but conflict resolution is complex (last-write-wins, CRDTs, application-level merging).
- Chain replication: A → B → C. Reduces load on the primary but adds latency to the end of the chain.
- Star topology: primary replicates to all replicas directly. Most common.
Your system uses async replication with 3 replicas. A user updates their profile, then immediately views it. They see the old profile. Why?
Pick one answer.
Your primary database dies. You have 3 async replicas. What must happen for the system to recover?
Pick one answer.
| Dimension | Synchronous | Asynchronous | Semi-synchronous |
|---|---|---|---|
| When primary returns success | After ALL replicas ACK the write | Immediately, before any replica ACKs | After AT LEAST ONE replica ACKs |
| Write latency | Highest (waits for slowest replica) | Lowest (no wait) | Medium |
| Data loss on primary failure | None — replicas have it all | Yes — recent in-flight writes can be lost | Reduced — at most the gap between primary and the ACKing replica |
| Availability impact | If any replica is down, writes block | Writes succeed even with replica failures | Writes succeed if the ACKing replica is alive |
| Throughput | Limited by slowest replica RTT | Limited only by primary disk | Limited by ACKing replica RTT |
| Replication lag | Zero (by definition) | Milliseconds to seconds | Zero to one replica, lagging on others |
| Typical use | Critical financial data, zero-loss requirements | Most read-heavy apps, social feeds | Production default for ACID systems |
| Real systems | MySQL Fully Sync, Spanner | PostgreSQL default, MySQL default, Cassandra | PostgreSQL `synchronous_commit=remote_write`, MySQL semi-sync |
Real example: PostgreSQL streaming replication.
PostgreSQL has shipped built-in streaming replication since 9.0 (2010). The mechanism is elegant: the primary ships its write-ahead log (WAL) — a stream of every byte modified — to replicas, which apply the WAL to their own data files in real time.
Configuration on the primary (postgresql.conf):
wal_level = replica
max_wal_senders = 10
synchronous_commit = remote_write # semi-sync: wait for one replica ACK
synchronous_standby_names = 'ANY 1 (replica1, replica2)'Configuration on a replica:
hot_standby = on # accepts read queries while replicating
primary_conninfo = 'host=primary.internal port=5432 user=replication'What this buys you:
- Read scaling: route SELECT-heavy workload (analytics, dashboards) to replicas, leaving the primary for writes.
- HA via Patroni: Patroni (a Python daemon) watches the primary, runs leader election via etcd or Consul, and on failure promotes a replica. Combined with pgBouncer (connection pooler) and HAProxy (LB), you get sub-minute automatic failover with no client code change.
- DR: ship WAL to an S3 bucket via
pg_receivewalor Barman; restore in another region for disaster recovery. PITR (point-in-time recovery) lets you replay WAL up to any chosen timestamp — useful for 'oops, I dropped that table at 14:32' scenarios. - Zero-downtime upgrades: detach a replica, upgrade it, swap it in as the new primary, upgrade the others.
The trade-offs to watch:
- Replication lag — async replicas can be 100ms-10s behind. Read-your-writes consistency requires routing the user's reads to the primary for a window after their write.
- Split-brain — if the network partitions between primary and monitor, the monitor may promote a replica while the old primary is still alive and accepting writes. Mitigations: STONITH (shoot the other node in the head), fencing, or quorum-based promotion (Patroni requires majority consensus).
- Write amplification — each write is applied N+1 times (primary + N replicas). For write-heavy workloads, replication does NOT scale writes — that's what sharding is for.
If a network partition isolates the primary from the failover monitor AND a replica is reachable from the monitor, the monitor may promote the replica — while the original primary is still alive and accepting writes from clients it can still reach. You now have two primaries diverging. When the partition heals, you must reconcile divergent write histories — typically by discarding one side's writes, which means data loss. Mitigations: (a) require quorum (majority of nodes) before promoting, so two primaries can't exist simultaneously; (b) STONITH — the monitor power-cycles the old primary before promoting the replica, guaranteeing it can't accept writes; (c) use consensus-based replication like Spanner or etcd which forbid split-brain by design. Without one of these, 'high availability' is a polite fiction.
You run PostgreSQL with async streaming replication: one primary + two replicas. A user updates their profile (write goes to primary, succeeds), then immediately loads their profile page (read goes to a replica). They see the OLD profile. Two seconds later, refreshing shows the new profile. What is happening, and what is the standard fix for read-your-writes consistency?
Pick one answer.
Replication write amplification — the hidden cost.
Every write to the primary is replicated to N replicas. With N=3 replicas, each write is applied 4 times total (primary + 3 replicas). This is write amplification, and it has three costs you need to size for:
- Storage cost. 4x the storage. With 1 TB of logical data and 3 replicas, you need 4 TB of storage. With 5 replicas, 6 TB. Sounds obvious, but teams consistently under-budget storage by ignoring replica count.
- Network cost. Each write is sent over the network N times. For a high-write workload (10K writes/sec), replication at 3x = 30K writes/sec of network traffic — significant, especially cross-region.
- CPU and IOPS on replicas. Replicas aren't free riders — they apply the WAL to their own data files, which consumes CPU and disk IOPS. A replica under heavy write load may have less capacity to serve reads than you expect.
Two practical implications:
- Replication does NOT scale writes. This is the most common misconception. Adding replicas gives you more read capacity and high availability — it does NOT give you more write capacity. The primary is still the write bottleneck. To scale writes, you need sharding.
- More replicas ≠ better. A common mistake is to add many replicas for 'more availability'. But each replica adds storage, network, and operational cost. The right number of replicas is determined by read QPS requirement + failover needs + geographic distribution — typically 2-5. More than that is wasted money.
The cost trade-off: each replica roughly doubles the storage cost (1 primary + 1 replica = 2x storage). At scale (multi-TB databases), this drives teams toward columnar storage (e.g., ClickHouse) or object-storage-based replicas (S3-backed read replicas in BigQuery) for read-heavy analytics workloads.
Engineering mental model
Mental model. Think of Replication as a deliberate boundary in a system. The boundary exists because something becomes harder to manage when everything is done in one place: latency, scale, failure isolation, consistency, cost, or team ownership. The useful question is not “what does Replication mean?” but “what pressure makes this boundary worth introducing, and what new failure mode does it create?”
Before choosing Replication, name the workload, the critical user path, the dominant bottleneck, the failure you are trying to absorb, and the trade-off you are willing to accept. If you cannot name those five things, the design is probably premature.
// Pseudocode
request = receive()
result = replication(request)
return result
// Production questions:
// 1. What happens on timeout?
// 2. Can this operation be retried safely?
// 3. What is the bottleneck?Back-of-the-envelope reasoning
Example: with 5 replicas, a majority quorum is 3. Losing 2 replicas still leaves a majority available for a quorum-based protocol.
Interactive thought experiment: Replication
Change the variables below and predict what breaks first in Replication. The production lab can later reuse these same inputs.
Change one variable at a time. Predict the failure mode first, then move the slider and see whether your mental model matches the simplified system response.
If you are stuck on Replication, start by drawing the request path and marking every network hop, stateful component, queue, cache and failure boundary. Then estimate where the system will saturate.
You increase traffic by 10× in a system using Replication. What should you inspect first?
Pick one answer.
Which statement is the safest engineering habit when using Replication?
Pick one answer.
You have dashboards for traffic, latency, errors and saturation. You can change the architecture, but every change has operational cost.
Production scenario: your system uses Replication, traffic suddenly spikes, and p99 latency doubles. What is your first move?
Interview drill
Answer this without notes: When would you choose Replication, and when would you intentionally avoid it? Mention at least one bottleneck it addresses, one failure mode it introduces, and one alternative. Then quantify the workload you are designing for.
For Replication, start with access patterns rather than brand names. Identify the dominant reads/writes, data relationships, consistency requirements, partition key, hot keys and failure behavior before choosing a storage strategy.
Numerical sanity check
A rough capacity check: required write throughput ≈ peak writes/s × average record size. At 5,000 writes/s and 2 KB average payloads, the raw incoming data stream is about 10 MB/s before indexes, replication and overhead.
Do not optimize for a memorized definition. Reason from the workload and failure mode.
A team proposes Replication because it 'scales'. What workload characteristic would make that choice a poor fit?
Pick one answer.
What you gain, what you pay
- +High availability — survive primary failure.
- +Read scaling — add replicas for more read capacity.
- +Geographic distribution — replicas near users reduce latency.
- +Backup — replicas can serve as live backups.
- −Replication lag — async replicas are eventually consistent.
- −Write amplification — each write is replicated N times.
- −Failover complexity — promoting a replica, updating routing.
- −Cost — more machines, more storage.
How this breaks in production
- Replication lag — stale reads from replicas.
- Split-brain — network partition causes two primaries, divergent writes.
- Write bottleneck — single primary can't handle write throughput.
- Data loss on failover — async replication means the last few writes may not be on any replica.
Don't fall into these traps
- •Reading from replicas for critical data (payments, permissions).
- •Using async replication when you need zero data loss on failover.
- •Forgetting to test failover — it works in theory but breaks in practice.
- •Not monitoring replication lag — a lagging replica is a time bomb.
Real systems using this
How real systems implement this
- PostgreSQL streaming replication — Async by default, sync available. Replicas can be read-only (hot standby) or take over (promoted).
- MySQL Group Replication — Multi-primary with conflict detection. Uses Paxos-like consensus for write agreement.
Practice saying it out loud
- Q1What is database replication? What are the trade-offs between sync and async?
- Q2How do you handle replication lag?
- Q3What happens when the primary fails? How do you promote a replica?
Further reading & references
Core explanations are original NO CAP material. External references are provided for deeper study and standards.
What next?
Mark as understood once the mental model clicks.
Next recommended
Sharding