Leader Election
Leader election is the process by which a set of distributed nodes choose exactly one of themselves to act as the coordinator — the leader that accepts writes, drives replication, or makes scheduling decisions. Having a single leader avoids the cost of full consensus on every operation: once elected, the leader can act unilaterally until it fails. The hard parts are detecting failure reliably, preventing split-brain (two leaders), and ensuring the new leader sees all committed data before serving traffic. Algorithms include Bully, Ring, and the term-based voting used in Raft.
How it works
A leader is the single node authorized to make a decision — accept writes, sequence operations, drive replication, or own a partition. The benefit is simplicity: instead of running consensus on every write, the system runs consensus once (the election), then the leader acts unilaterally until it fails. This is why every strongly consistent distributed system has a leader per shard or per cluster.
The leader election problem is: given N nodes, agree on which one is the leader, and detect when that leader has failed so a new one can be chosen. Three properties make this hard:
- Failure detection is unreliable. You cannot distinguish a crashed node from a slow one. Heartbeats time out, but a timeout could be caused by network delay, GC pause, or actual crash.
- Split-brain prevention. Two nodes must not both believe they are leader. This requires a quorum — only the side with quorum can elect a leader.
- Committed data preservation. The new leader must have all writes the old leader acknowledged. If not, an acknowledged write can vanish — a data-loss bug. This is why Raft's election rule restricts leadership to nodes whose logs are at least as up-to-date as a quorum.
Several leader election algorithms exist, with different trade-offs:
- Bully algorithm: the node with the highest ID becomes leader. Simple, but requires every node to know every other node's ID, and a single rejoin can disrupt leadership. Used in teaching; rarely in production.
- Ring algorithm: nodes are arranged in a logical ring; an election message circulates. The highest-ID node in the message becomes leader. More fault-tolerant than Bully but slower (O(N) message hops).
- Raft voting: every node has a randomized election timeout; the first to time out starts a new term and asks for votes. A node votes for at most one candidate per term; the candidate with a majority wins. Term numbers prevent stale leaders from staying in power.
- Lease-based election: a leader holds a time-bounded lease (e.g., 10 seconds). It must renew before expiry. If it fails to renew, anyone can take over. Used in Chubby and GFS master election. Requires bounded clock skew (TrueTime in Spanner).
In modern systems, Raft voting and lease-based election dominate. Bully and Ring are mostly of historical interest.
If two nodes both believe they are leader and both accept writes, you have split-brain. When the partition heals, you have two divergent histories with no safe way to merge. Every leader election algorithm must prevent this. The mechanism is always some form of quorum: only a group with a majority of nodes can elect a leader, so two partitions cannot both have leaders. Systems that violate this rule (older versions of Redis Sentinel, some early NoSQL databases) have shipped with split-brain bugs that caused production data loss. Fencing tokens — a monotonically increasing number stamped on every leader transition — are the final defense: even if a stale leader tries to write, downstream systems reject the write because the token is too old.
Election alone is not enough. After a new leader is elected, the old leader might still believe it is in charge — packets from it might be in flight, or it might have been paused (GC, VM stop) and resumed after the new leader took over. The standard defense is the fencing token (a.k.a. leader epoch).
The token is a monotonically increasing integer, incremented on every leader transition. Every write from the leader carries the current token. Downstream systems (storage, replicas, lock services) remember the highest token they have seen and reject any write with a lower token. So if the old leader (epoch=5) tries to write after the new leader (epoch=6) has been acknowledged, the storage rejects the write. This is the foundation of safe leader election in Kafka (leader epochs), Spanner (epoch numbers), and CockroachDB (epoch-based leases).
Without fencing tokens, a paused-and-resumed old leader can cause silent data loss — the classic 'stale leader' bug that affected early MongoDB and Redis setups.
Lease-based leadership is an alternative to constant heartbeats. The leader holds a lease for a fixed duration (e.g., 10 seconds). It must renew before expiry. If it fails to renew (because it crashed or network partitioned), the lease expires and another node can take over.
Leases work only if clock skew is bounded — if the leader's clock jumps, it might think it still has time when the lease has already expired elsewhere. Google's Chubby and Spanner use TrueTime (atomic clocks + GPS) to bound skew to ~7ms, which makes leases safe. On commodity hardware with NTP, you must add a safety margin: a leader that thinks its lease expires at T should stop serving at T - margin (e.g., 1 second before) to avoid operating on a possibly-expired lease.
Leases are more efficient than heartbeat-based election because they require less chatter, but they are harder to get right. Most production systems use Raft-style heartbeat election by default.
Why do Raft and other consensus-based leader elections use randomized election timeouts?
Pick one answer.
A new leader is elected after the old leader was paused for 10 seconds (long GC). The old leader resumes and tries to commit a write it received before the pause. What prevents this stale write from succeeding?
Pick one answer.
Why is the Bully algorithm rarely used in production, despite being simple to implement?
Pick one answer.
Engineering mental model
Mental model. Think of Leader Election as a deliberate boundary in a system. The boundary exists because something becomes harder to manage when everything is done in one place: latency, scale, failure isolation, consistency, cost, or team ownership. The useful question is not “what does Leader Election mean?” but “what pressure makes this boundary worth introducing, and what new failure mode does it create?”
Before choosing Leader Election, name the workload, the critical user path, the dominant bottleneck, the failure you are trying to absorb, and the trade-off you are willing to accept. If you cannot name those five things, the design is probably premature.
// Pseudocode
request = receive()
result = leader_election(request)
return result
// Production questions:
// 1. What happens on timeout?
// 2. Can this operation be retried safely?
// 3. What is the bottleneck?Back-of-the-envelope reasoning
Numerical lens: write down traffic, payload size, read/write ratio, peak multiplier and durability target before choosing a component. The numbers should justify the architecture.
Interactive thought experiment: Leader Election
Change the variables below and predict what breaks first in Leader Election. The production lab can later reuse these same inputs.
Change one variable at a time. Predict the failure mode first, then move the slider and see whether your mental model matches the simplified system response.
If you are stuck on Leader Election, start by drawing the request path and marking every network hop, stateful component, queue, cache and failure boundary. Then estimate where the system will saturate.
You increase traffic by 10× in a system using Leader Election. What should you inspect first?
Pick one answer.
Which statement is the safest engineering habit when using Leader Election?
Pick one answer.
You have dashboards for traffic, latency, errors and saturation. You can change the architecture, but every change has operational cost.
Production scenario: your system uses Leader Election, traffic suddenly spikes, and p99 latency doubles. What is your first move?
Interview drill
Answer this without notes: When would you choose Leader Election, and when would you intentionally avoid it? Mention at least one bottleneck it addresses, one failure mode it introduces, and one alternative. Then quantify the workload you are designing for.
A useful engineering lens for Leader Election: define the problem it solves, the simpler design that fails first, the constraint that forces you to introduce this concept, and the new failure modes the concept creates.
Numerical sanity check
Back-of-the-envelope reasoning beats fake precision. State your traffic, payload, concurrency and growth assumptions explicitly, then calculate enough to know whether the current architecture is orders of magnitude away from the target.
Do not optimize for a memorized definition. Reason from the workload and failure mode.
Imagine the simplest version of a system using Leader Election. What breaks first as traffic grows by 10×, and what would you change before reaching 100×?
Pick one answer.
What you gain, what you pay
- +Single-writer simplicity — the leader acts unilaterally; no per-write consensus needed.
- +Strong consistency — all writes flow through the leader, so ordering is guaranteed.
- +Efficient reads — reads on the leader are strongly consistent without a quorum round (with read-index caveat).
- +Predictable behavior — easier to reason about than leaderless (Dynamo-style) systems.
- −Leader is a bottleneck for writes — throughput is bounded by one node.
- −Failover takes time — election timeout (typically 150-300ms) plus new leader warm-up.
- −Single point of write failure — if the leader is unreachable, the cluster cannot accept writes until a new leader is elected.
- −Risk of split-brain if quorum rules are violated — must be carefully designed.
- −Geographic latency — writes must round-trip to the leader, which is painful if users are far from it.
How this breaks in production
- Split-brain — two leaders elected without quorum; mitigated by Raft-style term-based voting with majority requirement.
- Stale leader writes — paused-and-resumed old leader writes after new leader took over; mitigated by fencing tokens / leader epochs.
- Endless elections — livelock from candidate collisions; mitigated by randomized timeouts.
- False failure detection — slow node (GC pause) mistaken for crashed; triggers unnecessary elections; mitigated by longer timeouts and graceful shutdown.
- Leader with stale log — new leader does not have all committed data; mitigated by Raft's election rule (vote only for candidates whose log is at least as up-to-date).
- Network partition with quorum loss — both halves block; correct behavior but visible as downtime.
Don't fall into these traps
- •Treating leader election as a separate concern from consensus — they are the same problem.
- •Forgetting fencing tokens — stale leader writes cause silent data loss.
- •Setting election timeouts too short — false elections under load spike or GC pause.
- •Running leader-based clusters with even node count — partitions can leave no quorum on either side.
- •Placing the leader far from the writers — geo-distributed writes pay full RTT to leader.
- •Not handling leader election during deployment — rolling deploys can trigger elections; design for graceful leader step-down.
Real systems using this
How real systems implement this
- MongoDB Replica Set — A replica set has a primary (leader) and secondaries. Election uses a Raft variant. If the primary becomes unreachable, the secondaries elect a new primary in 10-12 seconds by default. Term numbers (term) prevent split-brain; only a majority can elect.
- Kafka partition leader / KRaft controller — Each Kafka partition has a leader broker that handles all writes for that partition. Broker failures trigger leader re-election via the controller (formerly ZooKeeper, now KRaft — Raft-based). Leader epochs fence off stale ISR writes.
- Google Chubby — Distributed lock service built on Paxos. A single master per cell holds a lease that must be renewed. Clients cache the master's identity; if the lease expires, a new master is elected. Used by GFS, Bigtable, and many Google internal systems.
- PostgreSQL streaming replication with Patroni — Patroni manages a Postgres cluster: one primary, N replicas. Uses etcd or Consul for leader election. On primary failure, the replica with the most advanced WAL position wins the election and is promoted.
Practice saying it out loud
- Q1Walk me through what happens when a Raft leader crashes. How is split-brain prevented?
- Q2What is a fencing token / leader epoch, and why is it necessary? What bug does it prevent?
- Q3Why are leader elections based on randomized timeouts rather than deterministic rules?
- Q4What is split-brain, and how do production systems prevent it?
- Q5Your database cluster has 4 nodes. Why might this be worse than 3 nodes for leader election?
Further reading & references
Core explanations are original NO CAP material. External references are provided for deeper study and standards.
What next?
Mark as understood once the mental model clicks.
Next recommended
Consensus (Paxos / Raft)