Distributed Transactions
A distributed transaction is a unit of work that spans multiple independent resources — different databases, message brokers, or services — and must either commit on all of them or roll back on all of them. The classical solution is Two-Phase Commit (2PC): a coordinator asks every participant to prepare, then to commit, blocking until both phases complete. The modern solution for microservices is the saga pattern: a sequence of local transactions, each with a compensating action that undoes its effect if a later step fails. Both have steep costs — 2PC sacrifices availability and latency for strong consistency; sagas sacrifice isolation for resilience.
How it works
ACID transactions are well-defined on a single database: atomicity (all-or-nothing), consistency (invariants preserved), isolation (concurrent transactions appear serial), durability (committed data survives crashes). When the transaction spans multiple resources — different databases, message brokers, or services — ACID becomes much harder.
The fundamental difficulty is the two generals problem: there is no way to atomically commit on two independent resources over an unreliable network. Every protocol is a workaround that trades off some property:
- 2PC trades availability for atomicity — if the coordinator crashes, the system blocks.
- Sagas trade isolation for resilience — intermediate states are visible, but the system never blocks.
- Transactional outbox trades atomicity for eventual consistency — the database write and the message publish are eventually consistent, not atomically committed.
The choice depends on the workload. Tight coupling and strong consistency (single-cluster databases): 2PC. Loosely coupled microservices with eventual consistency tolerance: sagas.
Two-Phase Commit (2PC) is the classical distributed transaction protocol. A coordinator orchestrates two phases:
- Prepare phase: coordinator sends
PREPAREto every participant. Each participant writes the work to a stable log (so it can survive a crash), then repliesVOTE YESorVOTE NO. If any participant votes NO, the coordinator aborts. - Commit phase: if all voted YES, the coordinator sends
COMMITto every participant. Each participant applies the work, releases locks, and repliesACK. Once the coordinator has all acks, the transaction is done.
2PC guarantees atomicity: either all participants commit or none do. The cost:
- Blocking on coordinator failure: if the coordinator crashes between prepare and commit, participants are stuck holding locks and cannot proceed. They must wait for the coordinator to recover — and during that wait, the locked resources are unavailable.
- Slow: 2-3 network round trips plus fsync on every participant. Latency adds up.
- Failure scenarios: participant crashes mid-protocol, coordinator crashes, network partition — each requires careful recovery logic.
2PC is used internally by distributed databases (Spanner, CockroachDB, Postgres with foreign data wrappers) where the participants are tightly coupled. It is rarely used across microservices because of the blocking cost.
During a coordinator failure or partition, 2PC blocks — no new transactions can complete, and existing ones are stuck. This is by design: atomicity is preserved at the cost of availability. The CAP theorem says you cannot have all three; 2PC chooses consistency over availability during partitions. For microservices where availability matters more, sagas (AP) are preferred — they accept intermediate inconsistency to stay available.
Sagas (Hector Garcia-Molina & Kenneth Salem, 1987) model a distributed transaction as a sequence of local transactions T1, T2, ..., Tn, each with a compensating transaction C1, C2, ..., Cn. If Ti fails, the saga runs C(i-1), C(i-2), ..., C1 to undo the work.
Example: place-order saga.
- T1: Order.create() — emits
OrderCreated. Compensating C1: Order.cancel(). - T2: Payment.charge() — emits
PaymentCharged. C2: Payment.refund(). - T3: Inventory.reserve() — emits
InventoryReserved. C3: Inventory.release(). - T4: Shipping.enqueue() — emits
ShippingQueued. C4: Shipping.cancel().
If T3 fails, the saga runs C2 (refund payment) and C1 (cancel order). The customer is made whole — no charge, no order — but the cancellation is a semantic action, not a technical rollback. The order existed in the database for a brief moment; other services may have observed it.
There are two flavors:
- Choreography: each service emits an event; the next service reacts. No central coordinator. Simple to start, harder to debug as the chain grows.
- Orchestration: a central orchestrator (e.g., AWS Step Functions, Temporal) calls each service in order and handles compensations. Easier to reason about, but the orchestrator is a new component to manage.
Sagas trade isolation for resilience. Intermediate states are visible: the order exists before payment succeeds; the customer may see a 'pending' order briefly. The application must be designed to handle this — status enums, idempotency, and outbox pattern are essential.
The transactional outbox pattern solves the most common distributed-transaction problem: 'atomically update the database and publish a message.' The naive approach — write to the DB, then publish to the broker — fails if the publish call crashes (the DB write is committed, but the message is lost). 2PC across the DB and broker works but is slow and brittle.
The outbox: write the message to an outbox table in the same database transaction as the business write. Both commit atomically (local transaction). A separate process (the 'outbox poller' or CDC like Debezium) reads the outbox table and publishes messages to the broker, then marks them as sent. If the publisher crashes, the next poll re-publishes — at-least-once delivery, but the message is never lost.
This is the de facto standard for cross-service event emission in microservices. It is a special case of the saga pattern: the database write is the local transaction; the message publish is the eventual side effect. Stripe, Uber, Netflix, and every modern event-driven microservice architecture use this pattern. The cost: eventual consistency (the message appears in the broker seconds after the DB write) and at-least-once delivery (consumers must be idempotent).
Why is Two-Phase Commit rarely used across microservices, even though it provides strong atomicity?
Pick one answer.
In a saga, what is a 'compensating transaction,' and how does it differ from a database rollback?
Pick one answer.
You need to update the orders database AND publish an 'OrderCreated' message to Kafka, atomically. What is the standard solution?
Pick one answer.
Engineering mental model
Mental model. Think of Distributed Transactions as a deliberate boundary in a system. The boundary exists because something becomes harder to manage when everything is done in one place: latency, scale, failure isolation, consistency, cost, or team ownership. The useful question is not “what does Distributed Transactions mean?” but “what pressure makes this boundary worth introducing, and what new failure mode does it create?”
Before choosing Distributed Transactions, name the workload, the critical user path, the dominant bottleneck, the failure you are trying to absorb, and the trade-off you are willing to accept. If you cannot name those five things, the design is probably premature.
SELECT id, created_at
FROM records
WHERE tenant_id = ?
ORDER BY created_at DESC
LIMIT 50;
-- Ask: which index makes this query predictable at scale?Back-of-the-envelope reasoning
Numerical lens: write down traffic, payload size, read/write ratio, peak multiplier and durability target before choosing a component. The numbers should justify the architecture.
Interactive thought experiment: Distributed Transactions
Change the variables below and predict what breaks first in Distributed Transactions. The production lab can later reuse these same inputs.
Change one variable at a time. Predict the failure mode first, then move the slider and see whether your mental model matches the simplified system response.
If you are stuck on Distributed Transactions, start by drawing the request path and marking every network hop, stateful component, queue, cache and failure boundary. Then estimate where the system will saturate.
You increase traffic by 10× in a system using Distributed Transactions. What should you inspect first?
Pick one answer.
Which statement is the safest engineering habit when using Distributed Transactions?
Pick one answer.
You have dashboards for traffic, latency, errors and saturation. You can change the architecture, but every change has operational cost.
Production scenario: your system uses Distributed Transactions, traffic suddenly spikes, and p99 latency doubles. What is your first move?
Interview drill
Answer this without notes: When would you choose Distributed Transactions, and when would you intentionally avoid it? Mention at least one bottleneck it addresses, one failure mode it introduces, and one alternative. Then quantify the workload you are designing for.
For Distributed Transactions, start with access patterns rather than brand names. Identify the dominant reads/writes, data relationships, consistency requirements, partition key, hot keys and failure behavior before choosing a storage strategy.
Numerical sanity check
A rough capacity check: required write throughput ≈ peak writes/s × average record size. At 5,000 writes/s and 2 KB average payloads, the raw incoming data stream is about 10 MB/s before indexes, replication and overhead.
Do not optimize for a memorized definition. Reason from the workload and failure mode.
A team proposes Distributed Transactions because it 'scales'. What workload characteristic would make that choice a poor fit?
Pick one answer.
What you gain, what you pay
- +Enables atomic multi-resource updates that local transactions cannot do.
- +2PC gives true ACID atomicity across resources (when participants are tightly coupled).
- +Sagas provide resilience without blocking — the system stays available under failure.
- +Outbox pattern solves the DB+message atomicity problem with minimal complexity.
- +Foundation for cross-service workflows in microservices.
- −2PC blocks on coordinator failure — degrades availability; CP not AP.
- −2PC is slow — multiple round trips plus fsync on every participant.
- −Sagas sacrifice isolation — intermediate states are visible; applications must handle them.
- −Sagas require designing a compensating action for every step — semantic, not technical.
- −All variants add operational complexity — coordinators, orchestrators, outbox pollers, CDC.
How this breaks in production
- 2PC coordinator crash mid-protocol — participants stuck holding locks; mitigated by recovery protocols but blocking is unavoidable.
- Saga intermediate state observed — another service sees a 'pending' order; mitigated by status enums and idempotency.
- Outbox poller crash — messages delayed but not lost (next poll re-publishes); mitigated by at-least-once consumers.
- Compensating action failure — refund fails; requires retry + manual intervention path.
- Out-of-order events in choreography sagas — events arrive in wrong order; mitigated by correlation IDs and idempotency.
- Long-running saga — locks resources for a long time; mitigated by shorter steps and explicit state machines.
Don't fall into these traps
- •Using 2PC across microservices — blocks too often, too slow, too tightly coupled.
- •Using sagas without designing compensating actions — failures leave the system in inconsistent state.
- •Naive dual-write (DB then publish) — loses messages on publish crash.
- •Assuming saga isolation — concurrent sagas can see each other's intermediate states; design for it.
- •Forgetting that outbox consumers must be idempotent — at-least-once delivery is the norm.
- •Not propagating a correlation ID across saga steps — debugging becomes impossible.
Real systems using this
How real systems implement this
- Google Spanner — Uses 2PC across participant groups for true ACID distributed transactions. TrueTime bounds clock uncertainty to ~7ms, allowing external consistency. The cost: latency of cross-group commits is multiple round trips. Used when strong consistency is non-negotiable.
- CockroachDB — Uses 2PC across replicas for ACID transactions in a horizontally scalable SQL database. Latency is the trade-off — every write goes through consensus (Raft) and may span multiple ranges.
- Temporal — Workflow orchestration engine that runs sagas as durable, resumable state machines. Each step is idempotent and replayable on crash. Compensations are first-class. Used by Stripe, Snap, and many others for cross-service workflows.
- Debezium — Open-source CDC (Change Data Capture) tool that reads the WAL/binlog of a database and publishes row-level changes to Kafka. Often used as the 'outbox poller' in the transactional outbox pattern — turns DB writes into event stream publishes with no dual-write problem.
Practice saying it out loud
- Q1Compare 2PC and sagas. When do you choose each?
- Q2What is a compensating transaction? Why is it not the same as a rollback?
- Q3How does the transactional outbox pattern work? What problem does it solve?
- Q4Your saga step 3 failed. Walk through what happens. What if a compensation also fails?
- Q5Why does 2PC block on coordinator failure? Can you fix this?
Further reading & references
Core explanations are original NO CAP material. External references are provided for deeper study and standards.
What next?
Mark as understood once the mental model clicks.
Next recommended
Two-Phase Commit