Saga Pattern
A saga is a sequence of local transactions, each on a different service, where every step has a compensating action that semantically undoes its effect if a later step fails. Unlike Two-Phase Commit, sagas never block — the system stays available under failure — but they sacrifice isolation: intermediate states are visible to other transactions, and the application must be designed to handle them. Sagas are the de facto standard for cross-service transactions in modern microservice architectures and the answer most senior engineers give to 'how do you do distributed transactions?'
How it works
A saga is a long-lived transaction modeled as a sequence of local transactions T1, T2, ..., Tn, each on a different service. Each Ti has a compensating transaction Ci that semantically undoes Ti's effect. If Ti fails, the saga executes C(i-1), C(i-2), ..., C1 in reverse order to roll back the work that was already done.
Example: place-order saga.
- T1: Order service creates the order (status=pending). C1: Order service marks the order as cancelled.
- T2: Payment service charges the customer. C2: Payment service refunds the charge.
- T3: Inventory service reserves stock. C3: Inventory service releases the reservation.
- T4: Shipping service enqueues a shipment. C4: Shipping service cancels the shipment.
If T3 fails (out of stock), the saga runs C2 (refund) and C1 (cancel order). The customer is made whole: no charge, no order. Crucially, the cancellation is a new write — the original order record exists for a brief moment, then is updated to 'cancelled.' Other services may have observed the intermediate 'pending' state. This is the saga's defining trade-off: resilience without blocking, in exchange for visible intermediate states and the loss of ACID isolation.
The original sagas paper (Garcia-Molina & Salem, 1987) predates microservices by decades — they were designed for long-running database transactions that could not hold locks. The same problem now spans microservices.
There are two ways to coordinate a saga:
Choreography: each service emits an event when it finishes its step; the next service listens for that event and acts. No central coordinator. Simple to start, naturally decoupled, but the workflow logic is distributed across services — debugging is hard, and as the chain grows, it becomes difficult to understand the full flow. Best for short sagas (2-3 steps) with stable business logic.
Orchestration: a central orchestrator (e.g., AWS Step Functions, Temporal, Camunda) calls each service in order, handles failures and compensations, and persists the saga state so it can resume on crash. Easier to reason about, easier to add new steps, and gives you a single place to monitor and modify the workflow. But the orchestrator is a new component to manage, and tightly couples the orchestrator to every service.
The general guidance: start with choreography for simple workflows; switch to orchestration once the saga grows past 3-4 steps, has complex branching, or requires central monitoring. Temporal and AWS Step Functions have made orchestration cheap enough that many teams default to it.
Regardless of variant, the orchestrator (or the event chain) must persist saga state so it survives crashes — a saga that loses its place on crash is broken.
Sagas do not provide ACID isolation. Between T1 (order created) and T4 (shipping enqueued), other transactions can see the intermediate 'pending' order. If two sagas concurrent on the same customer both reserve inventory, one will succeed and the other will compensate. The application must handle this: status enums (pending, confirmed, cancelled), idempotency on every step, optimistic concurrency control (version fields), and explicit handling of 'cannot compensate' scenarios (e.g., a refund that fails because the card is expired). This is the saga's deepest challenge — not the protocol, but the application-level invariants that must be designed around lost isolation.
Designing compensations is the hardest part of saga design. Key principles:
- Compensations are semantic, not technical. A refund is a new business transaction that creates a refund record. It is not an undo log. The original charge remains in the database; the refund is a separate entry that nets it out.
- Compensations must be idempotent. They may be retried due to network failures. A refund keyed on the original charge ID can be safely retried — the second call sees the refund already exists and no-ops.
- Some actions cannot be compensated. Sending an email cannot be un-sent. The compensation is to send a follow-up 'sorry, we cancelled your order' email. The saga must explicitly handle non-recoverable side effects.
- Compensations can also fail. A refund can fail (card expired, bank down). The saga must retry, fall back to manual intervention (a queue for human review), or accept the inconsistency (with monitoring).
- Pivot points. A saga has a 'pivot' — the step after which the business is committed to going forward. Before the pivot, failures trigger full rollback. After the pivot, failures are handled differently (e.g., retry indefinitely rather than compensate). For an order saga, the pivot is often 'payment captured' — once you have the customer's money, you do not refund just because shipping failed; you retry shipping.
The compensation design is where most saga implementations get it wrong — usually by forgetting case (3) or (4).
The transactional outbox pattern is the foundation of every saga step. To 'emit an event after a local transaction,' you cannot dual-write (DB then publish) — that loses events on crash. The outbox: write the business data AND the event to an outbox table in the same local database transaction. Both commit atomically. A separate process (poller or CDC like Debezium) reads the outbox and publishes the event to the broker.
This makes each saga step reliable: the local transaction either fully commits (business data + outbox event) or fully rolls back. The event is guaranteed to be published eventually (at-least-once), so the next saga step is guaranteed to be triggered. The cost: eventual consistency (the event appears in the broker seconds after the DB write) and idempotency on consumers.
Without the outbox, saga implementations either lose events (broken) or use 2PC across DB and broker (slow, blocking). The outbox is the standard solution and is used by Stripe, Uber, Netflix, and every modern event-driven architecture.
In a place-order saga, step T2 (Payment.charge) succeeds, but step T3 (Inventory.reserve) fails. What does the saga do?
Pick one answer.
What is the difference between choreography and orchestration in sagas, and when do you choose each?
Pick one answer.
Why does every saga step need the transactional outbox pattern, and what goes wrong without it?
Pick one answer.
Engineering mental model
Mental model. Think of Saga Pattern as a deliberate boundary in a system. The boundary exists because something becomes harder to manage when everything is done in one place: latency, scale, failure isolation, consistency, cost, or team ownership. The useful question is not “what does Saga Pattern mean?” but “what pressure makes this boundary worth introducing, and what new failure mode does it create?”
Before choosing Saga Pattern, name the workload, the critical user path, the dominant bottleneck, the failure you are trying to absorb, and the trade-off you are willing to accept. If you cannot name those five things, the design is probably premature.
// Pseudocode
request = receive()
result = saga_pattern(request)
return result
// Production questions:
// 1. What happens on timeout?
// 2. Can this operation be retried safely?
// 3. What is the bottleneck?Back-of-the-envelope reasoning
Numerical lens: write down traffic, payload size, read/write ratio, peak multiplier and durability target before choosing a component. The numbers should justify the architecture.
Interactive thought experiment: Saga Pattern
Change the variables below and predict what breaks first in Saga Pattern. The production lab can later reuse these same inputs.
Change one variable at a time. Predict the failure mode first, then move the slider and see whether your mental model matches the simplified system response.
If you are stuck on Saga Pattern, start by drawing the request path and marking every network hop, stateful component, queue, cache and failure boundary. Then estimate where the system will saturate.
You increase traffic by 10× in a system using Saga Pattern. What should you inspect first?
Pick one answer.
Which statement is the safest engineering habit when using Saga Pattern?
Pick one answer.
You have dashboards for traffic, latency, errors and saturation. You can change the architecture, but every change has operational cost.
Production scenario: your system uses Saga Pattern, traffic suddenly spikes, and p99 latency doubles. What is your first move?
Interview drill
Answer this without notes: When would you choose Saga Pattern, and when would you intentionally avoid it? Mention at least one bottleneck it addresses, one failure mode it introduces, and one alternative. Then quantify the workload you are designing for.
A useful engineering lens for Saga Pattern: define the problem it solves, the simpler design that fails first, the constraint that forces you to introduce this concept, and the new failure modes the concept creates.
Numerical sanity check
Back-of-the-envelope reasoning beats fake precision. State your traffic, payload, concurrency and growth assumptions explicitly, then calculate enough to know whether the current architecture is orders of magnitude away from the target.
Do not optimize for a memorized definition. Reason from the workload and failure mode.
Imagine the simplest version of a system using Saga Pattern. What breaks first as traffic grows by 10×, and what would you change before reaching 100×?
Pick one answer.
What you gain, what you pay
- +Non-blocking — the system stays available under failure, unlike 2PC.
- +Resilient — failures trigger compensations, not indefinite blocking.
- +Fits the microservice model — each service owns its data and emits events.
- +Scales naturally — no central coordinator needed for choreography variant.
- +Foundation of modern event-driven architectures.
- −No ACID isolation — intermediate states are visible to other transactions.
- −Requires designing a compensation for every step — semantic, not technical.
- −Some actions cannot be compensated (sent emails, real-world side effects).
- −Compensations can fail too — requires retry and manual intervention paths.
- −Debugging is hard — distributed traces and correlation IDs are essential.
How this breaks in production
- Lost events from naive dual-write — fixed by the transactional outbox pattern.
- Compensation failure — refund fails; mitigated by retry queue and manual intervention.
- Non-compensatable side effects — sent emails; mitigated by follow-up 'sorry' messages.
- Out-of-order events in choreography — events arrive in wrong order; mitigated by correlation IDs and idempotency.
- Concurrent sagas on the same resource — race conditions; mitigated by optimistic concurrency and version fields.
- Orchestrator crash — saga state must be persisted; mitigated by durable orchestrators (Temporal, Step Functions).
Don't fall into these traps
- •Using dual-write (DB then publish) instead of the outbox pattern — loses events.
- •Designing compensations as technical rollbacks instead of semantic undos.
- •Forgetting that some actions cannot be compensated — sent emails, real-world side effects.
- •Not making compensations idempotent — retries cause double refunds.
- •Assuming isolation — other transactions see intermediate states; design for it.
- •Not propagating correlation IDs — debugging becomes impossible across services.
- •Using choreography for long, complex flows — switch to orchestration.
Real systems using this
How real systems implement this
- Temporal — Open-source workflow orchestration engine that runs sagas as durable, resumable state machines. Each step is a function call; failures and compensations are first-class. Used by Stripe, Snap, HashiCorp, and many others for cross-service workflows. State persists across crashes; retries and compensations are declarative.
- AWS Step Functions — Managed orchestrator for saga-style workflows. Define states (success, failure, compensation) in ASL; AWS runs the state machine durably. Integrated with Lambda, SQS, SNS, and DynamoDB. Common for AWS-native microservice orchestration.
- Netflix — Uses choreographed sagas for much of its microservice architecture — each service emits events to Kafka, others react. Compensations are designed per use case. The scale and resilience requirements forced them away from 2PC-style coordination.
- Uber Cadence (now Temporal) — Uber built Cadence (the predecessor of Temporal) to manage long-running sagas across their microservices — trip booking, driver matching, payments. The saga state machine persists durably; even multi-day workflows (driver onboarding) are sagas.
Practice saying it out loud
- Q1Design a place-order saga across order, payment, inventory, and shipping services. Walk through the happy path and a failure case.
- Q2What is the difference between choreography and orchestration? When do you choose each?
- Q3Why does every saga step need the outbox pattern? What goes wrong without it?
- Q4How do you design a compensation for an action that has real-world side effects (sent email, shipped package)?
- Q5Compare sagas and 2PC. Why have sagas displaced 2PC in microservices?
Further reading & references
Core explanations are original NO CAP material. External references are provided for deeper study and standards.
What next?
Mark as understood once the mental model clicks.
Next recommended
Event Sourcing