Compensating Transaction

A compensating transaction is the answer to: "I committed step 3, but step 5 failed — how do I undo step 3?" In a single ACID database, you'd ROLLBACK. But across services, each commit is final — the inventory service doesn't hold a lock waiting for the payment service to confirm. You can't roll back; you have to compensate.

Compensation means: execute a new operation that semantically undoes the original. If you charged a card, refund it. If you reserved inventory, release it. If you sent a confirmation email, send a cancellation. The original committed transaction stays committed; the compensation is a new, separate transaction.

This is the heart of the Saga pattern: a sequence of local transactions, each with a compensation. On any failure, run compensations in reverse order. The end state is consistent — not because of ACID, but because the compensations restored invariants.

Key insight: compensation is not rollback. A rollback restores the previous state as if the transaction never happened — only possible when locks are held. A compensation accepts that the transaction happened (other systems may have observed it) and adds a new transaction that semantically reverses it.

Practical consequences:

• Compensations must be idempotent — they may be retried. A refund operation called twice must not double-refund.
• Compensations must be commutative with concurrent reads — between the original commit and the compensation, other services may have read the value. They saw the original; they need to handle the change.
• Some operations are hard to compensate — sending an email (you can't unsend it; you can send a follow-up), invoking an external API that's not idempotent, or triggering a physical action.
• Compensations can fail too — what if the refund call fails? You need retries, dead-letter queues, and human escalation. Saga orchestration frameworks (Temporal, Cadence, AWS Step Functions) handle this.

Sagas can be coordinated two ways:

• Choreography: each service emits events; the next service listens and proceeds. No central coordinator. Simple to start, hard to follow as the chain grows — debugging "where did this saga go?" requires tracing events across services.
• Orchestration: a central orchestrator (Temporal, Step Functions, a custom state machine) calls each service and handles compensations explicitly. More code, but the workflow is visible in one place and the orchestrator tracks state across retries and crashes.

Most production systems converge on orchestration for complex sagas because visibility and recoverability matter. Choreography works for short, simple workflows where each service can be reasonably assumed to handle its own compensations.

Either way, the saga state must be persisted — not in memory — so a crash mid-saga can resume. The orchestrator (or a saga-state table) tracks which steps completed and which compensations are still owed.

The classic saga weakness is isolation. Between step 3 (reserve inventory) and step 4 (charge), other queries can see the reserved inventory — they might show 'low stock' or reject a parallel order that would have succeeded if the saga had compensated. ACID's 'I' (isolation) is what sagas give up.

Mitigations:

• Semantic locks: mark records as 'pending' so other queries know not to act on them yet.
• Commutative operations: design steps so concurrent reads don't cause incorrect behavior (e.g., reserve inventory with a tentative count, not a hard decrement).
• Short sagas: keep the saga duration short to minimize the window of inconsistency.
• Compensating reads: queries that need a consistent snapshot read from a projection updated only after saga completion.

This is the fundamental trade-off of sagas: you lose ACID isolation but gain the ability to do transactions across services without distributed locks.

Mental model. Think of Compensating Transaction as a deliberate boundary in a system. The boundary exists because something becomes harder to manage when everything is done in one place: latency, scale, failure isolation, consistency, cost, or team ownership. The useful question is not “what does Compensating Transaction mean?” but “what pressure makes this boundary worth introducing, and what new failure mode does it create?”

Numerical lens: write down traffic, payload size, read/write ratio, peak multiplier and durability target before choosing a component. The numbers should justify the architecture.

Answer this without notes: When would you choose Compensating Transaction, and when would you intentionally avoid it? Mention at least one bottleneck it addresses, one failure mode it introduces, and one alternative. Then quantify the workload you are designing for.

Numerical sanity check

A rough capacity check: required write throughput ≈ peak writes/s × average record size. At 5,000 writes/s and 2 KB average payloads, the raw incoming data stream is about 10 MB/s before indexes, replication and overhead.