Retry
Retrying failed requests is the simplest reliability pattern — but done wrong, it makes failures worse. Retries must be idempotent, bounded, and use exponential backoff with jitter to avoid retry storms.
How it works
In a distributed system, transient failures are normal: a network blip, a momentary overload, a rolling deploy. Retrying the request is the natural response — but it must be done carefully.
A naive retry (if failed, try again immediately) can make things much worse. If 10,000 clients all retry at once when a service recovers, it goes down again immediately. This is called a retry storm.
Good retries follow 4 rules:
- Only retry idempotent operations. If the operation might have side effects (POST to /charge), retrying might duplicate the effect. Use idempotency keys to make non-idempotent operations safely retryable.
- Exponential backoff. Wait 100ms, then 200ms, then 400ms, then 800ms. Give the failing service time to recover.
- Add jitter. Add a random component to the delay (e.g., 100ms + random(0-50ms)) so clients don't all retry at the same instant.
- Bound the retries. After 3-5 attempts, give up. Infinite retries can hang the system.
Retries handle transient failures. But if a service is down for minutes, retrying every request for minutes is wasteful and harmful. A circuit breaker detects sustained failures and stops sending requests entirely — 'trips open' — for a cooldown period, then tests if the service has recovered ('half-open'). See the Circuit Breaker lesson.
Not all errors should be retried:
- Retry: 429 Too Many Requests, 500/502/503/504 server errors, network timeouts.
- Don't retry: 400 Bad Request, 401 Unauthorized, 403 Forbidden, 404 Not Found. These are client errors — retrying won't fix them.
Most HTTP clients (axios, fetch with wrappers) support configurable retry policies. Use them.
Your API client retries failed requests with a fixed 100ms delay. The downstream service goes down for 30 seconds. What happens?
Pick one answer.
Which HTTP status code should you NOT retry?
Pick one answer.
Engineering mental model
Mental model. Think of Retry as a deliberate boundary in a system. The boundary exists because something becomes harder to manage when everything is done in one place: latency, scale, failure isolation, consistency, cost, or team ownership. The useful question is not “what does Retry mean?” but “what pressure makes this boundary worth introducing, and what new failure mode does it create?”
Before choosing Retry, name the workload, the critical user path, the dominant bottleneck, the failure you are trying to absorb, and the trade-off you are willing to accept. If you cannot name those five things, the design is probably premature.
// Pseudocode
request = receive()
result = retry(request)
return result
// Production questions:
// 1. What happens on timeout?
// 2. Can this operation be retried safely?
// 3. What is the bottleneck?Back-of-the-envelope reasoning
Numerical lens: write down traffic, payload size, read/write ratio, peak multiplier and durability target before choosing a component. The numbers should justify the architecture.
Interactive thought experiment: Retry
Change the variables below and predict what breaks first in Retry. The production lab can later reuse these same inputs.
Change one variable at a time. Predict the failure mode first, then move the slider and see whether your mental model matches the simplified system response.
If you are stuck on Retry, start by drawing the request path and marking every network hop, stateful component, queue, cache and failure boundary. Then estimate where the system will saturate.
You increase traffic by 10× in a system using Retry. What should you inspect first?
Pick one answer.
Which statement is the safest engineering habit when using Retry?
Pick one answer.
You have dashboards for traffic, latency, errors and saturation. You can change the architecture, but every change has operational cost.
Production scenario: your system uses Retry, traffic suddenly spikes, and p99 latency doubles. What is your first move?
Interview drill
Answer this without notes: When would you choose Retry, and when would you intentionally avoid it? Mention at least one bottleneck it addresses, one failure mode it introduces, and one alternative. Then quantify the workload you are designing for.
For Retry, treat the system as a control loop: observe load and failure, choose a bounded response, and measure whether the response stabilizes the system instead of simply moving the bottleneck somewhere else.
Numerical sanity check
When estimating capacity, distinguish average from peak. If average traffic is 4,000 RPS and the observed peak-to-average factor is 3×, design the first pass around roughly 12,000 RPS, then leave headroom for failure and growth.
Do not optimize for a memorized definition. Reason from the workload and failure mode.
What is the earliest signal that Retry is becoming the bottleneck: latency, saturation, errors, queue depth, or something else? Why?
Pick one answer.
What you gain, what you pay
- +Transparently handles transient failures — users don't see them.
- +Simple to implement — most HTTP clients support it.
- +Improves perceived reliability without infrastructure changes.
- −Retrying non-idempotent operations can duplicate side effects (double charges).
- −Naive retries cause thundering herds that prevent recovery.
- −Retries add latency — a request that retries 3 times takes 3x longer to fail.
- −Too many retries can cascade — a slow service makes all its callers slow.
How this breaks in production
- Retry storms — all clients retry simultaneously and overwhelm the recovering service.
- Retrying non-idempotent operations — duplicates side effects (payments, sends).
- Infinite retries — a single failed request hangs forever.
- Retrying client errors (4xx) — wastes resources, never succeeds.
Don't fall into these traps
- •Using a fixed retry delay instead of exponential backoff with jitter.
- •Retrying every error — including 4xx client errors that will never succeed.
- •Not bounding the retry count — infinite retries can hang the system.
- •Forgetting to make operations idempotent before adding retries.
Real systems using this
How real systems implement this
- AWS SDK — Built-in exponential backoff with jitter for all API calls. Configurable max retries (default 3) and retryable status codes.
- gRPC — Configurable retry policy with exponential backoff, jitter, and per-method retry budgets to prevent cascading failures.
Practice saying it out loud
- Q1What is a retry storm, and how do you prevent it?
- Q2Which operations are safe to retry? How do you make non-idempotent operations retryable?
- Q3How does exponential backoff with jitter work, and why is jitter important?
Further reading & references
Core explanations are original NO CAP material. External references are provided for deeper study and standards.
What next?
Mark as understood once the mental model clicks.
Next recommended
Circuit Breaker