Circuit Breaker
A circuit breaker stops calling a failing service after a threshold of failures, giving it time to recover. It has three states: closed (normal), open (failing, stop calling), and half-open (testing if recovery happened). It prevents cascading failures and retry storms.
How it works
A circuit breaker is like an electrical fuse: when too many failures occur, it 'trips' and stops current from flowing. In software, this means: stop calling a failing service so it can recover, and so your service doesn't waste resources waiting for timeouts.
The pattern has three states:
- Closed (normal): requests flow through. Failures are counted.
- Open (tripped): requests fail immediately — no call is made to the downstream. This gives the downstream time to recover.
- Half-Open (testing): after a cooldown, a few requests are allowed through. If they succeed, the breaker closes. If they fail, it opens again.
Why is this better than just retrying?
- Fail fast: when the breaker is open, your service fails in 1ms instead of waiting 30 seconds for a timeout. Resources (threads, connections) are freed immediately.
- Let the downstream recover: if your service keeps hammering a recovering service, it can't recover. The breaker gives it breathing room.
- Prevent cascading failures: without a breaker, a slow downstream service causes your service to pile up connections, which causes services that depend on you to pile up, which takes down the whole system.
Retries handle transient failures (a single bad request). Circuit breakers handle sustained failures (the service is down). Use both: retry first (for transient blips), then circuit break (if retries keep failing). Retries without a breaker cause retry storms. A breaker without retries fails too aggressively.
Typical circuit breaker configuration:
- Failure threshold: e.g., 'open after 5 consecutive failures' or 'open after 50% failure rate over 20 requests'.
- Cooldown duration: e.g., 'stay open for 30 seconds before half-open'.
- Half-open probes: e.g., 'allow 3 requests through; if all succeed, close; if any fail, re-open'.
- Sliding window: count failures over the last N seconds or N requests, not since startup.
Libraries like Resilience4j (Java), Polly (.NET), and opossum (Node.js) implement this with sensible defaults.
Your service calls a downstream API that starts timing out. Without a circuit breaker, what happens?
Pick one answer.
A circuit breaker is in the Open state. A request comes in. What happens?
Pick one answer.
Engineering mental model
Mental model. Think of Circuit Breaker as a deliberate boundary in a system. The boundary exists because something becomes harder to manage when everything is done in one place: latency, scale, failure isolation, consistency, cost, or team ownership. The useful question is not “what does Circuit Breaker mean?” but “what pressure makes this boundary worth introducing, and what new failure mode does it create?”
Before choosing Circuit Breaker, name the workload, the critical user path, the dominant bottleneck, the failure you are trying to absorb, and the trade-off you are willing to accept. If you cannot name those five things, the design is probably premature.
// Pseudocode
request = receive()
result = circuit_breaker(request)
return result
// Production questions:
// 1. What happens on timeout?
// 2. Can this operation be retried safely?
// 3. What is the bottleneck?Back-of-the-envelope reasoning
Numerical lens: write down traffic, payload size, read/write ratio, peak multiplier and durability target before choosing a component. The numbers should justify the architecture.
Interactive thought experiment: Circuit Breaker
Change the variables below and predict what breaks first in Circuit Breaker. The production lab can later reuse these same inputs.
Change one variable at a time. Predict the failure mode first, then move the slider and see whether your mental model matches the simplified system response.
If you are stuck on Circuit Breaker, start by drawing the request path and marking every network hop, stateful component, queue, cache and failure boundary. Then estimate where the system will saturate.
You increase traffic by 10× in a system using Circuit Breaker. What should you inspect first?
Pick one answer.
Which statement is the safest engineering habit when using Circuit Breaker?
Pick one answer.
You have dashboards for traffic, latency, errors and saturation. You can change the architecture, but every change has operational cost.
Production scenario: your system uses Circuit Breaker, traffic suddenly spikes, and p99 latency doubles. What is your first move?
Interview drill
Answer this without notes: When would you choose Circuit Breaker, and when would you intentionally avoid it? Mention at least one bottleneck it addresses, one failure mode it introduces, and one alternative. Then quantify the workload you are designing for.
For Circuit Breaker, treat the system as a control loop: observe load and failure, choose a bounded response, and measure whether the response stabilizes the system instead of simply moving the bottleneck somewhere else.
Numerical sanity check
When estimating capacity, distinguish average from peak. If average traffic is 4,000 RPS and the observed peak-to-average factor is 3×, design the first pass around roughly 12,000 RPS, then leave headroom for failure and growth.
Do not optimize for a memorized definition. Reason from the workload and failure mode.
What is the earliest signal that Circuit Breaker is becoming the bottleneck: latency, saturation, errors, queue depth, or something else? Why?
Pick one answer.
What you gain, what you pay
- +Prevents cascading failures — one broken service doesn't take down everything.
- +Fails fast — frees resources instead of waiting for timeouts.
- +Gives downstream services time to recover.
- +Self-healing — automatically tests for recovery (half-open).
- −Adds complexity — another state to reason about.
- −Can cause 'flapping' if the threshold is too sensitive.
- −Requires tuning — wrong thresholds cause false trips or missed failures.
- −Doesn't help with non-transient failures (e.g., a 404).
How this breaks in production
- Threshold too sensitive — breaker trips on normal variance, causing unnecessary failures.
- Threshold too insensitive — breaker never trips, cascading failure still happens.
- Cooldown too short — downstream doesn't have time to recover before being hammered again.
- No fallback — when the breaker is open, the caller needs a fallback (cached data, default value, error.
Don't fall into these traps
- •Treating the breaker as a retry mechanism — it's the opposite, it stops retries.
- •Not providing a fallback — when the breaker is open, what does the user see?
- •Setting the threshold too low — normal latency variance trips the breaker.
- •Forgetting to monitor breaker state — you need to know when breakers are open.
Real systems using this
How real systems implement this
- Netflix Hystrix — Pioneered circuit breaking in microservices. Now deprecated in favor of Resilience4j, but the pattern is universal.
- Istio service mesh — Implements circuit breaking at the sidecar (Envoy proxy), so applications don't need circuit breaker code.
Practice saying it out loud
- Q1What is a circuit breaker? How does it differ from a retry?
- Q2Explain the three states of a circuit breaker.
- Q3How do you choose the failure threshold and cooldown duration?
- Q4What happens when the breaker is open — what does the caller do?
Further reading & references
Core explanations are original NO CAP material. External references are provided for deeper study and standards.
What next?
Mark as understood once the mental model clicks.
Next recommended
Bulkhead