Bulkhead
A bulkhead isolates resources so a failure in one part of the system doesn't take down everything else. Like watertight compartments in a ship's hull, if one compartment floods, the ship stays afloat. In software: separate thread pools, connection pools, or processes for different services or workloads.
How it works
The name comes from ship design: a ship's hull is divided into watertight compartments (bulkheads). If one compartment floods, the door is closed and the ship stays afloat. The damage is isolated.
In software, the same principle applies: if your service calls 3 downstream services (A, B, C), and they all share one thread pool, a failure in A can consume all threads and prevent B and C from being served. A bulkhead gives each downstream its own pool — A's failure only affects A's pool.
Bulkheads can be implemented at different levels:
- Thread pools: separate pool per downstream service. Most common.
- Connection pools: separate HTTP/TCP connection pool per downstream.
- Processes: separate processes (or containers) per service type. Strongest isolation.
- Machines: separate servers per workload. Physical isolation.
The stronger the isolation, the more resource-expensive. Thread pools are the sweet spot for most cases.
Bulkheads limit the blast radius. Circuit breakers stop calling a failing service entirely. Together: the circuit breaker prevents new calls to a failing service, and the bulkhead ensures that even if the breaker is slow to trip, other services aren't affected. This combination is the foundation of resilient microservice architecture.
Your service has a single thread pool of 200 threads and calls 3 downstream services. One downstream starts responding slowly. What happens without a bulkhead?
Pick one answer.
What is the relationship between bulkheads and circuit breakers?
Pick one answer.
Engineering mental model
Mental model. Think of Bulkhead as a deliberate boundary in a system. The boundary exists because something becomes harder to manage when everything is done in one place: latency, scale, failure isolation, consistency, cost, or team ownership. The useful question is not “what does Bulkhead mean?” but “what pressure makes this boundary worth introducing, and what new failure mode does it create?”
Before choosing Bulkhead, name the workload, the critical user path, the dominant bottleneck, the failure you are trying to absorb, and the trade-off you are willing to accept. If you cannot name those five things, the design is probably premature.
// Pseudocode
request = receive()
result = bulkhead(request)
return result
// Production questions:
// 1. What happens on timeout?
// 2. Can this operation be retried safely?
// 3. What is the bottleneck?Back-of-the-envelope reasoning
Numerical lens: write down traffic, payload size, read/write ratio, peak multiplier and durability target before choosing a component. The numbers should justify the architecture.
Interactive thought experiment: Bulkhead
Change the variables below and predict what breaks first in Bulkhead. The production lab can later reuse these same inputs.
Change one variable at a time. Predict the failure mode first, then move the slider and see whether your mental model matches the simplified system response.
If you are stuck on Bulkhead, start by drawing the request path and marking every network hop, stateful component, queue, cache and failure boundary. Then estimate where the system will saturate.
You increase traffic by 10× in a system using Bulkhead. What should you inspect first?
Pick one answer.
Which statement is the safest engineering habit when using Bulkhead?
Pick one answer.
You have dashboards for traffic, latency, errors and saturation. You can change the architecture, but every change has operational cost.
Production scenario: your system uses Bulkhead, traffic suddenly spikes, and p99 latency doubles. What is your first move?
Interview drill
Answer this without notes: When would you choose Bulkhead, and when would you intentionally avoid it? Mention at least one bottleneck it addresses, one failure mode it introduces, and one alternative. Then quantify the workload you are designing for.
For Bulkhead, treat the system as a control loop: observe load and failure, choose a bounded response, and measure whether the response stabilizes the system instead of simply moving the bottleneck somewhere else.
Numerical sanity check
When estimating capacity, distinguish average from peak. If average traffic is 4,000 RPS and the observed peak-to-average factor is 3×, design the first pass around roughly 12,000 RPS, then leave headroom for failure and growth.
Do not optimize for a memorized definition. Reason from the workload and failure mode.
What is the earliest signal that Bulkhead is becoming the bottleneck: latency, saturation, errors, queue depth, or something else? Why?
Pick one answer.
What you gain, what you pay
- +Isolates failures — one broken downstream doesn't take down everything.
- +Allows partial degradation — some features work while others fail.
- +Composable with circuit breakers for layered resilience.
- −More resource overhead — separate pools means more idle threads.
- −Harder to tune — how big should each pool be?
- −Adds complexity — more moving parts to monitor and debug.
- −Can cause under-utilization — if one pool is idle while another is saturated, you can't borrow.
How this breaks in production
- Pool too small — legitimate traffic gets rejected even when the downstream is healthy.
- Pool too large — too many concurrent calls overwhelm the downstream.
- No monitoring — if you don't know a pool is saturated, you can't diagnose the failure.
Don't fall into these traps
- •Using a single shared pool for all downstreams — defeats the purpose.
- •Making pools too small — causes false failures under normal load spikes.
- •Forgetting to monitor pool utilization — you need to know when pools are near saturation.
Real systems using this
How real systems implement this
- Netflix Hystrix — Popularized bulkheads in microservices: each downstream had its own thread pool, so a slow dependency couldn't starve others.
- Envoy proxy (Istio) — Configures per-upstream connection pools and circuit breakers at the sidecar, isolating failures without application code changes.
Practice saying it out loud
- Q1What is a bulkhead pattern? How does it differ from a circuit breaker?
- Q2Why would you use separate thread pools for different downstream services?
- Q3How do you decide the size of each bulkhead pool?
Further reading & references
Core explanations are original NO CAP material. External references are provided for deeper study and standards.
What next?
Mark as understood once the mental model clicks.
Next recommended
Circuit Breaker