Timeouts
A timeout is the maximum time you're willing to wait for an operation. Every external call — HTTP request, database query, RPC — must have one, or a slow dependency becomes a system-wide outage. The key distinctions are connection timeout vs read timeout, client-side vs server-side timeout, and ensuring deadlines propagate across the call chain.
How it works
A timeout is a deadline. You tell the system: "if this operation hasn't completed in T milliseconds, give up." That's it. But the consequences of getting it wrong are enormous.
There are two kinds of timeout you must set on every network call:
- Connection timeout: how long to wait for the TCP handshake to complete. If the host is unreachable, this fails fast instead of hanging for the OS default (often 60+ seconds).
- Read timeout: how long to wait for data after the connection is established. This catches slow responses — the server accepted the connection but is taking forever to respond.
Skipping either is a bug. Most HTTP clients (axios, requests, fetch) default to either no timeout or a very long one. You must override.
Setting a timeout on every call is necessary but not sufficient. Consider: a client gives your API a 1-second budget. Your API spends 200ms on logic, then calls a downstream service with a 1-second timeout. The downstream takes 900ms and you return to the client at 1100ms — but the client already gave up at 1000ms. You wasted 900ms of work.
The fix is deadline propagation (gRPC) or context cancellation (Go, Java). The client sends its deadline; each hop subtracts the time it has already used and passes the reduced deadline downstream. When the deadline expires, in-flight work is cancelled, freeing resources immediately. gRPC does this via the grpc-timeout header; Go propagates contexts; HTTP/2 has its own deadline signaling.
Without propagation, timeouts at each layer don't add up to a bounded end-to-end latency — they add up to N times the per-layer timeout.
How long should a timeout be? There's no universal answer, but there are rules:
- Match the SLO. If your p99 latency is 200ms, a 5-second timeout hides failures. A 500ms timeout surfaces them.
- Connection timeout should be short (1–5s). If you can't connect in 1s, the host is probably down.
- Read timeout should reflect normal behavior, plus headroom (p99 * 2 or 3, often 1–10s for APIs).
- Database query timeouts are often shorter (100ms–1s for OLTP). Long queries lock rows and should fail.
- Be careful with retries: a 1s timeout with 3 retries means 3s of latency if the service is down. Set the total budget, not just the per-attempt timeout.
Measure first, then set. Most production incidents involving timeouts come from defaults that are too long.
A client-side timeout aborts the client's wait. The server may not know the client gave up — it keeps doing work and sends a response to a closed socket. A server-side timeout aborts the server's work and returns an error to the client. You need both. The client-side timeout protects the client from waiting forever; the server-side timeout protects the server's resources. Without server-side timeouts, a single slow query can hold a database connection for the OS default (often 8 hours).
The reason timeouts matter so much is cascading failures. A typical cascade:
- A downstream service gets slow (DB at 80% CPU, queries take 2s instead of 100ms).
- Without timeouts, your service holds a thread per slow request.
- Your thread pool fills in seconds. New requests queue.
- Your service is now slow. Your callers hold threads waiting for you.
- Their thread pools fill. The cascade spreads outward.
With proper timeouts (say 500ms), step 2 frees the thread in 500ms instead of waiting 2s. The downstream query fails (good — it should), your service stays responsive, and the cascade is contained. Timeouts are the circuit breaker's partner: timeouts detect per-call failures; the breaker detects sustained failures.
Your service makes an HTTP call to a downstream API. The default timeout in your HTTP client is 60s. The downstream API sometimes hangs. What happens without setting a timeout?
Pick one answer.
A client calls your API with a 1-second deadline. Your API takes 300ms and then calls a downstream service. What timeout should you set on the downstream call?
Pick one answer.
What's the difference between a connection timeout and a read timeout?
Pick one answer.
Engineering mental model
Mental model. Think of Timeouts as a deliberate boundary in a system. The boundary exists because something becomes harder to manage when everything is done in one place: latency, scale, failure isolation, consistency, cost, or team ownership. The useful question is not “what does Timeouts mean?” but “what pressure makes this boundary worth introducing, and what new failure mode does it create?”
Before choosing Timeouts, name the workload, the critical user path, the dominant bottleneck, the failure you are trying to absorb, and the trade-off you are willing to accept. If you cannot name those five things, the design is probably premature.
// Pseudocode
request = receive()
result = timeout(request)
return result
// Production questions:
// 1. What happens on timeout?
// 2. Can this operation be retried safely?
// 3. What is the bottleneck?Back-of-the-envelope reasoning
Numerical lens: write down traffic, payload size, read/write ratio, peak multiplier and durability target before choosing a component. The numbers should justify the architecture.
Interactive thought experiment: Timeouts
Change the variables below and predict what breaks first in Timeouts. The production lab can later reuse these same inputs.
Change one variable at a time. Predict the failure mode first, then move the slider and see whether your mental model matches the simplified system response.
If you are stuck on Timeouts, start by drawing the request path and marking every network hop, stateful component, queue, cache and failure boundary. Then estimate where the system will saturate.
You increase traffic by 10× in a system using Timeouts. What should you inspect first?
Pick one answer.
Which statement is the safest engineering habit when using Timeouts?
Pick one answer.
You have dashboards for traffic, latency, errors and saturation. You can change the architecture, but every change has operational cost.
Production scenario: your system uses Timeouts, traffic suddenly spikes, and p99 latency doubles. What is your first move?
Interview drill
Answer this without notes: When would you choose Timeouts, and when would you intentionally avoid it? Mention at least one bottleneck it addresses, one failure mode it introduces, and one alternative. Then quantify the workload you are designing for.
For Timeouts, treat the system as a control loop: observe load and failure, choose a bounded response, and measure whether the response stabilizes the system instead of simply moving the bottleneck somewhere else.
Numerical sanity check
When estimating capacity, distinguish average from peak. If average traffic is 4,000 RPS and the observed peak-to-average factor is 3×, design the first pass around roughly 12,000 RPS, then leave headroom for failure and growth.
Do not optimize for a memorized definition. Reason from the workload and failure mode.
What is the earliest signal that Timeouts is becoming the bottleneck: latency, saturation, errors, queue depth, or something else? Why?
Pick one answer.
What you gain, what you pay
- +Prevents cascading failures — one slow dependency can't take down the whole system.
- +Frees resources (threads, connections, memory) on failure.
- +Provides bounded latency — clients know the worst-case wait.
- +Cheap to add — usually a single config line.
- −Too-short timeouts cause false failures on legitimately slow operations.
- −Retries on top of timeouts multiply latency (3 retries * 1s = 3s total).
- −Requires tuning per call — one global timeout rarely fits all calls.
- −Server may keep doing work after the client times out (wasted resources).
How this breaks in production
- Default timeout too long (or none) — slow dependency exhausts resources.
- Timeout too short — false failures during normal latency variance.
- No deadline propagation — end-to-end latency unbounded, work wasted after client gives up.
- Server-side work continues after client-side timeout — wasted resources and possible cascades.
Don't fall into these traps
- •Relying on client defaults (often 60s or unlimited).
- •Setting only the read timeout, not the connection timeout.
- •Not propagating deadlines across service hops.
- •Forgetting to set server-side timeouts on database queries (they often default to 'forever').
Real systems using this
How real systems implement this
- gRPC deadlines — Propagates the client's deadline through the `grpc-timeout` HTTP/2 header. Each hop subtracts the time it has used and passes the reduced deadline downstream, so total latency stays bounded.
- PostgreSQL statement_timeout — Server-side query timeout. A query that exceeds it is aborted by the server, freeing the connection and the locks it held — preventing a single slow query from blocking others.
Practice saying it out loud
- Q1Why do timeouts matter so much in distributed systems?
- Q2Explain connection timeout vs read timeout.
- Q3What is deadline propagation, and why is it necessary?
- Q4How do you choose a timeout value for a new API call?
Further reading & references
Core explanations are original NO CAP material. External references are provided for deeper study and standards.
What next?
Mark as understood once the mental model clicks.
Next recommended
Circuit Breaker