Throttling
Throttling is a server-side mechanism that limits how fast a client can send requests, smoothing bursts and protecting downstream resources. It uses the same algorithms as rate limiting (token bucket, leaky bucket) but is enforced at the consumer or service boundary, often dynamically based on system health. The key distinction: rate limiting rejects excess traffic with 429s; throttling delays or shapes it.
Foundational.
How it works
Throttling means deliberately limiting the rate at which a client is allowed to do work. It is closely related to rate limiting and the terms are often used interchangeably, but the intent differs:
- Rate limiting is a policy: "you may send at most N requests per minute." Excess traffic is rejected with a 429.
- Throttling is a control mechanism: "I will process your requests at no more than R per second." Excess traffic is delayed, queued, or shaped rather than rejected outright.
Throttling is what your AWS SDK does when it sees a ThrottlingException: it backs off and retries instead of failing. Throttling is what a Kafka consumer does when it pauses consumption to avoid overwhelming a downstream database.
Two algorithms dominate throttling:
Token bucket gives the client an allowance of capacity that refills over time. If the bucket has tokens, the request goes through immediately; if not, the client waits or gets rejected. Token bucket is good when bursts are legitimate (a user clicking rapidly) but average load must be controlled.
Leaky bucket turns a bursty input stream into a smooth output stream by queuing requests and processing them at a constant rate. It's good when downstream systems cannot tolerate bursts at all — for example, calling a third-party API that bills per second and has its own rate limits.
Both can be implemented per-client, per-tenant, or globally. Most production systems combine a per-client limit (fairness) with a global limit (protection).
Static limits are easy to reason about, but the right limit depends on system health. A common pattern is adaptive throttling: the service tracks its own latency, CPU, queue depth, or error rate, and tightens the limit as those signals degrade. AWS calls this the adaptive retry mode; gRPC's bedrock client does the same.
Adaptive throttling prevents the failure mode where a fixed limit is too high during degradation (letting the system die) or too low during health (rejecting traffic the system could have handled). The downside is harder debugging: the limit changes, so a client might be accepted at 9am and rejected at 9pm for the same load.
Rate limiting is a policy enforced at the edge: "no more than N per minute, else 429." Throttling is a control loop: "slow down to R per second so the system can keep up." A rate limiter answers a yes/no question; a throttler answers a "how fast" question. Production systems usually run both: a rate limiter at the API gateway for fairness, and throttling inside the service to protect its own dependencies.
Throttling isn't only a server concern. Client-side throttling is when a client deliberately limits its own request rate to avoid being rejected. AWS SDKs do this automatically: when they get a ThrottlingException, they switch into throttled mode and cap their own request rate, using the token bucket and adaptive algorithms above. This avoids the retry storm pattern — the client never sends more than the server can absorb.
Client-side throttling works because it's cooperative: the client and server agree on a rate. It doesn't work against adversarial clients (use rate limiting for those).
An API gateway returns 429 to clients that exceed 100 req/min. A service behind it is being overwhelmed because legit clients retry immediately. What's a better approach?
Pick one answer.
You're writing a Kafka consumer that writes to a database which can sustain 500 inserts/sec. How should you throttle?
Pick one answer.
Token bucket allows bursts; leaky bucket smooths them. When do you prefer leaky bucket?
Pick one answer.
Engineering mental model
Mental model. Think of Throttling as a deliberate boundary in a system. The boundary exists because something becomes harder to manage when everything is done in one place: latency, scale, failure isolation, consistency, cost, or team ownership. The useful question is not “what does Throttling mean?” but “what pressure makes this boundary worth introducing, and what new failure mode does it create?”
Before choosing Throttling, name the workload, the critical user path, the dominant bottleneck, the failure you are trying to absorb, and the trade-off you are willing to accept. If you cannot name those five things, the design is probably premature.
// Pseudocode
request = receive()
result = throttling(request)
return result
// Production questions:
// 1. What happens on timeout?
// 2. Can this operation be retried safely?
// 3. What is the bottleneck?Back-of-the-envelope reasoning
Numerical lens: write down traffic, payload size, read/write ratio, peak multiplier and durability target before choosing a component. The numbers should justify the architecture.
Interactive thought experiment: Throttling
Change the variables below and predict what breaks first in Throttling. The production lab can later reuse these same inputs.
Change one variable at a time. Predict the failure mode first, then move the slider and see whether your mental model matches the simplified system response.
If you are stuck on Throttling, start by drawing the request path and marking every network hop, stateful component, queue, cache and failure boundary. Then estimate where the system will saturate.
You increase traffic by 10× in a system using Throttling. What should you inspect first?
Pick one answer.
Which statement is the safest engineering habit when using Throttling?
Pick one answer.
You have dashboards for traffic, latency, errors and saturation. You can change the architecture, but every change has operational cost.
Production scenario: your system uses Throttling, traffic suddenly spikes, and p99 latency doubles. What is your first move?
Interview drill
Answer this without notes: When would you choose Throttling, and when would you intentionally avoid it? Mention at least one bottleneck it addresses, one failure mode it introduces, and one alternative. Then quantify the workload you are designing for.
A useful engineering lens for Throttling: define the problem it solves, the simpler design that fails first, the constraint that forces you to introduce this concept, and the new failure modes the concept creates.
Numerical sanity check
Back-of-the-envelope reasoning beats fake precision. State your traffic, payload, concurrency and growth assumptions explicitly, then calculate enough to know whether the current architecture is orders of magnitude away from the target.
Do not optimize for a memorized definition. Reason from the workload and failure mode.
Imagine the simplest version of a system using Throttling. What breaks first as traffic grows by 10×, and what would you change before reaching 100×?
Pick one answer.
What you gain, what you pay
- +Smooths bursts into sustainable load, protecting downstreams.
- +Cooperative — well-behaved clients get served, just slower.
- +Avoids retry storms that pure rate limiting can trigger.
- +Enables adaptive control: tighten the rate when the system is degraded.
- −Adds latency — throttled requests wait in queues.
- −Requires queue capacity: an unbounded queue is a memory leak under sustained overload.
- −Harder to reason about than a simple 429 — clients may not understand why they're slow.
- −Adaptive throttling is harder to debug: the limit moves.
How this breaks in production
- Unbounded queues — under sustained overload, queued requests grow until OOM.
- Head-of-line blocking — one slow request behind a throttle holds up faster ones.
- Fairness starvation — strict per-key throttling can starve small tenants when a hot key saturates the bucket.
- Adaptive feedback oscillation — over-tightening and over-loosening causes throughput to swing wildly.
Don't fall into these traps
- •Confusing throttling with rate limiting — they have different intent (shape vs reject).
- •Setting a static limit and never revisiting it as the system's capacity changes.
- •Throttling at the wrong layer — e.g., per-instance instead of globally.
- •Not propagating throttle state across instances — needs shared state (Redis) for global limits.
Real systems using this
How real systems implement this
- AWS SDK adaptive retry mode — Tracks throttling feedback from the service, maintains a token bucket, and dynamically caps the client's own request rate to avoid overwhelming the service — preventing retry storms.
- gRPC client-side throttling — Implements hedging and retry budgets that cap the percentage of in-flight requests that can be retries, preventing a slow upstream from cascading into a retry storm.
Practice saying it out loud
- Q1What's the difference between throttling and rate limiting? When do you use each?
- Q2Explain token bucket vs leaky bucket. Which would you choose for a third-party API that bills per request?
- Q3How does client-side adaptive throttling prevent retry storms?
- Q4How do you throttle across multiple instances of a service?
Further reading & references
Core explanations are original NO CAP material. External references are provided for deeper study and standards.
What next?
Mark as understood once the mental model clicks.
Next recommended
Circuit Breaker