Latency vs Throughput
Latency is how long one operation takes. Throughput is how many operations happen per unit time. They are related but distinct: you can have high throughput with high latency (batch processing) or low latency with low throughput (single-user app). Knowing which one to optimize is a core system design skill.
How it works
Latency is the time from 'I asked for something' to 'I got it.' Measured in milliseconds (or microseconds, or seconds). It's a per-operation metric.
Throughput is how many operations complete per unit time. Measured in requests per second (RPS), messages per second, or transactions per second. It's an aggregate metric.
A system can have low throughput and low latency (a personal blog), high throughput and high latency (batch ETL), or — the holy grail — high throughput and low latency (a well-designed API).
Little's Law connects them: Throughput ≈ Concurrency / Latency. If your system handles 100 concurrent requests at 50ms latency each, throughput ≈ 100 / 0.05 = 2,000 RPS. To increase throughput you can:
- Reduce latency (faster per request) — optimize code, cache, index.
- Increase concurrency (more parallel requests) — add servers, increase thread pool, use async I/O.
- Both — the best systems do both.
Average latency hides tail latency. If 99 requests take 10ms and 1 takes 1000ms, the average is ~20ms — looks fine. But 1% of your users see a 1-second delay. Always report p50 (median), p95, and p99. SLOs are usually written against p99: '99% of requests complete in under 200ms.'
Latency and throughput can conflict:
- Batching increases throughput but adds latency (you wait for the batch to fill).
- Caching reduces latency and increases throughput (fewer DB hits).
- Replication can reduce read latency but add write latency (sync to replicas).
- Sharding increases throughput but can add latency (cross-shard queries).
The right choice depends on what your users care about. A real-time game cares about latency. A data pipeline cares about throughput. A web API cares about both — but usually optimizes for p99 latency first, because a slow request feels worse than a rejected one.
Your API averages 50ms latency. But users complain it 'feels slow'. What should you check?
Pick one answer.
You're building a batch ETL pipeline that processes 10TB of data nightly. What should you optimize for?
Pick one answer.
Little's Law, derived. In 1961, John Little proved (and later published in 1961, formally in 2011) that for any stable queueing system: L = λ × W — the average number of items in the system (L) equals the arrival rate (λ) times the average time an item spends in the system (W). Rearranged for our purposes: throughput = concurrency / latency.
This is profound because it holds for any stable system — no assumptions about distribution, queue discipline, or service time. It connects three observable quantities you can always measure.
Concrete example: your API server handles 100 concurrent requests (concurrency=100), each taking 50ms (latency=0.05s). Throughput = 100 / 0.05 = 2,000 RPS. If you want 10,000 RPS, you have three dials:
- Reduce latency to 10ms → 100 / 0.01 = 10,000 RPS.
- Increase concurrency to 500 → 500 / 0.05 = 10,000 RPS.
- Both → 250 / 0.025 = 10,000 RPS.
The hard part: which dial is movable? Reducing latency often means finding the actual bottleneck (CPU? DB? lock contention?). Increasing concurrency often means adding servers (more $), increasing thread pool size (more memory), or going async (more code complexity). Little's Law tells you the equation; the system design tells you which variables you can actually move.
Why p99 — not average — is the SLO. Production latencies are never normally distributed. They have a long right tail: most requests finish fast (cache hits, simple queries), but a small fraction hit cold caches, GC pauses, lock contention, or disk seeks — and these outliers take 10x-100x the median.
If 99% of your requests take 10ms and 1% take 1000ms:
- Average: ~20ms (looks great in dashboards).
- Median (p50): 10ms (also looks great).
- p95: 10ms (still looks great).
- p99: 1000ms — and this is what users complain about.
The average hides the tail because 1% of a billion requests is 10 million requests per billion — a lot of users. A user who hits your API 20 times in a session has a ~18% chance of seeing at least one p99 latency. So even if your p99 is rare, users experience it regularly.
Good SLOs are written against multiple percentiles: 'p50 < 50ms, p95 < 100ms, p99 < 500ms.' This captures both the typical experience (p50) and the worst-case most users see (p99). p99.9 (one in a thousand) is useful for backend monitoring but rarely belongs in a user-facing SLO — only one in a thousand users sees it, and chasing p99.9 often means optimizing for noise.
The deeper insight: p99 is also a leading indicator. When p99 starts creeping up before p50 does, it usually means a bottleneck is forming (queue depth growing, cache hit rate dropping, GC pause increasing). Catching p99 drift early lets you act before users notice.
Batch processing optimizes for throughput at the cost of latency: collect 10,000 records, sort them, write to a columnar store, run analytics. Each record's 'latency' is hours, but throughput is enormous (Spark can process TB/hr). Real-time systems optimize for latency at the cost of throughput: each request is served from cache in <10ms, but the system can't match batch throughput because per-request overhead dominates. The architectural choice between batch and streaming is fundamentally a choice of which metric you're willing to sacrifice. Most mature systems do both: a real-time path for user-facing reads, a batch path for analytics. Lambda and Kappa architectures formalize this dual-path approach. Don't try to make one system do both — you'll get mediocre at each.
Why does serving an image from a CDN edge PoP typically give 5-10ms latency, while serving the same image from the origin gives 80-300ms latency — even when the origin has plenty of bandwidth and CPU?
Pick one answer.
Your server has a fixed concurrency cap of 100 (thread pool). At 500 RPS × 0.02s avg latency, you have 10 in-flight requests — well under the cap. At 1000 RPS × 0.02s = 20 in-flight, still under the cap. But p99 of 2000ms means 1% of requests take 2s, holding a thread for that whole time.
Your API has p50 latency of 20ms, p99 latency of 100ms, and handles 500 RPS. Traffic doubles to 1000 RPS. p50 stays at 20ms but p99 jumps to 2000ms. Users start timing out. Diagnose using Little's Law.
Engineering mental model
Mental model. Think of Latency vs Throughput as a deliberate boundary in a system. The boundary exists because something becomes harder to manage when everything is done in one place: latency, scale, failure isolation, consistency, cost, or team ownership. The useful question is not “what does Latency vs Throughput mean?” but “what pressure makes this boundary worth introducing, and what new failure mode does it create?”
Before choosing Latency vs Throughput, name the workload, the critical user path, the dominant bottleneck, the failure you are trying to absorb, and the trade-off you are willing to accept. If you cannot name those five things, the design is probably premature.
// Pseudocode
request = receive()
result = latency_vs_throughput(request)
return result
// Production questions:
// 1. What happens on timeout?
// 2. Can this operation be retried safely?
// 3. What is the bottleneck?Back-of-the-envelope reasoning
Example: 10M requests/day ÷ 86,400 ≈ 116 requests/s average. Design for peak rather than average; a 10× peak is ≈ 1,160 requests/s.
Interactive thought experiment: Latency vs Throughput
Change the variables below and predict what breaks first in Latency vs Throughput. The production lab can later reuse these same inputs.
Change one variable at a time. Predict the failure mode first, then move the slider and see whether your mental model matches the simplified system response.
If you are stuck on Latency vs Throughput, start by drawing the request path and marking every network hop, stateful component, queue, cache and failure boundary. Then estimate where the system will saturate.
You increase traffic by 10× in a system using Latency vs Throughput. What should you inspect first?
Pick one answer.
Which statement is the safest engineering habit when using Latency vs Throughput?
Pick one answer.
You have dashboards for traffic, latency, errors and saturation. You can change the architecture, but every change has operational cost.
Production scenario: your system uses Latency vs Throughput, traffic suddenly spikes, and p99 latency doubles. What is your first move?
Interview drill
Answer this without notes: When would you choose Latency vs Throughput, and when would you intentionally avoid it? Mention at least one bottleneck it addresses, one failure mode it introduces, and one alternative. Then quantify the workload you are designing for.
For Latency vs Throughput, treat the system as a control loop: observe load and failure, choose a bounded response, and measure whether the response stabilizes the system instead of simply moving the bottleneck somewhere else.
Numerical sanity check
When estimating capacity, distinguish average from peak. If average traffic is 4,000 RPS and the observed peak-to-average factor is 3×, design the first pass around roughly 12,000 RPS, then leave headroom for failure and growth.
Do not optimize for a memorized definition. Reason from the workload and failure mode.
What is the earliest signal that Latency vs Throughput is becoming the bottleneck: latency, saturation, errors, queue depth, or something else? Why?
Pick one answer.
What you gain, what you pay
- +Latency optimizations (caching, indexing) often improve user experience directly.
- +Throughput optimizations (batching, sharding) often reduce cost per operation.
- +Little's Law gives a clear mental model for reasoning about both.
- −Latency and throughput can conflict (batching, replication, sharding add latency for throughput).
- −Optimizing one without the other can create bottlenecks (fast code + single DB = not scalable).
- −Percentile measurement is harder than average — requires real monitoring.
How this breaks in production
- Optimizing average latency when p99 is the problem.
- Adding throughput (more servers) when latency is the bottleneck (slow code).
- Batching for throughput in a latency-sensitive system (makes it feel slower).
Don't fall into these traps
- •Reporting average latency instead of p99.
- •Treating 'fast' and 'scalable' as the same thing.
- •Forgetting that throughput without latency targets is meaningless — 1M RPS at 30s latency is useless for most apps.
Real systems using this
How real systems implement this
- CDNs (Cloudflare, Fastly) — Optimize for latency — cache at the edge so users get content in <50ms globally. Throughput is achieved by massive parallelism across edge locations.
- Kafka — Optimizes for throughput — batches messages, appends to logs sequentially. Per-message latency is higher than a synchronous queue, but throughput is dramatically higher.
Practice saying it out loud
- Q1What's the difference between latency and throughput? When do they conflict?
- Q2What is Little's Law and how is it useful?
- Q3Why do we measure p99 instead of average latency?
- Q4Your system has high throughput but users say it feels slow. What do you investigate?
Further reading & references
Core explanations are original NO CAP material. External references are provided for deeper study and standards.
What next?
Mark as understood once the mental model clicks.
Next recommended
Availability vs Consistency