Performance Monitoring
Performance monitoring tracks how fast your system is: latency (how long requests take), throughput (how many you handle), and error rate (how many fail). The three golden signals (plus saturation) make up the USE/RED method. The key tool is the latency histogram and its percentiles — p50 (median), p95 (95th percentile), p99 — because averages hide the long tail that affects real users.
Foundational.
How it works
Performance monitoring tracks four golden signals (Google SRE):
- Latency: time to serve a request. Tracked as percentiles (p50, p95, p99), not averages.
- Traffic: how much demand the service is handling (requests/sec, transactions/sec).
- Errors: the rate of failed requests (5xx, application errors).
- Saturation: how "full" the service is (CPU, memory, disk, queue depth, connection pool).
Together these tell you whether the system is healthy, fast, and within capacity. Two methodologies distill them:
- RED method (for services): Rate, Errors, Duration (latency). One chart per service.
- USE method (for resources): Utilization, Saturation, Errors. One chart per resource (CPU, disk, network).
A complete dashboard covers both: RED for each service, USE for each resource dependency.
Percentiles describe the distribution of latencies:
- p50 (median): half of requests are faster, half slower. The "typical" experience.
- p95: 95% of requests are faster than this. Captures the slow tail.
- p99: 99% of requests are faster. Captures the worst experience 1% of users see.
- p99.9: 1 in 1000 requests. Useful for large-volume services where 1% is still many users.
Why percentile, not average? Averages are skewed by outliers. If 1% of requests take 5 seconds and 99% take 50ms, the average is 100ms — looks fine, but 1% of users see 5 seconds. p99 = 5s reveals the problem.
Different percentiles capture different issues. p50 is affected by general slowness (DB load, code path length). p99 is affected by outliers (GC pauses, cold caches, retries, contention). A regression that affects only the tail shows up in p99 but not p50.
Track multiple percentiles. SLOs are usually set on p99 or p99.9 — "99% of requests complete within 300ms."
APM (Application Performance Monitoring) tools automate performance monitoring. They instrument your code (via agent or SDK) to capture:
- Per-request latency (with histograms).
- Per-service dependency latency (DB, cache, RPC).
- Error rates.
- Distributed traces on sampled requests.
- Host metrics (CPU, memory, GC).
Examples: Datadog, New Relic, Dynatrace (commercial); Jaeger, Zipkin, Prometheus+Grafana (open source).
The value of APM is correlation: when p99 spikes, you can drill into a slow trace, see which span was slow (e.g., a particular DB query), then look at the DB metrics to see if it was a query plan regression or saturation. Manual instrumentation can do this too, but APM does it out of the box.
The cost is real — APM agents add overhead (1-5% CPU) and the tools are priced by host or by request volume. Many teams use them anyway because the productivity gain dwarfs the cost.
Naive alerting ("page if p99 > 1s") fires constantly because latency has natural variance. Better: alert when the SLO burn rate is high — i.e., when you're consuming error budget faster than allowed. "SLO burn rate of 14x over 1 hour" means you'll exhaust the 30-day budget in ~2 hours if not fixed. This is the Google SRE multi-window multi-burn-rate approach: short-window fast-burn (page) catches acute outages; long-window slow-burn (ticket) catches chronic regressions. It dramatically reduces alert fatigue compared to raw thresholds.
Saturation tells you how close the system is to its limits. Useful saturation metrics:
- CPU utilization: high CPU = potential bottleneck. But CPU at 90% isn't always bad — if it's steady and the queue isn't growing.
- Memory utilization: high = OOM risk. Watch the trend, not just the level.
- Queue depth: request queue, DB connection pool, message queue lag. Growing queues are an early warning sign of overload.
- Connection pool utilization: at 80%+ you're near saturation; new connections may queue.
- Disk I/O wait: high = disk is the bottleneck.
Saturation is a leading indicator. Latency rises sharply near 100% utilization (queueing theory: at 80% utilization, average queue length is 4; at 90%, it's 9; at 99%, it's 99). Track utilization and alert before saturation, not after.
Your average request latency is 100ms. The p99 is 2 seconds. Which one should you investigate?
Pick one answer.
Latency is rising and CPU is at 95%. Your throughput (requests/sec) is steady. What's likely happening?
Pick one answer.
What's the difference between the RED and USE methods?
Pick one answer.
Engineering mental model
Mental model. Think of Performance Monitoring as a deliberate boundary in a system. The boundary exists because something becomes harder to manage when everything is done in one place: latency, scale, failure isolation, consistency, cost, or team ownership. The useful question is not “what does Performance Monitoring mean?” but “what pressure makes this boundary worth introducing, and what new failure mode does it create?”
Before choosing Performance Monitoring, name the workload, the critical user path, the dominant bottleneck, the failure you are trying to absorb, and the trade-off you are willing to accept. If you cannot name those five things, the design is probably premature.
// Pseudocode
request = receive()
result = performance_monitoring(request)
return result
// Production questions:
// 1. What happens on timeout?
// 2. Can this operation be retried safely?
// 3. What is the bottleneck?Back-of-the-envelope reasoning
Numerical lens: write down traffic, payload size, read/write ratio, peak multiplier and durability target before choosing a component. The numbers should justify the architecture.
Interactive thought experiment: Performance Monitoring
Change the variables below and predict what breaks first in Performance Monitoring. The production lab can later reuse these same inputs.
Change one variable at a time. Predict the failure mode first, then move the slider and see whether your mental model matches the simplified system response.
If you are stuck on Performance Monitoring, start by drawing the request path and marking every network hop, stateful component, queue, cache and failure boundary. Then estimate where the system will saturate.
You increase traffic by 10× in a system using Performance Monitoring. What should you inspect first?
Pick one answer.
Which statement is the safest engineering habit when using Performance Monitoring?
Pick one answer.
You have dashboards for traffic, latency, errors and saturation. You can change the architecture, but every change has operational cost.
Production scenario: your system uses Performance Monitoring, traffic suddenly spikes, and p99 latency doubles. What is your first move?
Interview drill
Answer this without notes: When would you choose Performance Monitoring, and when would you intentionally avoid it? Mention at least one bottleneck it addresses, one failure mode it introduces, and one alternative. Then quantify the workload you are designing for.
A useful engineering lens for Performance Monitoring: define the problem it solves, the simpler design that fails first, the constraint that forces you to introduce this concept, and the new failure modes the concept creates.
Numerical sanity check
Back-of-the-envelope reasoning beats fake precision. State your traffic, payload, concurrency and growth assumptions explicitly, then calculate enough to know whether the current architecture is orders of magnitude away from the target.
Do not optimize for a memorized definition. Reason from the workload and failure mode.
Imagine the simplest version of a system using Performance Monitoring. What breaks first as traffic grows by 10×, and what would you change before reaching 100×?
Pick one answer.
What you gain, what you pay
- +Quantifies "feels slow" into measurable numbers (p50, p95, p99).
- +Percentiles expose the tail that averages hide.
- +Saturation is a leading indicator — predicts latency before users feel it.
- +APM tools correlate latency, traces, and resource metrics for fast diagnosis.
- −APM agents add 1-5% overhead and cost.
- −Histograms are lossy — buckets aggregate away fine detail.
- −Naive threshold-based alerting causes fatigue.
- −p99 has high variance at low traffic volumes — needs smoothing.
How this breaks in production
- Alerting on averages instead of percentiles — outliers hidden until users complain.
- Alerting on raw thresholds instead of SLO burn rates — fatigue from natural variance.
- Missing saturation metrics — surprised by latency spikes when utilization crosses 90%.
- Tracking only p99 — misses general slowness that affects p50.
Don't fall into these traps
- •Using average latency instead of percentiles.
- •Alerting on every p99 spike (natural variance causes noise).
- •Not correlating latency with saturation — symptom without cause.
- •Skipping APM to save cost — the productivity loss is greater.
Real systems using this
How real systems implement this
- Google SRE Four Golden Signals — Latency, traffic, errors, saturation — the canonical set of metrics for any production service. Formalized in the Google SRE book and adopted across the industry.
- Datadog APM — Auto-instruments code via agent; captures per-service RED metrics, distributed traces, and resource USE metrics in one correlated view. Industry-standard commercial APM.
Practice saying it out loud
- Q1Why are percentiles better than averages for latency? What's the difference between p50, p95, and p99?
- Q2What are the four golden signals of monitoring?
- Q3Explain the RED and USE methods. When do you use each?
- Q4How would you alert on latency without causing alert fatigue?
Further reading & references
Core explanations are original NO CAP material. External references are provided for deeper study and standards.
What next?
Mark as understood once the mental model clicks.
Next recommended
Metrics, Logs, Traces