Metrics, Logs, Traces
Metrics, logs, and traces are the three pillars of observability. Metrics are aggregated numeric signals (cheap, queryable, good for alerting). Logs are discrete events (rich context, expensive to store, good for debugging). Traces follow a single request across services (causality, latency breakdown). A mature observability stack uses all three — they answer different questions.
How it works
Observability — the ability to ask arbitrary questions of a running system from the outside — rests on three signals. Each has a different shape, cost, and purpose:
- Metrics: numeric, aggregated, time-series. "requests_per_second{service=api, status=200} = 1240". Cheap to collect (one counter per service+label set), queryable ("what's the p99 latency?"), good for alerting and dashboards. Bad for debugging individual requests.
- Logs: discrete events with rich context. "user=42 fetched order=99 at 10:42:03". Expensive to store and search but unmatched for understanding what happened to a specific request. Good for debugging, bad for aggregation.
- Traces: a tree of spans following a single request across services. Shows causality ("API called auth, auth called DB"), per-hop latency, and where time was spent. Essential for distributed systems, expensive to collect (sample).
Metrics come in three core types:
- Counter: monotonically increasing (total requests, total errors). Use
rate()over time to get per-second values. - Gauge: a value at a point in time (memory used, queue depth, active connections). Can go up or down.
- Histogram: distribution of values (request latencies). Lets you compute p50, p95, p99. Stores counts in buckets, not individual values — efficient but lossy.
Metrics carry labels (dimensions) for filtering and grouping: http_requests_total{service=api, route=/orders, status=200}. Cardinality matters — a metric with a label per user ID will explode the time series database. Rule of thumb: keep cardinality under 10 per metric; under 100 is borderline; over 1000 is a problem.
Metrics are the right tool for: dashboards, alerting, capacity planning, SLO tracking. They're the wrong tool for: "why did this specific request fail?"
Logs are discrete events. The key decision is structured vs unstructured:
- Unstructured:
"User 42 fetched order 99 in 80ms". Human-readable, hard to query. - Structured:
{user_id:42, order_id:99, duration_ms:80, level:info, msg:"order_fetched"}. Queryable (level=error AND duration_ms>1000), aggregatable.
Structured logs are the modern default. They cost more storage but enable everything downstream: search, alerting, derived metrics. JSON is the standard format; tools like Loki, Elasticsearch, Datadog ingest and query it.
Log levels: DEBUG, INFO, WARN, ERROR, FATAL. Be deliberate — logging everything at INFO floods the system and costs money. Log the things you'd actually want to see in an incident: state transitions, errors with context, business events.
Correlate logs with traces by including the trace_id in every log line. This lets you jump from a slow span in a trace to the logs emitted during that span — the most powerful debugging combination.
A trace is a tree of spans, where each span represents a unit of work in a request. The root span is the incoming request; child spans are downstream calls (DB query, RPC, internal computation). Spans carry:
- Operation name:
db.insert,http.get,auth.verify - Start time and duration: when did it start, how long did it take
- Tags: key-value attributes (user_id, status, db.statement)
- Span context: trace_id, span_id, parent_span_id — for the tree structure
Distributed tracing propagates the trace context across services via headers (W3C Trace Context: traceparent). Each service creates spans for its work, linked to the same trace_id. The result: a single view of the entire request, even across 20 services.
The cost is real — collecting 100% of traces is too much data. Production systems sample: 1% of normal traffic, 100% of errors (tail-based sampling). OpenTelemetry is the open standard that most modern systems use.
Individually, each pillar has limits. The power is in correlation: a slow trace shows you where the time went (the DB span), the metrics show you whether this is new (p99 spike at 10am), and the logs show you what happened (the DB was doing a full table scan). This requires shared identifiers — trace_id in logs, trace_id in metrics exemplars, consistent service labels across all three. OpenTelemetry does this for you; rolling your own is error-prone.
Your p99 latency spiked from 100ms to 800ms at 10am. Which pillar tells you *where* the time went?
Pick one answer.
You add a `user_id` label to a per-request metric so you can filter by user. What goes wrong?
Pick one answer.
Why is tail-based sampling preferred over head-based sampling for traces?
Pick one answer.
Engineering mental model
Mental model. Think of Metrics, Logs, Traces as a deliberate boundary in a system. The boundary exists because something becomes harder to manage when everything is done in one place: latency, scale, failure isolation, consistency, cost, or team ownership. The useful question is not “what does Metrics, Logs, Traces mean?” but “what pressure makes this boundary worth introducing, and what new failure mode does it create?”
Before choosing Metrics, Logs, Traces, name the workload, the critical user path, the dominant bottleneck, the failure you are trying to absorb, and the trade-off you are willing to accept. If you cannot name those five things, the design is probably premature.
// Pseudocode
request = receive()
result = metrics_logs_traces(request)
return result
// Production questions:
// 1. What happens on timeout?
// 2. Can this operation be retried safely?
// 3. What is the bottleneck?Back-of-the-envelope reasoning
Numerical lens: write down traffic, payload size, read/write ratio, peak multiplier and durability target before choosing a component. The numbers should justify the architecture.
Interactive thought experiment: Metrics, Logs, Traces
Change the variables below and predict what breaks first in Metrics, Logs, Traces. The production lab can later reuse these same inputs.
Change one variable at a time. Predict the failure mode first, then move the slider and see whether your mental model matches the simplified system response.
If you are stuck on Metrics, Logs, Traces, start by drawing the request path and marking every network hop, stateful component, queue, cache and failure boundary. Then estimate where the system will saturate.
You increase traffic by 10× in a system using Metrics, Logs, Traces. What should you inspect first?
Pick one answer.
Which statement is the safest engineering habit when using Metrics, Logs, Traces?
Pick one answer.
You have dashboards for traffic, latency, errors and saturation. You can change the architecture, but every change has operational cost.
Production scenario: your system uses Metrics, Logs, Traces, traffic suddenly spikes, and p99 latency doubles. What is your first move?
Interview drill
Answer this without notes: When would you choose Metrics, Logs, Traces, and when would you intentionally avoid it? Mention at least one bottleneck it addresses, one failure mode it introduces, and one alternative. Then quantify the workload you are designing for.
A useful engineering lens for Metrics, Logs, Traces: define the problem it solves, the simpler design that fails first, the constraint that forces you to introduce this concept, and the new failure modes the concept creates.
Numerical sanity check
Back-of-the-envelope reasoning beats fake precision. State your traffic, payload, concurrency and growth assumptions explicitly, then calculate enough to know whether the current architecture is orders of magnitude away from the target.
Do not optimize for a memorized definition. Reason from the workload and failure mode.
Imagine the simplest version of a system using Metrics, Logs, Traces. What breaks first as traffic grows by 10×, and what would you change before reaching 100×?
Pick one answer.
What you gain, what you pay
- +Metrics: cheap, queryable, ideal for dashboards and alerting.
- +Logs: rich context, ideal for debugging specific events.
- +Traces: causal structure, ideal for distributed-system latency analysis.
- +Combined: answer any diagnostic question if signals are correlated.
- −Metrics: limited detail; cardinality can explode.
- −Logs: expensive at scale; search is slow without structure.
- −Traces: require sampling; complex to instrument across services.
- −Combined: tooling complexity (Prometheus + Loki + Tempo, or Datadog) and cost.
How this breaks in production
- High-cardinality metrics labels exploding the TSDB.
- Unstructured logs that can't be queried in an incident.
- Trace context not propagated across service boundaries — broken traces.
- Sampling that drops the interesting traces (errors) — keep tail-based.
Don't fall into these traps
- •Using metrics for high-cardinality data (user IDs in labels).
- •Logging unstructured text instead of JSON.
- •Not propagating trace IDs across services, breaking the trace tree.
- •Sampling 100% of traces (cost) or 0.1% and missing errors (visibility).
Real systems using this
How real systems implement this
- OpenTelemetry — Open standard for emitting metrics, logs, and traces with shared context (trace IDs in logs, exemplars in metrics). The de-facto way to instrument a modern service.
- Grafana Stack (Prometheus + Loki + Tempo) — Open-source pillar stack: Prometheus for metrics, Loki for logs, Tempo for traces. Grafana visualizes all three with shared trace IDs for correlation.
Practice saying it out loud
- Q1Compare metrics, logs, and traces. When would you use each?
- Q2What is cardinality, and why does it matter for metrics?
- Q3How does distributed tracing work across service boundaries?
- Q4Why is correlation between the three pillars so powerful, and how do you achieve it?
Further reading & references
Core explanations are original NO CAP material. External references are provided for deeper study and standards.
What next?
Mark as understood once the mental model clicks.
Next recommended
Instrumentation