Instrumentation
Instrumentation is the code that emits metrics, logs, and traces from inside your application — the difference between "the service is slow" and "the db.insert call in /orders is taking 800ms." Modern instrumentation is built on OpenTelemetry, structured logging, and consistent labeling. Without instrumentation, observability tools have nothing to show; with good instrumentation, every question about production is answerable in minutes.
How it works
Instrumentation is the practice of adding observability signals to your code. There are three categories:
- Metrics: emit counters, gauges, and histograms as the code runs. "increment http_requests_total{route=/orders, status=200}."
- Logs: emit structured log events. "{"level":"info","msg":"order_created","order_id":42,"user_id":7,"trace_id":"abc"}."
- Traces: start spans for units of work, with parent-child relationships. "span: db.insert order_id=42, parent_span: http /orders."
The goal is to instrument once and emit to any backend (Prometheus, Datadog, Jaeger, Honeycomb) via OpenTelemetry. This avoids vendor lock-in and lets you standardize across services.
Structured logging is non-negotiable in modern systems. Instead of "Order 42 created for user 7", emit {"msg":"order_created","order_id":42,"user_id":7,"level":"info","ts":"2024-01-15T10:42:03Z","trace_id":"abc"}. JSON is the standard format.
Why structured? Because logs are queried and aggregated. With unstructured text, finding "all errors for user 7 today" requires a regex search across gigabytes. With structured JSON, it's level=error AND user_id=7 AND ts>today — instant, indexed.
Log the things you'd want in an incident:
- State transitions: "order status changed from pending to confirmed."
- Business events: "payment received for order 42."
- Errors with context: not just "500 internal error" but "db.insert failed: connection refused, order_id=42, user_id=7."
- Slow operations: "db.query took 1800ms (threshold 1000ms)."
Always include the trace_id in log lines. This lets you click from a slow span in a trace to the logs emitted during that span — the most powerful debugging combination in modern observability.
Good metrics instrumentation:
- Count what matters: http_requests_total, orders_created, errors_total. Use counters for cumulative totals.
- Gauge what's stateful: queue_depth, active_connections, cache_size. Gauges go up and down.
- Histogram what's distributed: request_latency, db_query_duration. Use buckets to compute percentiles.
- Label for filtering and grouping: route, status_code, tenant. But keep cardinality bounded.
The cardinality trap: each unique label combination is a separate time series. A metric with user_id as a label creates one series per user — millions of series, even if rarely queried. Rule of thumb: low-cardinality labels only (route, status, instance, tenant). High-cardinality data (user IDs, request IDs) goes in logs and traces, not metrics.
Use a metrics library (Prometheus client, OpenTelemetry metrics) that handles the bookkeeping — incrementing, exposing on /metrics, batching. Manual metrics are error-prone.
Tracing instrumentation is the most work but the most valuable for distributed systems. The pattern:
- Propagate context: every outbound call (HTTP, gRPC, message queue) carries the trace context (W3C
traceparentheader). This connects spans across services. - Create spans: wrap each unit of work in a span. The span carries: name, start time, duration, attributes (key-value tags), events (timestamped logs within the span), and status.
- Sample: don't trace 100% of requests (too much data). Sample 1% of normal traffic, 100% of errors. Use tail-based sampling if possible — keep all errors, sample the rest.
Auto-instrumentation: most languages have libraries that auto-instrument common operations (HTTP server, HTTP client, DB driver, message queue). Use these as a baseline. Add custom spans for business logic.
The result: a trace shows the full request lifecycle across services, with per-span timing. "The /orders request took 2s because the db.insert took 1.8s, which was waiting on a lock."
OpenTelemetry auto-instrumentation libraries (and APM agents like Datadog's) capture the standard signals: HTTP requests, DB calls, cache hits, message queue sends. This is enough to see the shape of your system. But it doesn't capture business context: which tenant, which feature flag, which checkout step. For that, you need manual instrumentation — custom spans around business operations, custom attributes on existing spans. The pattern: start with auto-instrumentation to get something working, then add manual instrumentation where it matters — the critical paths and the business events.
Common instrumentation pitfalls:
- High-cardinality labels on metrics — explodes the TSDB.
- Unstructured logs — searchable only by text match, not aggregatable.
- Trace context not propagated across service boundaries — traces are broken.
- Logging PII or secrets — legal and security problems.
- Insufficient sampling — collecting 100% of traces is too much data.
- Excessive sampling — collecting 0.01% means you miss rare events.
- Not including business context in spans and logs — the trace shows what happened but not for whom.
A useful test: pick a recent production issue. Could you have diagnosed it from the traces, logs, and metrics you currently emit? If not, what's missing? That's the next instrumentation to add.
You add a `user_id` label to a per-request metric so you can filter latency by user. What's the problem?
Pick one answer.
Why should every log line include the trace_id?
Pick one answer.
Auto-instrumentation captures HTTP, DB, and cache calls. What does it miss that you need to add manually?
Pick one answer.
Engineering mental model
Mental model. Think of Instrumentation as a deliberate boundary in a system. The boundary exists because something becomes harder to manage when everything is done in one place: latency, scale, failure isolation, consistency, cost, or team ownership. The useful question is not “what does Instrumentation mean?” but “what pressure makes this boundary worth introducing, and what new failure mode does it create?”
Before choosing Instrumentation, name the workload, the critical user path, the dominant bottleneck, the failure you are trying to absorb, and the trade-off you are willing to accept. If you cannot name those five things, the design is probably premature.
// Pseudocode
request = receive()
result = instrumentation(request)
return result
// Production questions:
// 1. What happens on timeout?
// 2. Can this operation be retried safely?
// 3. What is the bottleneck?Back-of-the-envelope reasoning
Numerical lens: write down traffic, payload size, read/write ratio, peak multiplier and durability target before choosing a component. The numbers should justify the architecture.
Interactive thought experiment: Instrumentation
Change the variables below and predict what breaks first in Instrumentation. The production lab can later reuse these same inputs.
Change one variable at a time. Predict the failure mode first, then move the slider and see whether your mental model matches the simplified system response.
If you are stuck on Instrumentation, start by drawing the request path and marking every network hop, stateful component, queue, cache and failure boundary. Then estimate where the system will saturate.
You increase traffic by 10× in a system using Instrumentation. What should you inspect first?
Pick one answer.
Which statement is the safest engineering habit when using Instrumentation?
Pick one answer.
You have dashboards for traffic, latency, errors and saturation. You can change the architecture, but every change has operational cost.
Production scenario: your system uses Instrumentation, traffic suddenly spikes, and p99 latency doubles. What is your first move?
Interview drill
Answer this without notes: When would you choose Instrumentation, and when would you intentionally avoid it? Mention at least one bottleneck it addresses, one failure mode it introduces, and one alternative. Then quantify the workload you are designing for.
A useful engineering lens for Instrumentation: define the problem it solves, the simpler design that fails first, the constraint that forces you to introduce this concept, and the new failure modes the concept creates.
Numerical sanity check
Back-of-the-envelope reasoning beats fake precision. State your traffic, payload, concurrency and growth assumptions explicitly, then calculate enough to know whether the current architecture is orders of magnitude away from the target.
Do not optimize for a memorized definition. Reason from the workload and failure mode.
Imagine the simplest version of a system using Instrumentation. What breaks first as traffic grows by 10×, and what would you change before reaching 100×?
Pick one answer.
What you gain, what you pay
- +Makes a service debuggable — every production question answerable in minutes.
- +Structured logs and trace context enable correlation across pillars.
- +OpenTelemetry prevents vendor lock-in.
- +Business context in traces/logs turns data into actionable insight.
- −Adds code complexity — instrumentation is its own concern.
- −Has performance cost (1-5% CPU for full instrumentation).
- −Cardinality traps can break the metrics backend.
- −Requires ongoing maintenance — instrumentation rots as code changes.
How this breaks in production
- High-cardinality metric labels exploding the TSDB.
- Unstructured logs that can't be queried in an incident.
- Trace context not propagated across service boundaries — broken traces.
- Logging secrets/PII — security and compliance violations.
Don't fall into these traps
- •Using unstructured text logs instead of JSON.
- •Forgetting to include trace_id in log records.
- •Adding high-cardinality labels (user_id, request_id) to metrics.
- •Relying only on auto-instrumentation — misses business context.
Real systems using this
How real systems implement this
- OpenTelemetry — CNCF standard for instrumentation. Provides SDKs in every major language that emit metrics, logs, and traces via a unified API, exportable to any backend via OTLP.
- Datadog APM agent — Auto-instruments HTTP, DB, and cache calls via bytecode manipulation or library hooks. Adds business context via custom spans. Commercial but turnkey.
Practice saying it out loud
- Q1What is OpenTelemetry and why has it become the standard for instrumentation?
- Q2Why should high-cardinality data (user_id, request_id) go in logs and traces, not metrics?
- Q3How does trace context propagation work across service boundaries?
- Q4What's the difference between auto-instrumentation and manual instrumentation?
Further reading & references
Core explanations are original NO CAP material. External references are provided for deeper study and standards.
What next?
Mark as understood once the mental model clicks.
Next recommended
Metrics, Logs, Traces