Health Monitoring
Health monitoring continuously checks whether a service is functioning, typically via a /health endpoint that returns 200 OK or an error. The two probes — liveness ("is the process alive?") and readiness ("is the process ready to serve traffic?") — drive orchestration decisions: restarting a dead container vs removing a not-ready one from the load balancer. Confusing them is one of the most common causes of cascading failures in Kubernetes.
How it works
Health monitoring answers two questions continuously: is the process alive? Is the process ready to serve traffic? The two are distinct and require different probes.
- Liveness probe: "is the process running?" If it fails, the orchestrator (Kubernetes) restarts the container. Use this to catch deadlocks, infinite loops, and memory leaks that leave the process running but non-functional.
- Readiness probe: "is the process ready to serve traffic?" If it fails, the load balancer removes it from rotation but the orchestrator doesn't restart it. Use this for transient unavailability: waiting on a DB connection, warming a cache, or a downstream service being down.
The cardinal rule: liveness failures restart, readiness failures reroute. Confusing them causes problems.
A typical health endpoint:
-
GET /health (liveness): returns 200 if the process can answer HTTP at all. The simplest version just returns 200 unconditionally — "the process is up." A better version checks that critical background threads are alive.
-
GET /ready (readiness): returns 200 if the service can serve traffic. This typically checks downstream dependencies: can I reach the DB? Is the cache warm? Is the message queue reachable? If any critical dependency is down, return 503.
The trap: making /ready check too many things. If /ready depends on every downstream service, then any downstream outage takes your service out of the load balancer — even if your service could serve some requests degraded. Use graceful degradation instead: /ready should check critical dependencies, not all of them.
Also: don't make /ready do expensive work. A 200ms /ready check called every 5s by 100 pods is real load. Keep it fast (<10ms).
The most common mistake: using liveness for readiness, or vice versa. Example:
- Liveness checks the DB: if the DB is down, the liveness probe fails, the orchestrator restarts the container. The container restarts, hits the same down DB, restarts again — a restart storm that does nothing to fix the problem and wastes resources.
- Readiness never fails: the service stays in the load balancer even when it can't serve requests. Users see timeouts instead of being routed to a healthy replica.
Another classic: liveness probe with a too-short initial delay. The container hasn't finished starting; liveness fails; the orchestrator kills it before it's ready. This creates a CrashLoopBackOff. Use a startup probe instead — it tells the orchestrator to wait without killing.
Finally: health endpoints that return 200 even when the service is broken. This happens when /health only checks "can I respond to HTTP" but not "am I functional." The probe is a tautology. Always include real checks.
Health checks (liveness/readiness probes) are internal: the orchestrator asks the service if it's healthy. Availability monitoring is external: a synthetic probe from outside the data center asks if users can reach the service. Both are needed. Health checks catch container-level issues (zombie processes, broken dependencies). Availability monitoring catches network-level issues (DNS, load balancer, region routing). A service can pass its liveness probe but be unreachable from the internet — only availability monitoring catches that.
Health monitoring is the foundation of self-healing: when the orchestrator detects a problem, it acts automatically — restarting containers, rerouting traffic, scaling up. This means most transient failures (a deadlocked thread, a memory spike, a brief downstream outage) are handled without human intervention.
But self-healing has limits:
- It can't fix configuration bugs — restarting doesn't change the code.
- It can mask underlying issues — if a service is constantly restarting, the symptom (low availability) is hidden but the root cause is still there.
- It can cause cascading failures if misconfigured — e.g., aggressive liveness probes during a downstream outage cause restart storms.
Monitor the monitors: track restart counts, readiness flaps, and liveness failures. If they're spiking, there's a problem to investigate — even if the service looks healthy from outside.
Your service depends on a downstream API. The downstream goes down. Your service can still serve cached responses but not fresh data. What should your readiness probe do?
Pick one answer.
What goes wrong if you use the liveness probe to check the database connection?
Pick one answer.
Your Java service takes 60 seconds to start (JVM warmup, cache loading). Liveness probe fails during startup. What's the right fix?
Pick one answer.
Engineering mental model
Mental model. Think of Health Monitoring as a deliberate boundary in a system. The boundary exists because something becomes harder to manage when everything is done in one place: latency, scale, failure isolation, consistency, cost, or team ownership. The useful question is not “what does Health Monitoring mean?” but “what pressure makes this boundary worth introducing, and what new failure mode does it create?”
Before choosing Health Monitoring, name the workload, the critical user path, the dominant bottleneck, the failure you are trying to absorb, and the trade-off you are willing to accept. If you cannot name those five things, the design is probably premature.
// Pseudocode
request = receive()
result = health_monitoring(request)
return result
// Production questions:
// 1. What happens on timeout?
// 2. Can this operation be retried safely?
// 3. What is the bottleneck?Back-of-the-envelope reasoning
Numerical lens: write down traffic, payload size, read/write ratio, peak multiplier and durability target before choosing a component. The numbers should justify the architecture.
Interactive thought experiment: Health Monitoring
Change the variables below and predict what breaks first in Health Monitoring. The production lab can later reuse these same inputs.
Change one variable at a time. Predict the failure mode first, then move the slider and see whether your mental model matches the simplified system response.
If you are stuck on Health Monitoring, start by drawing the request path and marking every network hop, stateful component, queue, cache and failure boundary. Then estimate where the system will saturate.
You increase traffic by 10× in a system using Health Monitoring. What should you inspect first?
Pick one answer.
Which statement is the safest engineering habit when using Health Monitoring?
Pick one answer.
You have dashboards for traffic, latency, errors and saturation. You can change the architecture, but every change has operational cost.
Production scenario: your system uses Health Monitoring, traffic suddenly spikes, and p99 latency doubles. What is your first move?
Interview drill
Answer this without notes: When would you choose Health Monitoring, and when would you intentionally avoid it? Mention at least one bottleneck it addresses, one failure mode it introduces, and one alternative. Then quantify the workload you are designing for.
A useful engineering lens for Health Monitoring: define the problem it solves, the simpler design that fails first, the constraint that forces you to introduce this concept, and the new failure modes the concept creates.
Numerical sanity check
Back-of-the-envelope reasoning beats fake precision. State your traffic, payload, concurrency and growth assumptions explicitly, then calculate enough to know whether the current architecture is orders of magnitude away from the target.
Do not optimize for a memorized definition. Reason from the workload and failure mode.
Imagine the simplest version of a system using Health Monitoring. What breaks first as traffic grows by 10×, and what would you change before reaching 100×?
Pick one answer.
What you gain, what you pay
- +Self-healing: orchestrator restarts or reroutes automatically.
- +Catches zombie processes and deadlocks that metrics might miss.
- +Drives load balancer decisions without manual intervention.
- +Cheap to implement — one HTTP endpoint per service.
- −Misconfigured probes cause cascading failures (restart storms).
- −Probes that check too many dependencies amplify outages.
- −Adds load to the service (probe traffic).
- −Tautological probes (always return 200) give false confidence.
How this breaks in production
- Liveness probe that checks downstream → restart storm when downstream is down.
- Readiness probe that never fails → broken service stays in rotation.
- Probe with too-short initial delay → CrashLoopBackOff during slow startup.
- Probe that does expensive work → contributes to the load it's monitoring.
Don't fall into these traps
- •Using liveness where readiness is needed (restarts vs reroutes).
- •Making /ready check every dependency — over-coupled.
- •Probes returning 200 unconditionally (tautology).
- •No startup probe for slow-starting apps (JVM, large cache warmups).
Real systems using this
How real systems implement this
- Kubernetes — Three probe types — liveness (restart), readiness (reroute), startup (wait) — defined in the pod spec. Standard pattern for self-healing container orchestration.
- AWS ALB Target Group health checks — Periodically calls a configured path on each target; failed targets are removed from routing automatically. Equivalent of Kubernetes readiness at the load balancer layer.
Practice saying it out loud
- Q1What's the difference between liveness and readiness probes? When do you use each?
- Q2Why is it a mistake to make the liveness probe check the database?
- Q3How do you handle a slow-starting service in Kubernetes?
- Q4What's the relationship between health checks and graceful degradation?
Further reading & references
Core explanations are original NO CAP material. External references are provided for deeper study and standards.
What next?
Mark as understood once the mental model clicks.
Next recommended
Availability Monitoring