Service Discovery
Service discovery is the mechanism by which services locate each other in a dynamic environment where instances come and go. It replaces hard-coded IP addresses and DNS with a registry that tracks live instances and answers 'where is the auth service right now?'. Two flavors: DNS-based (simple, slower to propagate) and registry-based (Consul, etcd, ZooKeeper, Kubernetes API — fast, watchable, often paired with client-side or server-side load balancing).
How it works
The problem:
In a monolith, calling another module is a function call — auth.check(token). The address is fixed: it is in the same process.
In microservices, calling another service is a network request — but to where? The auth service runs 6 instances behind an autoscaler. This morning there were 4. At 2 PM traffic spiked and 4 more came up. At 2:05 PM one was killed by an OOM. The IP you configured last week is wrong.
You need a way for service A to ask 'where can I reach the auth service right now?' and get a current, healthy answer. That is service discovery.
Three components define any service discovery system:
- Registration — instances announce 'I am auth instance #5, reachable at 10.0.3.42:8080'.
- Lookup — callers query 'give me a healthy auth instance'.
- Health — the registry removes instances that fail health checks, so callers never get a dead address.
Two styles of service discovery:
1. DNS-based (the simpler style)
Each service gets a DNS name: auth.svc.cluster.local. The DNS server (often CoreDNS in Kubernetes, or a cloud provider's private DNS) resolves it to one or more instance IPs.
- Pros: zero code changes. Any HTTP client that already does DNS lookups works.
- Cons: DNS caching means callers may keep using a stale IP for the TTL window (often 30s+). DNS does not expose health — it returns all instances, healthy or not. Load balancing is implicit (round-robin in DNS, or relies on a separate load balancer).
- Use case: Kubernetes Services (ClusterIP + kube-dns), AWS Cloud Map in DNS mode, simple internal services.
2. Registry-based (the dynamic style)
A dedicated registry (Consul, etcd, ZooKeeper, or the Kubernetes API server) holds the live list of instances. Callers either query the registry on each call or watch it for changes and keep a local, always-current copy.
- Pros: sub-second propagation, exposes health, supports client-side load balancing.
- Cons: caller must integrate with the registry SDK or sidecar. More moving parts.
- Use case: large microservice deployments, gRPC service meshes, anything that needs fast failover.
Modern platforms blend both: Kubernetes gives each service a DNS name (auth.svc.cluster.local) that is backed by the API server's live list of endpoints. Callers can use either the DNS (simple, cached) or watch the endpoints directly (fast, accurate).
Client-side vs server-side load balancing:
Once a caller has a list of healthy instances, someone has to pick one. Two patterns:
- Server-side LB: a load balancer sits between caller and instances. The caller asks the LB; the LB picks an instance. Caller code is simple. Adds a network hop and a stateful middleman. Examples: AWS ALB, classic NGINX in front of services.
- Client-side LB: the caller fetches the instance list from the registry and picks one itself (round-robin, least-connections, etc.). No middleman, no extra hop. Caller code is more complex — but libraries (gRPC's built-in LB, Ribbon, Envoy sidecar) hide it. Examples: gRPC client load balancing, Istio/Envoy sidecars.
Service meshes (Istio, Linkerd) make this elegant: a sidecar proxy runs next to every service. The sidecar handles discovery, health, load balancing, retries, and mTLS — the application code sees a simple localhost call to the sidecar. The mesh centralizes what would otherwise be scattered client library code.
Most production registries run as a small cluster (3 or 5 nodes) and use a consensus protocol (Raft for Consul and etcd, Zab for ZooKeeper) to replicate data. Consensus means a quorum (N/2+1) must agree before a write is committed. This makes the registry strongly consistent and survive node loss, at the cost of: (a) one extra network hop for the consensus round, and (b) the registry being unavailable if quorum is lost. Three nodes tolerate one failure; five nodes tolerate two. Never run an even number of consensus nodes — you cannot form a quorum.
Your service calls `auth.svc.cluster.local` (DNS-based discovery). After an auth instance crashes, callers still send traffic to its old IP for 30 seconds. Why?
Pick one answer.
You are deploying a Consul cluster as a service registry. How many nodes should you run, and why?
Pick one answer.
In a service mesh like Istio, where does service discovery happen?
Pick one answer.
Engineering mental model
Mental model. Think of Service Discovery as a deliberate boundary in a system. The boundary exists because something becomes harder to manage when everything is done in one place: latency, scale, failure isolation, consistency, cost, or team ownership. The useful question is not “what does Service Discovery mean?” but “what pressure makes this boundary worth introducing, and what new failure mode does it create?”
Before choosing Service Discovery, name the workload, the critical user path, the dominant bottleneck, the failure you are trying to absorb, and the trade-off you are willing to accept. If you cannot name those five things, the design is probably premature.
// Pseudocode
request = receive()
result = service_discovery(request)
return result
// Production questions:
// 1. What happens on timeout?
// 2. Can this operation be retried safely?
// 3. What is the bottleneck?Back-of-the-envelope reasoning
Numerical lens: write down traffic, payload size, read/write ratio, peak multiplier and durability target before choosing a component. The numbers should justify the architecture.
Interactive thought experiment: Service Discovery
Change the variables below and predict what breaks first in Service Discovery. The production lab can later reuse these same inputs.
Change one variable at a time. Predict the failure mode first, then move the slider and see whether your mental model matches the simplified system response.
If you are stuck on Service Discovery, start by drawing the request path and marking every network hop, stateful component, queue, cache and failure boundary. Then estimate where the system will saturate.
You increase traffic by 10× in a system using Service Discovery. What should you inspect first?
Pick one answer.
Which statement is the safest engineering habit when using Service Discovery?
Pick one answer.
You have dashboards for traffic, latency, errors and saturation. You can change the architecture, but every change has operational cost.
Production scenario: your system uses Service Discovery, traffic suddenly spikes, and p99 latency doubles. What is your first move?
Interview drill
Answer this without notes: When would you choose Service Discovery, and when would you intentionally avoid it? Mention at least one bottleneck it addresses, one failure mode it introduces, and one alternative. Then quantify the workload you are designing for.
A useful engineering lens for Service Discovery: define the problem it solves, the simpler design that fails first, the constraint that forces you to introduce this concept, and the new failure modes the concept creates.
Numerical sanity check
Back-of-the-envelope reasoning beats fake precision. State your traffic, payload, concurrency and growth assumptions explicitly, then calculate enough to know whether the current architecture is orders of magnitude away from the target.
Do not optimize for a memorized definition. Reason from the workload and failure mode.
Imagine the simplest version of a system using Service Discovery. What breaks first as traffic grows by 10×, and what would you change before reaching 100×?
Pick one answer.
What you gain, what you pay
- +Services find each other automatically as instances come and go.
- +Decouples callers from the deployment topology of their dependencies.
- +Enables autoscaling, rolling deploys, and failover without reconfiguring callers.
- +Health-aware routing — dead instances are removed from the registry.
- −Adds a moving part (the registry) that itself must be highly available.
- −DNS-based discovery has TTL propagation delay — fast failover needs registry-based or client-side LB.
- −Registry-based discovery requires client SDKs or sidecars — operational complexity.
- −A registry outage can take down all inter-service communication (mitigate with local caches and fallback).
How this breaks in production
- Stale DNS cache causing callers to keep using a dead instance until TTL expires.
- Registry outage — all services lose the ability to find each other (mitigate with client-side caching of last-known-good lists).
- Split-brain in the registry cluster (rare with Raft, but misconfiguration can cause it) returning inconsistent instance lists.
- Health-check false positives marking healthy instances down, or false negatives keeping dead instances in the list.
- Runaway registration (instances that never deregister — 'ghosts') polluting the registry.
Don't fall into these traps
- •Using DNS-based discovery with a long TTL when fast failover is required.
- •Running an even number of consensus nodes (2 or 4) — destroys fault tolerance.
- •Forgetting to deregister on shutdown, leaving 'ghost' instances in the registry.
- •Hard-coding IPs 'just for now' — they never get replaced and cause an outage later.
- •Skipping health checks — the registry returns dead instances, callers fail, and no one knows why.
- •Trusting the registry blindly without client-side retry or fallback when the registry itself is down.
Real systems using this
How real systems implement this
- Kubernetes — Each Service gets a stable virtual IP and DNS name (`auth.svc.cluster.local`). The kube-apiserver maintains a live Endpoints list of healthy pod IPs behind it. CoreDNS resolves the name to the virtual IP, and kube-proxy (or a CNI) load-balances across the endpoints. Callers can also watch the Endpoints object directly for client-side LB.
- HashiCorp Consul — Agents run on every node, register services, and run health checks. A Consul server cluster (Raft, typically 3 or 5 nodes) stores the registry. Clients can query via HTTP API or DNS. Used by HashiCorp Nomad and many non-Kubernetes microservice platforms.
- Apache ZooKeeper — A hierarchical key-value store with Zab consensus, used by Kafka (for broker and topic metadata), HDFS NameNode HA, and many older service-discovery systems. Older than Consul/etcd but battle-tested at Yahoo, Twitter, and LinkedIn scale.
Practice saying it out loud
- Q1Design service discovery for a microservices platform. Compare DNS-based and registry-based approaches.
- Q2A caller keeps using a dead instance for 30 seconds after a crash. What is happening and how do you fix it?
- Q3Why must a Consul/etcd/ZooKeeper cluster have an odd number of nodes?
- Q4How does a service mesh like Istio change the service discovery story for application developers?
- Q5What happens when the service registry itself goes down? How do you keep services talking to each other?
Further reading & references
Core explanations are original NO CAP material. External references are provided for deeper study and standards.
What next?
Mark as understood once the mental model clicks.
Next recommended
Load Balancers