Load Balancers
A load balancer distributes incoming traffic across multiple servers. It is the foundational scaling primitive: it enables horizontal scaling, fault tolerance, and rolling deploys. Without it, you have one server and a single point of failure.
How it works
A load balancer sits in front of a pool of backend servers. Incoming requests arrive at the load balancer; it picks a backend and forwards the request; the backend responds; the load balancer relays the response to the client. To the client, the load balancer is the server. To the backend, the load balancer is the client.
The choice of which backend to pick is the algorithm. The choice of how to detect failures is the health check. These two decisions, plus whether to operate at L4 (transport) or L7 (application), define the load balancer.
The problem:
You have one application server. At 100 requests/sec, it's fine — CPU at 30%, memory stable, 50ms latency. Life is good.
Then traffic grows. At 1,000 requests/sec:
- CPU hits 90%.
- Memory pressure causes GC pauses.
- Latency climbs to 200ms.
- Users start seeing timeouts.
At 10,000 requests/sec, the server is dead. OOM crash, connection exhaustion, or the OS just stops responding.
The naive fix: buy a bigger server. Double the CPU, double the RAM. But that only gets you 2x — you're still one machine, and now it's expensive. And it's still a single point of failure: if that one machine dies, your entire service is down.
The right fix: add more servers and a load balancer to distribute traffic across them.
Load balancing algorithms:
| Algorithm | How it works | Best for |
|---|---|---|
| Round Robin | Cycle through backends in order. Simple, ignores load. | Even backend capacity, simple setups |
| Least Connections | Pick the backend with fewest active requests. | Uneven request durations (some slow, some fast) |
| IP Hash | Hash client IP → same client always goes to same backend. | Sticky sessions without cookies |
| Weighted | Give stronger backends more traffic (weight). | Mixed-capacity backends (e.g., 4-core vs 8-core) |
| Random | Pick a random backend. Cheap, surprisingly effective at scale. | Very large backend pools |
| Least Response Time | Pick the backend with the fastest average response. | Latency-sensitive apps |
Production load balancers often combine: least-connections + health checks + slow-start for newly-added backends. NGINX and HAProxy default to round-robin; AWS ALB defaults to round-robin; Envoy defaults to least-connections.
For horizontal scaling to work, backends must be stateless — no in-memory sessions, no local file uploads, no per-server caches. All shared state lives in the database, Redis, or object storage. If a backend holds session state, you need sticky sessions (IP hash), which defeats the LB's ability to fail over. Move sessions to Redis. Move file uploads to S3. Then any backend can serve any request.
L4 vs L7 load balancing:
L4 (transport layer): The load balancer sees TCP/UDP packets. It forwards bytes without parsing HTTP. Fast, simple, opaque.
- Examples: HAProxy (L4 mode), AWS NLB, iptables.
- Pros: very fast, low overhead, protocol-agnostic (works for any TCP service).
- Cons: can't route by URL/headers, can't modify requests, no TLS termination.
- Use cases: database connection pooling, gRPC, raw TCP services.
L7 (application layer): The load balancer parses HTTP. It can route by URL path, headers, cookies. It can terminate TLS, modify requests, add headers.
- Examples: NGINX, AWS ALB, Envoy, HAProxy (L7 mode).
- Pros: smart routing (path-based, header-based), TLS termination, content-based decisions.
- Cons: slower (must parse HTTP), more overhead, HTTP-only.
- Use cases: web APIs, microservices, path-based routing.
Most modern systems use both: an L4 LB at the edge for raw throughput, L7 LBs behind it for smart routing.
Health checks:
A load balancer must know which backends are alive. It does this via health checks — periodic requests to each backend to verify it's responding.
- Active health check: the LB sends a request (e.g.,
GET /healthevery 5 seconds). If the backend responds 200, it's healthy. If it responds 5xx or times out 3 times in a row, the LB marks it unhealthy and stops sending traffic. - Passive health check: the LB monitors real traffic. If a backend starts returning 5xx errors or timing out, the LB marks it unhealthy without sending a separate health check.
Without health checks, a dead backend keeps receiving traffic until the LB notices. With health checks, the LB detects failure in seconds and routes around it.
Health endpoint best practices:
GET /healthshould return 200 with no side effects.GET /health/readyshould check dependencies (DB, cache) and return 503 if any are down.- Health checks should be cheap (no DB queries — just a 'I'm alive' signal).
- Health checks should have a short timeout (1-2s).
Your web app stores user sessions in memory on each server. After adding a load balancer with round-robin, users complain they get logged out randomly. What is the root cause?
Pick one answer.
When would you choose L4 over L7 load balancing?
Pick one answer.
Your load balancer has 3 backends. One backend starts returning 500 errors. What should happen?
Pick one answer.
| Dimension | Round Robin | Least Connections |
|---|---|---|
| Decision basis | Cyclic position in pool | Current active connection count per backend |
| Best when | All backends have equal capacity and similar request durations | Request durations vary widely (some slow, some fast) |
| Failure mode | Slow backend accumulates queued requests because RR keeps sending to it | Avoids overloading slow backends; new requests go to the least-loaded |
| Cost per decision | O(1) — increment a counter | O(N) — must compare all backends (or maintain a heap) |
| State required | Last index (1 integer) | Per-backend connection count |
| Sticky behavior | Deterministic given same client order | Adapts to live load |
| Default in | NGINX, AWS ALB, HAProxy | Envoy, HAProxy (option), Linkerd |
| Example fail | A 10s request hits backend A; RR sends 4 more to A in the meantime; A is overloaded while B sits idle. | LC sees A has 5 active connections; sends the next request to B (0 active). |
Real example: AWS ALB vs NLB — when to pick which.
AWS offers two managed load balancers that look similar but solve very different problems.
Application Load Balancer (ALB) operates at L7. It terminates TLS, parses HTTP, and routes by URL path, host header, or HTTP header. It supports weighted target groups (canary deploys), WebSocket and HTTP/2 natively, and integrates with ECS/Kubernetes via IP-target mode. ALB is the right choice for any HTTP-based API or web app — it can route /api/v1/* to one service and /api/v2/* to another on the same listener. It does NOT support non-HTTP protocols. Pricing is per-LCU (Load Balancer Capacity Unit, a blend of new connections, active connections, bandwidth, and rule evaluations).
Network Load Balancer (NLB) operates at L4. It sees TCP/UDP packets, forwards them to targets by 4-tuple, and terminates nothing (TLS passthrough is the default). NLB is the right choice for: TCP-based protocols (database connections, SMTP, custom binary protocols), UDP (DNS, gaming, QUIC), extreme throughput (millions of connections/sec, zonal-static anycast IPs that survive AZ failures), and ultra-low-latency requirements where every microsecond of L7 parsing matters. Pricing is per NLB-hour + per-GB processed.
A common production pattern uses both: a single NLB at the edge terminates TLS for the apex domain; ALBs behind it do path-based routing for each microservice. The NLB gives you a static, anycast IP that doesn't change across AZ failures; the ALBs give you per-service path routing and ECS integration. Costs are higher than a single ALB, but you get both the L4 throughput ceiling and the L7 flexibility.
A single LB in front of 50 stateless app servers is just 1 server with 50 backends — and that 1 server can die. AWS mitigates this with managed HA across AZs (ALB/NLB automatically), but self-hosted NGINX/HAProxy does NOT — you must run an active-active pair with VRRP/Keepalived or front them with an L4 anycast. The classic failure: a team runs HAProxy in Docker on one EC2 instance 'just for now', it dies in a maintenance event, the whole site is down even though every app server is healthy. Design the LB's redundancy before you need it.
Your image-processing API has 6 backend servers. Each request downloads an image, generates 5 thumbnails, and uploads them to S3 — request durations range from 200ms (small image) to 30 seconds (huge image). Users report intermittent timeouts. The load balancer uses round-robin. What is the strongest fix?
Pick one answer.
Real failure: the sticky-session trap that took down a major e-commerce site.
A pattern seen repeatedly in incident postmortems: a team runs N app servers behind a load balancer, holds user sessions in memory on each server, and uses sticky sessions to route each user to the server holding their session.
What kills them: when one of the N servers dies (hardware, deploy, OOM), every user whose session lived on that server is suddenly logged out and loses in-flight cart contents. If 1 of 5 servers dies, 20% of users are simultaneously affected. The support team gets a flood of complaints, the engineering team rolls back the deploy, and the cause is misdiagnosed as 'the deploy' rather than the architecture.
The fix that prevents recurrence: externalize sessions to Redis. Now sessions survive any single server death, the load balancer can use round-robin or least-connections, deploys are zero-downtime, and the team can scale by adding servers without worrying about session affinity.
The deeper lesson: sticky sessions are not a scaling strategy — they're an admission that the architecture is not actually horizontally scaled. True horizontal scaling requires statelessness, and statelessness requires externalized state. Anything else is a single-point-of-failure dressed up as redundancy.
Engineering mental model
Mental model. Think of Load Balancers as a deliberate boundary in a system. The boundary exists because something becomes harder to manage when everything is done in one place: latency, scale, failure isolation, consistency, cost, or team ownership. The useful question is not “what does Load Balancers mean?” but “what pressure makes this boundary worth introducing, and what new failure mode does it create?”
Before choosing Load Balancers, name the workload, the critical user path, the dominant bottleneck, the failure you are trying to absorb, and the trade-off you are willing to accept. If you cannot name those five things, the design is probably premature.
// Pseudocode
request = receive()
result = load_balancers(request)
return result
// Production questions:
// 1. What happens on timeout?
// 2. Can this operation be retried safely?
// 3. What is the bottleneck?Back-of-the-envelope reasoning
Numerical lens: write down traffic, payload size, read/write ratio, peak multiplier and durability target before choosing a component. The numbers should justify the architecture.
Interactive thought experiment: Load Balancers
Change the variables below and predict what breaks first in Load Balancers. The production lab can later reuse these same inputs.
Change one variable at a time. Predict the failure mode first, then move the slider and see whether your mental model matches the simplified system response.
If you are stuck on Load Balancers, start by drawing the request path and marking every network hop, stateful component, queue, cache and failure boundary. Then estimate where the system will saturate.
You increase traffic by 10× in a system using Load Balancers. What should you inspect first?
Pick one answer.
Which statement is the safest engineering habit when using Load Balancers?
Pick one answer.
You have dashboards for traffic, latency, errors and saturation. You can change the architecture, but every change has operational cost.
Production scenario: your system uses Load Balancers, traffic suddenly spikes, and p99 latency doubles. What is your first move?
Interview drill
Answer this without notes: When would you choose Load Balancers, and when would you intentionally avoid it? Mention at least one bottleneck it addresses, one failure mode it introduces, and one alternative. Then quantify the workload you are designing for.
For Load Balancers, treat the system as a control loop: observe load and failure, choose a bounded response, and measure whether the response stabilizes the system instead of simply moving the bottleneck somewhere else.
Numerical sanity check
When estimating capacity, distinguish average from peak. If average traffic is 4,000 RPS and the observed peak-to-average factor is 3×, design the first pass around roughly 12,000 RPS, then leave headroom for failure and growth.
Do not optimize for a memorized definition. Reason from the workload and failure mode.
What is the earliest signal that Load Balancers is becoming the bottleneck: latency, saturation, errors, queue depth, or something else? Why?
Pick one answer.
What you gain, what you pay
- +Horizontal scaling — add more servers to handle more traffic.
- +Fault tolerance — lose a server, traffic shifts to the others.
- +Rolling deploys — drain one server, update it, repeat. Zero downtime.
- +Health checking — bad backends are removed automatically.
- −The LB itself can be a single point of failure (mitigate with active-active pairs).
- −Adds a network hop and a small amount of latency (~1ms).
- −Requires stateless backends — moving session state to shared storage is real work.
- −Cost: LB software/hardware, plus the compute to run it.
How this breaks in production
- The LB itself fails — single point of failure. Mitigated by active-active LB pairs (both receive traffic, either can handle it all).
- Sticky sessions as a default — they defeat failover and complicate scaling.
- Forgetting health checks — a dead backend keeps receiving traffic until the LB notices.
- Single LB without redundancy — the LB is now your SPOF.
- Mixed L4 and L7 without thinking — L7 in front of L4 in front of L7 is over-engineering.
Don't fall into these traps
- •Sticky sessions as a default. They defeat failover and complicate scaling. Move sessions to Redis instead.
- •Forgetting health checks. A dead backend will keep receiving traffic until the LB notices.
- •Single LB without redundancy. The LB is now your SPOF.
- •Mixing L4 and L7 without thinking. L7 in front of L4 in front of L7 is a common over-engineering smell.
Real systems using this
How real systems implement this
- NGINX — The most popular open-source L7 LB. Used as a reverse proxy, load balancer, and static file server. Powers ~30% of the web.
- AWS ALB — Managed L7 load balancer with path-based routing, TLS termination, target group health checks, and integration with ECS/EKS. Pay per LCUs (load balancer capacity units).
Practice saying it out loud
- Q1Design a load balancer for a globally-distributed API with 1M requests per second.
- Q2How do you ensure your load balancer is not a single point of failure?
- Q3When would you use sticky sessions, and what do they cost you?
- Q4Explain the difference between L4 and L7 load balancing with examples.
- Q5How do health checks work? What's the difference between active and passive?
Further reading & references
Core explanations are original NO CAP material. External references are provided for deeper study and standards.
What next?
Mark as understood once the mental model clicks.
Next recommended
Horizontal Scaling