Load Balancing

A traffic director that spreads requests across identical backend servers so none is overwhelmed — using a strategy plus health checks.

10 min read Watch it move Build it

A load balancer sits in front of a pool of identical backend servers (upstreams) and spreads incoming requests across them so no single one is overwhelmed. It's the piece that turns *one server* into *a fleet* — the foundation of horizontal scaling. Two things define it: the strategy that picks a server, and the health checks that keep dead servers out of rotation.

The distribution strategies

1Round-robin — hand requests to each server in turn, cycling through the list: 1, 2, 3, 1, 2, 3... Simple and perfectly even when servers and requests are uniform.
2Least-connections — send each request to the server with the fewest active connections right now. Adapts when some requests run long and others are quick, so a slow server stops accumulating work.
3Consistent hashing — hash a key from the request (client IP, session ID, cache key) to pick the server, so the *same* client keeps landing on the *same* server (sticky routing), and adding/removing a server only remaps a small fraction of keys.

Why sticky routing matters

Round-robin assumes servers are stateless. If a server caches per-user data or holds a session in memory, you want the same user to return to the same server — that's where consistent hashing (or an explicit session affinity) earns its place, without the full reshuffle a plain hash % N would cause when the pool changes.

L4 vs L7

Load balancers operate at one of two layers. Layer 4 (transport) balances on TCP/UDP: it sees IP addresses and ports, forwards packets fast, and never inspects the payload — cheap and protocol-agnostic. Layer 7 (application) understands HTTP: it can route on URL path, headers, or cookies (/api/* to one pool, /images/* to another), terminate TLS, and retry failed requests — smarter, but more work per request.

Client --> [ Load Balancer ]
                 |  strategy picks a healthy server
     +-----------+-----------+
     v           v           v
  server1     server2     server3
  (healthy)   (healthy)   (FAILED - pulled from rotation)

L4: routes on IP:port      (fast, opaque)
L7: routes on HTTP path,   (smart: /api -> pool A,
    headers, cookies              /img -> pool B)

Health checks

A load balancer must never send traffic to a dead server. It runs periodic health checks — probing each upstream (GET /healthz returning 200, or a TCP connect) on an interval. A server that fails a few consecutive checks is pulled from rotation; when it passes again it's added back. This is what makes a fleet self-healing: a crashed server simply stops receiving requests, and users never notice.

Passive vs active, and flapping

Active checks probe on a schedule; passive checks watch real traffic for errors/timeouts. Set thresholds carefully — one failed probe shouldn't eject a server (a blip), but waiting too long sends real users to a dead box. Require N consecutive failures to remove and M successes to re-add, so a server doesn't flap in and out.

OperationTimeSpace

Round-robin pick · just advance a counterO(1)O(n)

Least-connections pick · scan for the min (or a heap)O(n)O(n)

Consistent-hash pick · ring lookupO(log n)O(n)

Check yourself

Requests vary wildly in duration — some finish in 5 ms, others take 30 s. Which strategy best avoids piling long requests onto one server?