Module F-11·18 min read

L4 vs L7, round-robin vs least-connections vs hashing, health checks that lie, and connection draining during a deploy.

JJS

Written by Jatin Jain Saraf · Senior Software Engineer

Module F-11 — Load Balancing

Q: A service's `/health` endpoint runs `SELECT 1` against Postgres and pings Redis. During a two-second database hiccup, the service is fully unavailable for several minutes. What happened, and what's the correct design?

Every instance checks the same dependency, so all health checks failed simultaneously and the balancer had no healthy backends left — a two-second blip became a total outage, extended by the staggered re-entry and the queued retries afterwards. Liveness should be shallow, readiness decorrelated with a minimum-healthy floor, and dependency failure should trigger degradation rather than self-removal. — The module traces this six-step sequence and stresses that correlated failure is the mechanism: because every instance depends on the same database, a shared blip fails every check at once and leaves the balancer with nothing to route to. It also notes the aftermath — staggered re-entry plus a stampede of queued requests and retries — which is why the outage outlasts the trigger by minutes, and warns that if the same check is the liveness probe the orchestrator will restart the fleet into a cold-cache state. Option B extends the window before failure without addressing correlation; a longer outage still trips it. Option C misidentifies the culprit: any shared dependency in the check produces the same correlated failure. Option D describes a capability that would defeat the purpose of health checking, and the module's answer is to change what the check asserts, not to ignore it.

Q: A team migrates internal traffic to gRPC and keeps its existing L4 load balancer. Requests distribute very unevenly across backends. Why, and what are the options?

L4 distributes *connections*, but a gRPC client holds one long-lived HTTP/2 connection carrying thousands of requests — so all of a client's traffic lands on whichever backend it first connected to. The fixes are L7 request-level balancing, client-side balancing, or a mesh sidecar, and a maximum connection age to force periodic redistribution. — The module calls this the decisive difference between L4 and L7 and ties it back to Module F-6: connection count only tracks request count under HTTP/1.1, and gRPC's multiplexing breaks that relationship entirely. An L4 balancer will distribute connections perfectly and load terribly. The listed remedies are L7 request-level balancing, client-side balancing, or a sidecar, plus a connection-age limit so long-lived connections eventually redistribute. Option C is wrong on the mechanics — the algorithm is irrelevant when the unit of distribution is a connection that never ends. Option A is false; an L4 balancer forwards encrypted bytes without needing to read them, which is precisely why it's cheap. Option D would matter for a heterogeneous fleet but doesn't explain pinning, which persists regardless of weights.

Q: Every deploy produces a brief spike of 502 errors. The shutdown handler calls `server.close()` on `SIGTERM` and then exits once in-flight requests finish. What's missing?

Readiness must fail *first*, followed by a wait for the balancer to notice, and only then `server.close()`. Closing the listener while the balancer is still routing means requests already sent to this instance are refused — and HTTP/2 or gRPC clients additionally need a `GOAWAY`, since existing connections keep delivering requests. — The module presents the ordering as the whole point: fail readiness, wait for propagation (the balancer polls, so deregistration isn't instantaneous — this is what a Kubernetes `preStop` sleep is for), then stop accepting, then finish in-flight work under a bounded grace period, then release resources. Skipping the first two steps is exactly what produces per-deploy 502s. It also notes that closing the listener doesn't stop requests arriving over existing multiplexed connections, hence `GOAWAY`. Option A normalises a fixable defect, and the module's position is that correct draining makes deploys invisible while incorrect draining costs error budget forever. Option B makes it strictly worse by dropping all in-flight requests immediately. Option D contradicts the explicit guidance to bound the grace period, since an unbounded wait lets one long request block the deploy.

What this module covers: The component every request passes through and almost nobody designs deliberately. This module covers what a load balancer actually does beyond distribution, the real difference between L4 and L7 and what each can and cannot do, the distribution algorithms and why least-connections beats round-robin for tail latency, and then the two subjects that cause real outages: health checks that turn one dependency blip into a total outage, and connection draining that drops in-flight requests on every deploy. Plus the layers of load balancing that stack on top of each other, from DNS down to a sidecar proxy.

A Load Balancer Does Four Jobs

Distribution is the obvious one and the least interesting. The full list:

Distribute requests across healthy backends.
Detect health and stop sending traffic to instances that can't serve it.
Terminate TLS, so backends speak plaintext internally and the expensive handshake happens once, close to the user (Module F-5).
Drain — remove an instance from rotation without dropping the requests it's already handling.

Jobs 2 and 4 are where the incidents live. Job 1 is a configuration choice; jobs 2 and 4 are design decisions with failure modes.

Analogy: a load balancer is the host at a restaurant. Seating guests evenly is the easy part. The hard parts are noticing that one section's waiter has walked out (health checks) and closing a section without ejecting the people mid-meal (draining). A host who mishandles either of those empties the restaurant.

L4 vs L7: What Each Can See

L4 (transport layer) balancers forward TCP connections. They see IP addresses and ports and nothing else — the payload is opaque, which is the point: it's cheap, extremely fast, and works for any TCP protocol.

L7 (application layer) balancers terminate the connection, parse HTTP, and make decisions per request. They see method, path, headers, and cookies.

	L4	L7
Unit of distribution	Connection	Request
Can route by path/header	No	Yes
Can retry a failed request	No (it doesn't know what a request is)	Yes
Can rewrite, compress, add headers	No	Yes
Sees TLS content	No (unless terminating)	Yes (terminates)
Overhead	Very low	Higher (parsing, buffering, re-encryption)
Works with	Any TCP protocol	HTTP, gRPC, WebSocket upgrades

The decisive point, and it follows straight from Module F-6: with HTTP/2 or gRPC, one client holds one long-lived connection carrying thousands of requests. An L4 balancer distributing connections therefore pins all of that traffic to one backend. So:

HTTP/1.1 traffic: L4 is often fine; connection count roughly tracks request count.
HTTP/2, gRPC, or any long-lived-connection protocol: you need L7 request-level balancing, or client-side balancing, or a mesh sidecar (Module A-5). L4 will distribute connections beautifully and load terribly.

There's also a hybrid worth knowing: an L4 balancer can route by TLS SNI without decrypting, which gives you per-hostname routing while keeping end-to-end encryption. Useful when compliance forbids terminating TLS at the edge.

Algorithms, and What They're Actually Optimising

Round-robin — next backend in sequence. Simple, and correct only when every request costs the same and every backend is identical. Neither is usually true.

Weighted round-robin — proportional to declared capacity. The standard answer for a mixed fleet during a migration between instance types.

Least connections — send to the backend with the fewest in-flight requests. This is the better default for most HTTP services, and the reason is Module F-4: request costs vary enormously (the 900-line order next to the 2-line one). Round-robin keeps feeding an instance that's already stuck behind expensive work; least-connections naturally routes around it, because the stuck instance's in-flight count stays high. It's a feedback signal rather than a fixed rotation.

Least response time — least connections weighted by observed latency. Better still in theory; more sensitive to noisy measurements, and it can amplify oscillation if the measurement window is short.

Hash-based (consistent hashing) — route by a hash of client IP, a cookie, or a URL. Two legitimate uses:

Session affinity without server-side session storage — with all four of Module F-9's costs.
Cache locality: hashing on the request path means the same URL always reaches the same backend, so its in-process cache stays warm. This is genuinely valuable for cache-heavy services, and consistent hashing (Module P-7) is what keeps it from reshuffling every key when the backend count changes.

Power of two random choices — pick two backends at random, send to the less loaded. Deserves to be better known: it gets most of the benefit of least-connections while needing almost no global state, which matters when the balancing decision is made by many independent clients or sidecars that can't see each other's counts.

Health Checks: How to Cause a Total Outage

This is the most valuable section in the module, because the failure is severe, common, and produced by well-intentioned engineering.

Two distinct questions, often conflated:

Liveness: is this process alive and should it be restarted if not?
Readiness: can this instance serve traffic right now?

The tempting mistake is a "thorough" health check:

typescript

// ⚠️ This design converts a database blip into a full outage.
app.get('/health', async (req, res) => {
  await db.query('SELECT 1');           // is the database up?
  await redis.ping();                   // is the cache up?
  await paymentProvider.status();        // is the payment vendor up?
  res.send('ok');
});

It looks responsible. Now trace what happens when the database has a two-second hiccup:

text

1. Database briefly stops responding.
2. Every instance's health check fails — simultaneously, because they all
   check the same database.
3. The load balancer marks every backend unhealthy.
4. With no healthy backends, the balancer returns 503 for all traffic.
5. The database recovers after two seconds.
6. Health checks pass again, but instances re-enter rotation over the next
   30+ seconds, and now face a stampede of queued requests and retries.

A two-second dependency blip became a minutes-long total outage, and the health check caused it. Worse, if the check is also the liveness probe, the orchestrator starts killing and restarting healthy processes — turning a transient issue into a full fleet cold start with empty caches and cold connection pools.

The rules that avoid this:

Liveness checks must be shallow. Is the process running and the event loop responsive? Nothing else. A liveness check that touches a dependency will eventually restart your entire fleet over someone else's outage.
Readiness may check dependencies the instance cannot work without — carefully. If the database is genuinely required, briefly failing readiness is defensible; but never let all instances fail simultaneously on a shared dependency. Correlated failure is the mechanism, so decorrelate it: stagger check intervals, require multiple consecutive failures, and prefer a "minimum healthy" floor in the balancer that keeps some backends in rotation regardless.
Never health-check a third-party dependency. If the payment provider is down, you should degrade — disable checkout, show a message, queue the intent (Module A-7) — not remove yourself from the internet.
Readiness should reflect capacity, not just correctness. An instance with a saturated queue can honestly say "not ready" and let the balancer route elsewhere. That's cooperative load shedding (Module A-8), and it's the most under-used feature of readiness probes.

Why this matters in production: the health check is the one piece of code with the authority to remove your entire service from the internet. It deserves more design attention than it usually gets, and less thoroughness.

Draining: The Requests You Drop on Every Deploy

When an instance is removed — a deploy, a scale-down, a node replacement — it's already handling requests. Naive removal drops them, and you'll see it as a small spike of 502s on every single deploy that everybody has learned to ignore.

Correct graceful shutdown:

typescript

// The ordering here is the whole point.
process.on('SIGTERM', async () => {
  // 1. Fail readiness FIRST, so the balancer stops sending new requests.
  //    Then wait long enough for it to notice — the balancer polls, and
  //    deregistration is not instantaneous.
  isShuttingDown = true;
  await sleep(READINESS_PROPAGATION_MS);   // typically 5–15s

  // 2. Stop accepting new connections, let in-flight requests finish.
  server.close();
  await Promise.race([
    inFlightRequestsDrained(),
    sleep(GRACE_PERIOD_MS),                // bounded — don't hang forever
  ]);

  // 3. Now release resources: pools, consumers, subscriptions.
  await Promise.all([db.end(), redis.quit(), queueConsumer.stop()]);
  process.exit(0);
});

app.get('/ready', (req, res) =>
  isShuttingDown ? res.status(503).send('draining') : res.send('ok')
);

Three details that are the difference between working and nearly working:

The order is fail-readiness, wait, then stop accepting. Closing the listener first means requests already routed to you get refused. The sleep is not superstition — it covers the interval between your readiness turning red and the balancer acting on it. In Kubernetes this is what a preStop hook with a sleep is for.
The grace period must be bounded. An instance waiting indefinitely for a long-lived request blocks the deploy. Bound it, and accept that requests exceeding the bound get cut.
Long-lived connections need protocol-level cooperation. For HTTP/2 and gRPC, send GOAWAY so clients establish new connections elsewhere; "stop accepting new connections" doesn't stop requests arriving on existing ones (Module F-6). For WebSockets, you have to close them and rely on client reconnection (Module A-10).

Get this right and deploys are invisible. Get it wrong and every deploy has a small error budget cost, forever (Module O-2).

Load Balancing Happens at Several Layers

A single request can be balanced repeatedly:

text

DNS (GeoDNS / weighted records)     → picks a region
  Anycast / global load balancer    → picks an edge PoP
    Regional L7 load balancer       → picks a service
      Service mesh sidecar or       → picks an instance
      client-side balancer

Each layer has different granularity and different reaction time:

DNS-based balancing is coarse and slow to change. TTLs aren't reliably honoured, so removing a region by DNS leaves a tail of clients still going there for minutes (Module F-5). Adequate for region selection, unsuitable for failover.
Anycast advertises one IP from many locations and lets network routing pick the closest. Fast and automatic — and it can shift traffic mid-connection during a routing change, which TCP handles poorly and QUIC handles better.
Regional L7 is where per-request policy lives: routing, retries, rate limits, header manipulation.
Client-side or sidecar balancing is what gRPC and service meshes use to solve the HTTP/2 pinning problem — the client picks the instance, so there's no single balancer holding one connection per client (Module A-5).

The practical warning: balancing at many layers makes traffic hard to reason about. When one instance is hot, the cause may be at any layer, and the layers can fight — a sticky cookie at L7 defeating an even distribution decided at the edge. Know which layer owns the decision.

And a note on the balancer itself: it is a component with limits — connection ceilings, throughput ceilings, and its own cold-start behaviour. Managed balancers often warm up rather than instantly absorbing a spike, which is exactly why load tests need a ramp rather than an instantaneous step (Module O-5).

Where This Shows Up in Your Stack

Kubernetes: a Service is L4 (kube-proxy, connection-level); an Ingress or Gateway controller is L7. This is why gRPC inside Kubernetes distributes badly through a plain Service and needs a mesh or headless service with client-side balancing. Readiness and liveness probes map directly onto the two health-check questions above.
Node.js: liveness must verify the event loop isn't blocked, not just that the process exists — a process stuck in a synchronous loop still accepts the TCP connection and answers nothing. Checking event-loop lag is the meaningful signal.
PostgreSQL: connections are balanced differently. A pooler (pgBouncer) is closer to a connection multiplexer than a load balancer, and read-replica routing is usually done in the application or by a proxy that understands read vs write statements (Modules P-4, P-5).
Serverless: the platform is the load balancer, and you don't configure the algorithm. What you do control is concurrency limits per function, which is the only lever you have to stop one endpoint's spike from starving another's capacity — a bulkhead in Module A-7's terms.

Summary

A balancer does four jobs: distribute, health-check, terminate TLS, drain. The last two cause the outages.
L4 distributes connections; L7 distributes requests. With HTTP/2 or gRPC, one connection carries thousands of requests, so L4 pins load to one backend — use L7, client-side, or sidecar balancing.
Least connections is a better default than round-robin because request costs vary; it routes around a backend that's stuck. Power-of-two-choices gets most of the benefit without global state.
Hash-based routing buys session affinity or cache locality — with consistent hashing to avoid reshuffling when the fleet changes.
A deep health check can cause a total outage: every instance checks the same dependency, all fail together, and no backends remain. Keep liveness shallow, decorrelate readiness, never health-check a third party, and degrade instead of self-removing.
Readiness should reflect capacity, which makes it a cooperative load-shedding signal.
Draining order matters: fail readiness, wait for propagation, stop accepting, finish in-flight with a bounded grace period, then release resources. Long-lived connections need GOAWAY or explicit closure.
Balancing stacks across DNS, anycast, regional L7, and sidecars — different granularity, different reaction times, and they can fight each other.

The next module, Caching Fundamentals, moves to the layer that removes work rather than distributing it — the cache hierarchy from browser to CDN to application to database, and why hit ratio is the only number that matters.

Knowledge Check

A service's /health endpoint runs SELECT 1 against Postgres and pings Redis. During a two-second database hiccup, the service is fully unavailable for several minutes. What happened, and what's the correct design?

A team migrates internal traffic to gRPC and keeps its existing L4 load balancer. Requests distribute very unevenly across backends. Why, and what are the options?

Every deploy produces a brief spike of 502 errors. The shutdown handler calls server.close() on SIGTERM and then exits once in-flight requests finish. What's missing?

Test your knowledge with more question sets

Sign in to keep reading

The rest of this module is free — sign in with Google to unlock it and track your progress.

PreviousModule F-10: Compute Models: VMs, Containers, Serverless, and the Edge Next Module F-12: Caching Fundamentals

Discussion

Join the discussion

Loading comments...