What this module covers: Why a list of addresses in a config file is stale before it deploys, and the five answers in the order they were invented — static config, DNS, client-side discovery, server-side discovery, and the sidecar. The DNS trap that catches everyone: a connection pool holding keep-alive sockets never re-resolves, so a TTL you tuned carefully changes nothing. Self-registration versus platform registration, and the stale entry left by a process that was killed. Health checks in depth, including the readiness probe that turns a database blip into a total outage, and connection draining without which every deploy returns 502s. Then the service mesh: what the data plane genuinely gives you — mTLS with rotating identities, uniform retries and breakers, traffic shifting, and per-hop telemetry — and a frank accounting of what it costs in latency, memory, operational surface and specialist skill, because that bill is routinely understated. Finally, the ladder of cheaper options and the conditions under which each rung is the right one.
Addresses Are Not Stable, So a List of Them Is Always Wrong
In fixed infrastructure, payments.internal resolved to 10.0.4.12 and stayed there for years. A config file was a reasonable answer.
Then instances became ephemeral. Autoscaling adds and removes them by the minute; a rolling deploy replaces every one of them; a spot instance is reclaimed with two minutes' notice (Module O-8); a container is rescheduled onto a different node with a different IP. So the question a caller needs answered is no longer "what is the address of payments" but:
Which instances of payments exist right now, which of them are healthy, and which should I send this request to?
That is service discovery, and it has three parts that are worth keeping separate because different mechanisms solve them to different standards: a registry of what exists, health information about what is usable, and a selection policy for choosing among the usable ones (Module F-11's load-balancing algorithms).
Analogy: a hospital switchboard. A printed directory of extensions is fine while consultants have permanent offices. Once they rotate between wards each shift, the directory is wrong by Tuesday — so you need a live board showing who is on duty and not currently in surgery, plus a rule for which of three available registrars to page. Notice that "on duty" and "free to take a call" are different facts, and conflating them is how you page a surgeon mid-operation or, worse, mark the entire department unavailable because the ward telephone is broken.
Five Mechanisms, in the Order They Were Invented
1. Static configuration
Addresses in a config file or environment variable. Correct for fixed infrastructure and for a small number of stable endpoints. It fails the moment instances are created and destroyed automatically, and it fails silently — the config still parses, requests just go nowhere.
2. DNS
Universal, requires no library, and works from any language. Also full of traps, and they are worth enumerating because DNS-based discovery is where most teams start.
TTL is a hint, not a contract. Many clients cache more aggressively than the TTL suggests. The JVM historically cached resolutions for the process lifetime by default; various libraries and OS resolvers cache independently. A 5-second TTL you set carefully may be observed by nobody.
No health awareness beyond removal. DNS tells you an address exists; whether the process behind it is answering is a separate question, and removing a record propagates only as fast as the caches allow.
Round-robin only. No least-connections, no zone awareness, no outlier ejection. You get whatever order the resolver returns.
No ports.SRV records carry them and support is patchy enough that most systems do not rely on it.
And the trap that catches nearly everyone:
A connection pool holding keep-alive sockets does not re-resolve DNS.
Your HTTP client resolves payments.internal once, opens ten connections, and reuses them for hours because keep-alive is doing exactly what it is supposed to (Module F-6). The instance behind those sockets is terminated, or the load balancer's IPs rotate, and your client keeps trying the address it resolved at start-up — while dig from the same host returns the correct answer, which is what makes this so confusing to diagnose. The mitigations are a maximum connection lifetime (recycle sockets every 60 seconds regardless of health), a maximum requests-per-connection, or a client that re-resolves on a schedule. This is also why AWS documents that clients of an ALB must respect DNS TTLs: the ALB's own addresses change, and a long-lived pool will eventually be pointed at nothing.
3. Client-side discovery
The client queries a registry (Consul, etcd, Eureka, ZooKeeper), receives the full list of healthy instances, and load-balances itself.
Gains: no extra network hop, and the client can be genuinely smart — least-connections, zone-aware routing to avoid cross-AZ charges (Module O-8), subsetting so each client talks to a subset of a large fleet rather than holding a thousand connections, and immediate reaction to a registry change.
Costs: a discovery library per language, which must be kept in step across every service — the coupling Module A-4 warned about — and every client is now responsible for getting load balancing and health interpretation right. In a polyglot estate this is the thing that does not scale, because "upgrade the discovery library everywhere" becomes a quarterly project.
4. Server-side discovery
The client talks to one stable address — a load balancer, a Kubernetes Service, a proxy — which knows the instances.
Gains: language-agnostic, centralised policy, nothing to upgrade in the clients.
Costs: an extra hop on every call, and the balancer is a critical-path component you must make highly available. It also cannot do client-aware things like zone-local preference as well as a client can, though modern balancers approximate it.
5. Sidecar (the mesh data plane)
A proxy process runs beside every service instance. The application connects to localhost, and the sidecar handles discovery, load balancing, retries, encryption and telemetry.
Gains: language-agnostic and client-side-smart, with no shared balancer to bottleneck, and policy managed centrally without touching application code.
Costs: the subject of the second half of this module.
Extra hop
Per-language work
Smart LB
Central policy
Static config
no
none
no
no
DNS
no
none
no
no
Client-side
no
a library per language
yes
hard
Server-side
yes
none
limited
yes
Sidecar
yes (localhost)
none
yes
yes
Registration: Who Tells the Registry, and What Happens When Nobody Does
Self-registration. The service registers itself on start-up and deregisters on shutdown. Simple, and it has a specific failure: a process that is SIGKILLed, OOM-killed, or on a node that lost power never deregisters. The registry now advertises an instance that does not exist, and every client sending traffic there gets connection refused — or worse, a timeout, if the whole node vanished and the packets go nowhere.
The mitigation is a TTL with a heartbeat: the registration expires unless renewed, so a dead instance ages out. Which is the same lease pattern as Module P-19's job claim, Module A-2's lock and Module P-16's idempotency record, for the same reason — any ownership must expire on its own — and it comes with the same tuning trade: a short TTL detects death quickly and generates constant heartbeat write traffic against a consensus system (Module A-1), while a long TTL leaves a black hole in the pool for its duration.
Third-party registration. The platform registers on the service's behalf, because the platform is the thing that knows whether the process exists. Kubernetes does this: a pod passing its readiness probe is added to the EndpointSlice, and one that dies is removed by the control plane whether or not it managed to run any shutdown code.
Prefer platform registration. It removes an entire class of stale-entry bug, because the component doing the registering is not the component that died.
Health Checks Are Where Discovery Goes Wrong
Discovery decides where; health decides whether. Getting the second wrong causes outages, and there is one mistake that causes most of them.
Liveness and readiness are different questions
Liveness: "is this process broken beyond recovery?" A failing liveness check means restart me.
Readiness: "should I receive traffic right now?" A failing readiness check means take me out of the pool, but leave me running.
Conflating them produces a restart loop: a slow dependency makes the check fail, the platform restarts a perfectly healthy process, the restart makes things worse, and the loop continues until someone intervenes.
The readiness probe that causes a total outage
Here is the anti-pattern, and it has taken down large systems:
ts
The database has a two-second blip — a failover, a brief lock storm, a network hiccup. Every instance fails its readiness check simultaneously, because they all depend on the same database. Every instance is removed from the load balancer. There are now zero healthy backends, so the balancer serves 503s for everything — including the 90% of requests that did not need the database at all. A partial degradation has become a total outage, caused entirely by the health check.
The rule:
Readiness answers "can I serve requests," not "are my dependencies healthy."
Dependency health is the job of timeouts, retries and circuit breakers (Modules A-6, A-7), which degrade the affected requests rather than removing the whole instance. A readiness probe should check what is local: is the HTTP server listening, is initialisation complete, are the caches warmed, is the event loop responsive. If a dependency check must influence readiness, it needs a floor — most load balancers support a minimum-healthy-percentage or "fail open when all backends are unhealthy" behaviour, which is a safety net worth configuring explicitly.
Detection semantics
Active checks probe on an interval; detection latency is interval × failure threshold, which with a 10-second interval and three failures is 30 seconds of sending traffic to a dead instance. Passive checks (outlier detection) observe real traffic and eject an instance that returns errors or times out, which is much faster because production traffic is the probe. Use both: active to gate rotation membership, passive to eject a misbehaving instance between probes.
Draining, without which every deploy serves errors
Terminating an instance correctly is a sequence, and skipping any step produces user-visible errors on every single deploy:
text
Step 2 is the one that matters. Deregistration is not instantaneous — a load balancer's or mesh's view takes hundreds of milliseconds to seconds to update, and DNS-based paths take longer. A process that exits immediately on SIGTERM is still receiving requests when it closes its listener, and each of those becomes a 502. The fix is a deliberate delay: a preStop sleep of a few seconds, or a shutdown handler that fails readiness first and only then begins draining.
ts
And the grace period must exceed the longest in-flight request, or the platform will SIGKILL mid-request — which for a payment (Module P-24) is exactly the ambiguous failure the whole system is built to survive, generated on purpose by your own deploy.
The Service Mesh: What the Sidecar Buys
A mesh is a data plane (a proxy — usually Envoy — beside every workload) plus a control plane (Istio, Linkerd, Consul Connect) that configures them. Traffic is intercepted transparently, so the application makes an ordinary HTTP call to localhost or to the service name and the proxy does the rest.
What you get, roughly in order of how much it justifies the cost:
1. mTLS everywhere, with automatic identity and rotation. This is the strongest single argument. Every workload gets a cryptographic identity (SPIFFE), every hop is mutually authenticated and encrypted, and certificates rotate hourly without anyone writing code. Doing this by hand across a polyglot estate — issuing, distributing, rotating and validating certificates in every language's TLS stack — is a large project that is never finished. A mesh makes it a configuration setting, which is also the shortest path to satisfying a zero-trust requirement (Module A-13).
2. Uniform resilience policy. Timeouts, retries with budgets, outlier ejection and circuit breaking (Modules A-6, A-7) applied consistently, without a library per language and without hoping every team configured theirs correctly. Note the important caveat: a proxy can only retry requests it knows are safe, so retry policy must distinguish idempotent from non-idempotent operations, and a mesh that blindly retries a POST will duplicate work (Module P-16).
3. Traffic shifting. Route 1% of traffic to a new version, or route by header for a specific tenant, as a config change rather than a deploy — which is the mechanism canary releases need (Module O-4).
4. Telemetry for every hop, for free. Request rate, error rate and latency distribution per service pair, without instrumenting anything. This is genuinely valuable, because "which call between which two services regressed" is otherwise expensive to answer (Module O-1). It does not replace application tracing — the proxy cannot see inside your process — but it gives you the golden signals of the network for nothing.
5. Authorisation between services. "Only the checkout service may call the payments service, on this path, with this method," enforced at the proxy rather than in application code.
And What It Costs, Frankly
This bill is routinely understated in mesh evaluations.
Latency: two proxy hops per call. Your sidecar and theirs, each adding roughly 0.5–2 ms depending on configuration and load — and considerably more at p99 under CPU contention, because the proxy competes for the same cores as your application. For a 50 ms call that is noise. For a 3 ms internal cache lookup it is a doubling, and on a request that fans out to eight services (Module A-4) the amplification arithmetic applies to the added latency too.
Memory and CPU, multiplied by every pod. A sidecar is typically 50–150 MB of resident memory plus meaningful CPU under load. At 500 pods that is 25–75 GB of RAM spent on proxies, plus the CPU, plus the control plane. It is a real line item (Module O-8), and it is the reason ambient or sidecar-less modes — a shared per-node proxy instead of one per pod — are the current direction of the technology, trading some per-workload isolation for a large reduction in overhead.
Operational surface: you have adopted another distributed system. The control plane can fail, needs upgrading in step with your platform, and speaks a configuration model (xDS, VirtualService, DestinationRule) that is its own body of knowledge. Most meshes fail static when the control plane is unavailable — proxies keep their last-known config and keep working — which is the right design and worth verifying for yours rather than assuming.
Debugging gains a suspect. Every unexplained 503 now has one extra candidate: was that my application, the local sidecar, or the remote sidecar? Meshes produce their own response codes and flags, and learning to read them is a skill your on-call rotation now needs. Related surprises: traffic interception via iptables interacts badly with init containers that need network access before the proxy is ready, non-HTTP protocols may need explicit configuration, and a service doing its own TLS inside the mesh gets double encryption unless configured otherwise.
A sidecar OOM takes the application's network with it. The failure is total for that pod and looks like the application being unable to reach anything.
Skill cost. A mesh is a specialty. A team without platform-engineering capacity will operate it badly, and a badly-operated mesh is worse than no mesh — it adds latency and failure modes while its security and policy features sit misconfigured.
The ladder: cheaper rungs first
Situation
Right rung
≤ 5 services, one language
DNS or platform Service, plus a good resilience library
API gateway (Module F-15) — this is not what a mesh is for
Many services, polyglot, mTLS mandated
mesh
Many services, no platform team
mesh is a liability; standardise a library instead
The north–south versus east–west distinction is worth naming because it prevents a common confusion: a gateway handles traffic entering your system from outside (authentication, rate limiting, TLS termination, routing — Module F-15), while a mesh handles traffic between your services. They are complementary, and a mesh is not a replacement for a gateway.
Why this matters in production: the failures in this module are concentrated in the boring half, not the exciting half. Almost nobody has an outage caused by choosing Consul over etcd; a great many teams serve 502s on every deploy because nothing waits for deregistration to propagate, take a total outage because a readiness probe checks the database, or spend a day on "DNS is correct but traffic still goes to the old instance" before discovering keep-alive. Get registration, readiness and draining right before evaluating a mesh — a mesh will not fix any of the three, and it will hide them behind more machinery.
Where This Shows Up in Your Stack
DNS and Node.js: the keep-alive trap is the one to check for first. undici/fetch agents pool connections aggressively, so set a maximum connection lifetime and a maximum requests-per-connection so sockets recycle and DNS is re-consulted. Node caches resolutions per lookup rather than forever (unlike the historic JVM default), but the pool holding open sockets is the real staleness source, not the resolver.
Kubernetes:Service plus EndpointSlice is server-side discovery with platform registration — the best default for most teams and enough on its own for a long time. Get three things right: readiness probes that check only local health, a preStop sleep so deregistration propagates before the process stops listening, and a terminationGracePeriodSeconds longer than your longest request. Topology-aware routing keeps traffic in-zone, which is both faster and cheaper (Module O-8).
Consul / etcd / ZooKeeper: a registry backed by consensus (Module A-1), which means every registration and heartbeat is a consensus write. Aggressive heartbeat intervals across a large fleet can saturate the registry — the classic way to take down a discovery system is to make its TTLs too short.
Envoy / Istio / Linkerd: Envoy is the data plane in most meshes and is worth understanding independently, since it is also an excellent standalone proxy. Verify that your control plane fails static, budget the sidecar memory per pod in your capacity model, and evaluate ambient mode if per-pod overhead is significant at your scale.
PostgreSQL: the database needs discovery too, and it is the case with the least tolerance for staleness. After a failover, clients must reach the new primary — via a stable name updated by the HA tool (Patroni plus HAProxy or a floating endpoint), or via a connection string listing both hosts with target_session_attrs=read-write. Pointing at a stale primary yields writes to a read-only node or, in the worst case, split brain (Module A-1). A pooler such as PgBouncer also serves as the stable address in front of a changing primary, which is worth having for the connection-count reasons in Module P-4 anyway.
Summary
Discovery answers three separate questions: what instances exist (registry), which are usable (health), and which to pick (selection policy). Different mechanisms answer them to very different standards.
DNS is universal and full of traps: TTL is a hint many clients ignore, there is no health awareness beyond record removal, selection is round-robin only, and ports need SRV records that are patchily supported.
The trap that catches everyone: a connection pool with keep-alive never re-resolves DNS.dig returns the right answer while your client keeps using the address it resolved at start-up. Fix it with a maximum connection lifetime and a maximum requests-per-connection so sockets recycle — this is also why load-balancer clients must respect TTLs, since the balancer's own IPs rotate.
Client-side discovery avoids the extra hop and enables genuinely smart load balancing — least-connections, zone-aware routing that saves cross-AZ charges, subsetting — at the cost of a library per language that must be upgraded everywhere, which is what fails in a polyglot estate. Server-side discovery is language-agnostic with central policy, at the cost of a hop and a critical-path component.
Prefer platform registration over self-registration. A process that is SIGKILLed or whose node vanished never deregisters, leaving a black hole in the pool; a TTL with heartbeats ages it out, with the familiar trade between detection speed and constant write load against a consensus system.
Liveness means restart me; readiness means stop sending traffic. Conflating them produces restart loops when a dependency is slow.
A readiness probe that checks a shared dependency can cause a total outage. A two-second database blip fails every instance's probe simultaneously, removes every backend, and turns a partial degradation into 503s for all traffic — including requests that never needed the database. Readiness answers "can I serve," never "are my dependencies healthy"; dependency failure belongs to timeouts and circuit breakers, which degrade affected requests rather than the whole instance. Configure a fail-open floor as a safety net.
Use active checks to gate rotation membership and passive outlier detection to eject fast, since detection latency for active checks is interval × threshold — 30 seconds of traffic into a dead instance is typical.
Draining has a step everyone omits: waiting for deregistration to propagate. Fail readiness, wait a few seconds, then stop accepting and drain in-flight work. Without the wait, every deploy serves 502s. And the grace period must exceed your longest request, or the platform SIGKILLs mid-request and manufactures the exact ambiguous failure your payment system is built to survive.
A mesh's strongest justification is mTLS with automatic per-workload identity and hourly rotation — a project that is never finished by hand across a polyglot estate becomes a configuration setting. Then uniform retries, timeouts and breakers without a library per language; percentage- and header-based traffic shifting for canaries; per-service-pair golden signals for free; and service-to-service authorisation at the proxy. Note that a proxy can only safely retry idempotent requests.
The costs, stated frankly: two proxy hops per call at roughly 0.5–2 ms each and worse at p99 under CPU contention; 50–150 MB plus CPU per pod, which is 25–75 GB of RAM at 500 pods; another distributed system to operate and upgrade, with its own configuration language; one more suspect in every 503 investigation; interception surprises with init containers, non-HTTP protocols and in-app TLS; and a sidecar OOM that removes the pod's entire network. Ambient/sidecar-less modes exist precisely to reduce the per-pod overhead.
Climb the ladder in order: DNS or a platform Service for a handful of services, client-side discovery plus a resilience library up to a dozen or so, a gateway for north–south traffic (which a mesh does not replace), and a mesh when you have many services, multiple languages, and a mandate for mTLS — but not without a platform team, because a badly-operated mesh adds latency and failure modes while its best features sit misconfigured.
The real outages are in the boring half. Nobody's incident was caused by choosing Consul over etcd; plenty are caused by no drain delay, by a readiness probe that checks the database, and by keep-alive holding a stale address. Fix those three before evaluating a mesh, which will hide them rather than solve them.
The next module, Timeouts, Retries, and Backoff, is the resilience policy a mesh would apply on your behalf — and which you need to understand regardless, because the defaults are wrong in a specific and dangerous direction. It covers retry storms as self-inflicted denial of service, why jitter is not optional, retry budgets as the mechanism that bounds amplification, and deadline propagation, without which a retry deep in a call graph is work nobody is waiting for any more.
Knowledge Check
A service's readiness endpoint executes SELECT 1 against Postgres and a Redis PING. During a 3-second Postgres failover, the service returns 503 for *all* traffic for nearly a minute — including endpoints that never touch Postgres — and recovery takes far longer than the failover itself. What is the mechanism and the correct design?
After migrating to an autoscaled fleet behind an internal load balancer, a Node service intermittently gets connection timeouts to a downstream service. dig from the same host returns correct, current addresses. Restarting the calling service fixes it, for a while. What is happening?
A platform team with eight services in one language, all deployed on Kubernetes, proposes adopting a full service mesh, citing observability and resilience. The team has no dedicated platform engineers. What does the module advise, and what would change the recommendation?
Test your knowledge with more question sets
Sign in to access a wider variety of questions and get notified when new practice sets are added to this module.
// DANGEROUS: readiness reports on a dependency you share with every other instance.app.get("/ready",async(_req, res)=>{await db.query("SELECT 1");// ← a shared dependencyawait redis.ping();// ← another one res.send("ok");});
1. Mark unready / deregister → the balancer stops sending new requests
2. WAIT for propagation → ← the step everyone omits
3. Finish in-flight requests → stop accepting, drain the server
4. Close pools, flush, exit