Module A-7·28 min read

Failing fast on purpose, isolating pools so one dependency cannot eat the whole thread budget, and shedding features instead of dropping requests.

JJS

Written by Jatin Jain Saraf · Senior Software Engineer

Module A-7 — Circuit Breakers, Bulkheads, and Graceful Degradation

Q: A team's circuit breaker opens for their address-validation service several times a day, disabling validation for everyone. Investigation shows the service is healthy throughout; the failures counted are `422` responses to malformed postcodes submitted by one integration partner. Separately, the same breaker never opens when the service degrades to 9-second response times at 100% success. What two configuration changes does the module prescribe?

Exclude `4xx` from the failure predicate — a `422` means the dependency is healthy and rejected the input, so a client bug should never take it down for everyone — and add a **slow-call-rate** threshold, because a dependency at 100% success and 9 seconds is as damaging as one that is down and an error-rate breaker cannot see it. — Both are the module's named misconfigurations: counting `4xx` lets a client bug or scanner open a breaker against a working dependency, and the slow-call-rate threshold is the check most implementations omit even though it catches the degradation mode an error breaker misses. Option A is the strongest near-miss and is half-workable — lowering the timeout would indeed convert 9-second calls into recorded failures — but it treats the `422` problem by tolerating more of it rather than classifying it correctly, so a larger volume from that partner still trips the breaker, and it forces a timeout choice that Module A-6 warns should come from the successful latency distribution rather than from breaker mechanics. Option B moves backwards on the module's first configuration rule: a consecutive counter never trips on a partially-failing dependency, which is the case it most needs to catch. Option C relocates the problem; a mesh breaker also needs its failure predicate configured, and the module notes a mesh cannot degrade at all.

Q: A service caches aggressively in Redis, achieving a 95% hit ratio. Its cache-miss path is a moderately expensive Postgres query. When Redis becomes unavailable, the service correctly falls back to Postgres — and Postgres immediately saturates, taking down endpoints unrelated to the cached data. What does the module say went wrong?

The fallback depends on a resource the cache was protecting: losing a 95% hit ratio redirects roughly 20× the read load onto Postgres, so the fallback converted a cache outage into a database outage. The degraded path must sit behind a bulkhead and request coalescing, with excess load shed, accepting degraded service rather than manufacturing a second failure. — The module presents this as the most common broken fallback and does the 20× arithmetic explicitly, prescribing a limiter plus coalescing rather than either extreme. Option C is necessary and the module recommends it — tight Redis timeouts plus a breaker — but it changes *when* traffic reaches Postgres, not *how much*: after the breaker opens, 100% of reads still go to the database, so the saturation is unchanged. Option B overcorrects into a rule the module does not endorse: a database-backed degraded path is legitimate and often correct, provided it is bounded, and refusing all reads is a harsher outcome than serving a limited fraction. Option D reduces the probability without addressing the design, and Module A-1 notes Redis failover is not itself safe, so a replica does not remove the failure mode.

Q: Three weeks after a deploy that changed a hostname, a product manager asks why personalised recommendations have disappeared from the homepage. Engineering finds the recommendations breaker has been open continuously since that deploy. Error rates, latency and SLO attainment were all normal throughout. What does the module identify as the missing instrumentation, and why is this failure shape specific to degradation?

Degradation removes failure from the error rate: requests succeed, latency may even improve because a fast fallback replaced a slow call, so every dashboard stays green while the feature is gone. The instrumentation needed is breaker state per dependency with an alert on prolonged open, a **degraded-serve rate as an SLI** shown alongside the error rate, and fallback execution counts. — This is the module's named production trap, told with the same three-week shape and the same discovery route — a product manager noticing an engagement drop — and it lists exactly the metrics that would have caught it, with the framing that a degraded-serve rate is the error rate you decided not to show users. Option A is the plausible mechanical explanation and is wrong on the facts as given: a half-open probe against a hostname that no longer resolves fails every time regardless of cooldown, so the breaker reopening promptly changes nothing — the problem is that nobody was told. Option C describes a real principle from the module, but here the fallback *worked* — that is precisely why nothing alerted; the untested path was not the defect. Option D would likely have caught this particular deploy and does nothing for the general case, since any dependency can become unreachable long after a deploy, which is why the module's answer is observability rather than release process.

What this module covers: What a circuit breaker adds that a timeout cannot — stopping calls entirely once it is clear they will fail, which both saves your resources and removes the load keeping the dependency down. The state machine in enough detail to configure it correctly: why a rolling failure rate with a minimum-volume guard beats a consecutive-failure count, why a 4xx must not count as a failure, and why a slow-call-rate threshold matters as much as an error threshold. Bulkheads as resource isolation, and the specific Node.js form of them, since there are no thread pools to partition. Then the part that is design work rather than configuration: graceful degradation — classifying every dependency as critical or optional, choosing a fallback that is genuinely cheaper and does not depend on the thing that failed, and deciding fail-open versus fail-closed explicitly. Finally the production insight that ties it together: degradation removes failure from your error rate, so a system can serve fallbacks permanently while every dashboard stays green.

A Timeout Bounds Each Call; a Breaker Stops Making Them

Module A-6 gave you a bound per call. Now consider a dependency that is genuinely down, with a 2-second timeout and 500 requests per second arriving:

text

500 req/s × 2s each  =  1,000 concurrent in-flight requests
                        …all of which will fail
                        …each holding a socket, a slot, and memory
                        …and all of it is load on a dependency that is trying to restart

The timeout did its job — each request failed in bounded time — and the aggregate is still an outage. You are spending your entire concurrency budget discovering, five hundred times a second, something you already knew.

A circuit breaker is the observation that after enough failures you can stop asking. It converts a 2-second failing call into a sub-millisecond local rejection, which does three things at once:

Frees your resources. Nothing is held for two seconds.
Removes load from the dependency, giving it room to recover — the exit from Module A-6's metastable state.
Fails fast enough for a fallback to be viable. A 2-second wait before serving a cached value is a bad experience; a 1-millisecond rejection then a cache read is a good one.

Point 2 is easy to undervalue and is often the decisive one. A dependency at 100% error rate does not recover while receiving its full traffic plus retries; a breaker is how the traffic stops without a human intervening.

Analogy: the electrical circuit breaker the pattern is named for, and it is worth being precise about why it exists. It does not protect the appliance that shorted — that appliance is already broken. It protects the rest of the house from the fault, by isolating a circuit so the wiring does not overheat and the other rooms keep their lights on. Then it needs a human, or a delay, before reclosing — because closing it immediately onto an unfixed short achieves nothing except another trip.

The State Machine, and the Four Ways It Is Misconfigured

text

        failure rate over window > threshold
CLOSED ──────────────────────────────────────► OPEN
  ▲                                             │  reject immediately, no call
  │                                             │  wait cooldown (e.g. 10s)
  │  probe succeeds                             ▼
  └──────────────────────── HALF-OPEN ◄─────────┘
                                │  allow 1–3 trial requests
                                └──► probe fails → back to OPEN

Simple enough to write in an afternoon, and four configuration decisions determine whether it helps or hurts.

1. Use a failure rate over a rolling window, with a minimum-volume guard. The obvious implementation counts consecutive failures, and it fails in both directions: on a low-traffic endpoint, five consecutive failures might span twenty minutes and open a breaker for a problem long since resolved; and on a partially-degraded dependency where every other call succeeds, a consecutive counter never reaches five while half your requests fail.

// Rolling window, rate-based, with a minimum volume so a 1-of-1 failure cannot trip it.
const stats = window.last(10_000);                 // last 10 seconds
if (stats.total >= 20 && stats.failureRate > 0.5) open();

The minimum-volume guard is what stops the most annoying false positive: the first request after a quiet period fails transiently, giving a 100% failure rate over a sample of one, and the breaker opens for a dependency that is fine.

2. A 4xx is not a failure. This is the misconfiguration that causes the most confusion. A 400, 404 or 422 means the dependency is healthy and rejected your request — validation failed, the resource does not exist, the input was malformed. Counting those as breaker failures means a client bug, a scanner probing for URLs, or one badly-formed integration can open your breaker and take down a working dependency for everyone.

Count as failures: timeouts, connection errors, 5xx, and 429 only if you choose to treat rate limiting as a signal to back off entirely (usually better handled by honouring Retry-After, Module P-18).

3. Threshold on slow calls too, not just errors. A dependency returning 100% success at 8 seconds per call is as damaging as one that is down — worse, in a sense, because nothing looks like an error. A slow-call-rate threshold ("if more than 60% of calls exceed 1 second, open") catches the degradation mode that an error-rate breaker misses entirely, and it is the check most implementations omit.

4. Half-open must admit a trickle, not a flood. A cooldown that expires and lets all 500 requests/second through at once re-hammers a recovering dependency and immediately reopens. Admit one to three concurrent probes, require a small number of successes before closing, and consider ramping — half-open at 5% of traffic, then 25%, then closed. The reopening path should also back off exponentially: a dependency that has failed its probe four times should not be probed every 10 seconds forever.

Granularity: per dependency, and per operation where they differ. One breaker for "the payments service" is usually right; one breaker for "the whole downstream world" is useless, because an unrelated failure blocks working functionality. If one endpoint of a dependency is expensive and flaky while the rest are fine, give that endpoint its own breaker.

And distinguish this from outlier ejection (Module A-5), which operates at instance level: the load balancer or mesh removes the one bad replica from rotation, which is the right response to one sick instance. A circuit breaker responds to the dependency as a whole being unusable. You want both, and they are not substitutes — outlier ejection routes around a partial failure, a breaker gives up on a total one.

Bulkheads: Stop One Dependency Consuming Everything

A ship's hull is divided into watertight compartments so a breach floods one and not the vessel. The system equivalent is: partition the resource that requests compete for, so one dependency's slowness cannot consume all of it.

Concretely, the resource being protected is different in different runtimes, and naming it matters:

Runtime	The scarce shared resource	Bulkhead form
JVM / thread-per-request	worker threads	a thread pool per dependency
Node.js	event-loop time, sockets, and pending promises	a concurrency semaphore per dependency
Any	database connections	a separate pool per workload
Any	worker capacity	separate queues and worker pools (Module P-19)
Any	the whole process	a separate deployment (Module A-4)

The failure a bulkhead prevents:

text

Without: 200 concurrent slots, shared.
  The recommendations service slows to 8s. Requests to it accumulate.
  Within seconds all 200 slots hold recommendation calls.
  Checkout — which never calls recommendations — gets no slot. Total outage.

With:  recommendations capped at 20 concurrent.
  Slot 21 onwards is rejected immediately → the fallback runs.
  Checkout's 180 slots are untouched. Recommendations are degraded. Nothing else is.

In Node there are no request threads to partition, which leads people to conclude bulkheads do not apply. They do — the shared resources are the socket pool, memory held by pending promises, and event-loop time. The mechanism is a semaphore:

import pLimit from "p-limit";
const limits = {
  recommendations: pLimit(20),     // optional feature: small allowance
  payments:        pLimit(60),     // critical path: generous
  search:          pLimit(30),
};

// Rejecting when full is the point — queueing indefinitely recreates the problem.
async function callRecommendations(userId: string) {
  if (limits.recommendations.pendingCount > 50) throw new BulkheadFull();
  return limits.recommendations(() => fetchRecs(userId));
}

Note the explicit rejection when the queue is deep. A semaphore that queues without bound is not a bulkhead — it is a delay, and the memory held by the queue is the resource you were trying to protect.

Sizing comes from Little's Law (Module F-4): a dependency with a 50 ms p99 that must sustain 200 requests/second needs 200 × 0.05 = 10 concurrent slots. Give it 15 for headroom, not 200. Sizing every pool for the worst case defeats the isolation, because the sum of the worst cases is the shared pool you started with.

And the most valuable bulkhead is often the coarsest: separate connection pools per workload. The web role and the reporting role should not draw from the same Postgres pool, or Module P-22's analytical query starves checkout — which is the same argument as Module P-8's per-tenant connection budget, applied per workload instead of per tenant.

Graceful Degradation Is the Design Work

Breakers and bulkheads decide when to stop calling something. Degradation decides what the user gets instead, and it is the part that cannot be configured — it requires a decision per dependency, and often a product decision rather than an engineering one.

Start by classifying every dependency:

Dependency	Class	If it fails
Primary database (write path)	critical	fail the request; there is no honest fallback
Auth service	critical, fail closed	reject — never fail open on authorisation
Payment provider	critical for checkout	fail fast, do not queue (Module P-24)
Cache (Redis)	optional	read from source — but see the stampede warning below
Recommendations	optional	show a static popular-items list
Personalisation	optional	serve the generic variant
Search	optional-ish	fall back to a simple database filter, or recent items
Avatar / image service	optional	initials on a coloured background
Fraud scoring	a business decision	accept and flag for manual review, or reject — not yours to choose
Analytics / telemetry	optional	drop it silently; never fail a user request for a metric

The fraud row is there deliberately. "If the fraud service is down, do we accept payments unscored?" is a question about risk appetite and money, and an engineer choosing it unilaterally in a catch block has made a business decision by accident. The right output of this exercise is a table someone in the business has agreed to.

Three rules for a fallback

1. It must not depend on what failed. The most common broken fallback:

// BROKEN: the fallback path hits the same overloaded database.
try   { return await cache.get(key); }
catch { return await db.expensiveQuery(); }   // ← 100% of traffic now goes here

Redis going down does not merely remove the cache; it redirects the entire cache hit rate onto the database (Module P-12's stampede). A cache with a 95% hit ratio failing means the database sees 20× its normal read load — so the fallback has converted a cache outage into a database outage. The correct handling is a fallback plus a limiter: serve from the database behind a bulkhead and a request-coalescing lock, shed the excess (Module A-8), and accept degraded service rather than manufacturing a second failure.

2. It must be genuinely cheaper. A fallback that costs more than the real path is a load amplifier during an incident. Static content, a precomputed list, a cached-but-stale value, or an empty state all qualify. Recomputing something expensive does not.

3. Stale data needs a bound and a signal. Serving a cached value from a failed dependency is often the best fallback available — with two conditions: a maximum staleness beyond which you would rather show nothing (a 4-hour-old price is worse than "temporarily unavailable"), and an indication to the user or the client that the data is stale, so downstream systems do not treat it as authoritative.

Fail open or fail closed, decided per dependency

Fail closed (deny on failure): authentication, authorisation, fraud limits where the business chose strictness, quota enforcement that gates money. If the check cannot run, the answer is no.
Fail open (allow on failure): rate limiters for fairness, feature flags, personalisation, analytics, non-security enrichment.

Write the choice down per dependency. The default in most code is "fail open," silently, because it is what an empty catch block does — and a security check that fails open because nobody decided is the worst kind of vulnerability, since it appears only during an incident when nobody is looking at authorisation.

The fallback must be exercised, or it is decoration

An untested fallback path is worse than none, because it creates the belief that you are protected while the code has never run. And it will first execute during an incident, which is the least forgiving moment to discover a null-pointer error in it.

Two mechanisms make this real: kill switches you can flip deliberately (a feature flag per optional dependency, exercised in staging and occasionally in production), and fault injection to make dependencies fail on purpose (Module O-6, which exists largely to prove that this module's mechanisms work).

A caveat on flags: the flag system must not be a dependency that can fail. Flags should be cached locally with a long TTL and a compiled-in default, so an unreachable flag service means "last known values," never "no features."

The Insight That Catches Teams: Degradation Hides Failure

Here is the production trap unique to this module.

Once degradation is working, a failing dependency produces no errors. Requests succeed. Latency may even improve, because a fast fallback replaced a slow call. Your error-rate dashboard is flat, your SLO is intact, and your alerts are quiet.

So a system can serve the generic homepage instead of the personalised one, for three weeks, because a breaker opened after a deploy changed a hostname and nothing ever closed it. Nobody notices. The feature is simply gone, and the first report is from a product manager asking why engagement dropped.

The instrumentation that prevents this:

Breaker state as a metric, per dependency, with an alert on "open for longer than N minutes." An open breaker is a suppressed failure, and suppressed failures need owners.
A degraded-serve rate as an SLI: the percentage of requests served with any fallback. This belongs on the same dashboard as your error rate, because it is the error rate you decided not to show users.
Bulkhead rejection counts per dependency, which tell you a pool is undersized before it becomes visible.
Fallback execution counts, so a fallback that has never run in production is identifiable as untested.

Why this matters in production: the mechanisms in this module are the difference between a dependency failure and an outage, and they are also the mechanisms most likely to be quietly misconfigured — a breaker counting 4xx, a bulkhead sized for the worst case, a fallback that hits the failed database, a security check failing open by omission. None of those is visible in normal operation. All of them are visible during an incident, which is why the chapter's real conclusion is that these paths must be exercised deliberately rather than trusted.

Where This Shows Up in Your Stack

Node.js: the shared resource is the event loop and the socket pool, so bulkheads are semaphores (p-limit or equivalent) that reject when full rather than queueing without bound. opossum is the common breaker; configure it rate-based with a volume threshold, exclude 4xx from the failure predicate, and add a slow-call threshold. And remember a synchronous CPU-bound handler blocks everything including your breaker's own timers (Module P-19).
PostgreSQL: the highest-value bulkhead you can deploy is separate connection pools per workload — web, background jobs, reporting — with a statement_timeout per role (Module P-22). It stops one workload's slowness from starving the others in the way a shared pool guarantees (Module P-4).
Redis: the archetypal optional dependency, and the archetypal broken fallback. Tight timeouts (tens of milliseconds), a breaker, and a degraded mode that reads from the source behind a limiter, because a cache with a 95% hit ratio failing sends 20× read load to the database. Request coalescing (one fetch per key, others wait) is what makes the degraded mode survivable (Module P-12).
Payment providers and other third parties: breakers here must fail fast rather than queue, because charging a customer twenty minutes after they saw a failure produces disputes (Module P-24). Third parties are also where per-operation breakers earn their keep — refunds being down does not mean charges are.
Service mesh: gives you outlier ejection and connection-pool circuit breaking at the proxy without application code (Module A-5), which is genuinely valuable. What it cannot do is degrade: a proxy can fail a call, but only your application knows that a failed recommendations call should render popular items. Mesh-level breaking and application-level fallbacks are complementary, not alternatives.
Feature flags: the mechanism for deliberate degradation and the kill switches that make fallbacks exercisable (Module O-4). Cache flag values locally with compiled-in defaults so the flag service is never a hard dependency.

Summary

A timeout bounds each call; a breaker stops making them. At 500 requests/second with a 2-second timeout against a dead dependency you hold 1,000 concurrent doomed requests — each individually well-behaved, collectively an outage. A breaker converts that into sub-millisecond local rejections, freeing your resources, removing the load that prevents the dependency recovering, and failing fast enough for a fallback to be worth having.
Trigger on a failure rate over a rolling window with a minimum-volume guard, not a consecutive-failure count — which trips late on low-traffic endpoints and never trips on a half-failing dependency, and which lets a single failure after a quiet period open the breaker at a "100% failure rate."
A 4xx is not a failure. It means the dependency is healthy and rejected your request, so counting it lets a client bug or a URL scanner take down a working dependency for everyone.
Add a slow-call-rate threshold. A dependency at 100% success and 8 seconds per call is as damaging as one that is down, and an error-rate breaker never sees it.
Half-open admits a trickle, not a flood — one to three concurrent probes, a few successes before closing, ideally a ramp — and the reopen path should back off, so a persistently dead dependency is not probed every ten seconds forever.
Breaker granularity is per dependency, and per operation where they differ. Distinguish it from outlier ejection, which removes one sick instance from rotation: routing around a partial failure and giving up on a total one are different jobs and you want both.
A bulkhead partitions the resource requests compete for, and naming that resource matters: threads on the JVM, but in Node it is event-loop time, sockets and pending promises, so the form is a semaphore that rejects when full — one that queues without bound is a delay, not a bulkhead. Size from Little's Law, not from the worst case, because the sum of worst cases is the shared pool you were trying to escape.
The highest-value bulkhead is usually separate connection pools per workload, which stops a reporting query from starving checkout.
Classify every dependency as critical or optional, and treat some as business decisions. "Do we accept payments when fraud scoring is down?" is a risk-appetite question, and an engineer answering it inside a catch block has made a business decision by accident.
A fallback must not depend on what failed. The archetypal bug: a cache miss falls through to the database, so Redis failing redirects a 95% hit rate onto Postgres and converts a cache outage into a database outage. Serve degraded behind a limiter with request coalescing, and shed the excess.
A fallback must be cheaper than the real path, or it amplifies load during an incident; and stale data needs a staleness bound and a signal, because a four-hour-old price is worse than "temporarily unavailable."
Decide fail-open versus fail-closed per dependency and write it down. Authentication, authorisation and money-gating quotas fail closed; fairness limiters, flags, personalisation and analytics fail open. The default in code is "fail open, silently," because that is what an empty catch does — and a security check that fails open by omission is a vulnerability that appears only during an incident.
An untested fallback is worse than none, because it creates false confidence and first executes at the worst possible moment. Kill switches make it exercisable; fault injection (Module O-6) proves it.
The flag system must never be a hard dependency: cache locally with compiled-in defaults so unreachable means "last known values," not "no features."
Degradation removes failure from your error rate, so a breaker can stay open for three weeks after a deploy changed a hostname and nobody notices — the feature is simply gone. Instrument breaker state with an alert on prolonged open, a degraded-serve rate as an SLI on the same dashboard as errors, bulkhead rejection counts, and fallback execution counts so an untested path is identifiable.

The next module, Backpressure and Load Shedding, addresses the case these mechanisms do not cover. A breaker responds to a dependency failing; shedding responds to you being over capacity with every dependency healthy. It covers queue depth as the earliest honest signal, why an unbounded queue is a lie rather than a buffer, admission control that rejects before work is started, and how to prioritise traffic when there is not enough capacity for all of it.

Knowledge Check

A team's circuit breaker opens for their address-validation service several times a day, disabling validation for everyone. Investigation shows the service is healthy throughout; the failures counted are 422 responses to malformed postcodes submitted by one integration partner. Separately, the same breaker never opens when the service degrades to 9-second response times at 100% success. What two configuration changes does the module prescribe?

A service caches aggressively in Redis, achieving a 95% hit ratio. Its cache-miss path is a moderately expensive Postgres query. When Redis becomes unavailable, the service correctly falls back to Postgres — and Postgres immediately saturates, taking down endpoints unrelated to the cached data. What does the module say went wrong?

Three weeks after a deploy that changed a hostname, a product manager asks why personalised recommendations have disappeared from the homepage. Engineering finds the recommendations breaker has been open continuously since that deploy. Error rates, latency and SLO attainment were all normal throughout. What does the module identify as the missing instrumentation, and why is this failure shape specific to degradation?

Test your knowledge with more question sets

Sign in to keep reading

The rest of this module is free — sign in with Google to unlock it and track your progress.

PreviousModule A-6: Timeouts, Retries, and Backoff Next Module A-8: Backpressure and Load Shedding

Discussion

Join the discussion

Loading comments...