Module O-6 — Chaos Engineering and Fault Injection
What this module covers: That every failure-handling path built across Phase 3 is untested code which runs for the first time during an incident — and that chaos engineering is the practice of running it deliberately instead. The experimental discipline: a steady-state hypothesis, a specific fault, the smallest possible blast radius, and a comparison. The prerequisites, stated honestly, because most teams attempt this before they can measure anything. Why latency injection is more revealing than error injection — systems generally handle errors and hang on slowness. Blast-radius control along six dimensions, and the automated abort tied to error-budget burn that is what makes a production experiment defensible rather than reckless. GameDays, which test the humans, the runbooks and the access that automated chaos cannot. Then the payoff pass: a table of the claims made in Phases 2 and 3 and the specific experiment that proves each one, because a circuit breaker that has never opened is a hypothesis and a fallback that has never run is a comment.
Every Failure Path Is Untested Code
Consider what Phase 3 built: circuit breakers, bulkheads, fallbacks, retry budgets, deadline propagation, load shedding, failover, degraded modes. Now ask a question about each one:
When did that code last execute in production?
For most systems the answer is "during the last incident," or "never." Which means the mechanisms you are relying on to survive a failure are the mechanisms least likely to work, because they have never run, and they will first run at the moment you can least afford a surprise. A fallback with a null-pointer error in it is worse than no fallback, because it creates the belief that you are protected (Module A-7).
Chaos engineering is the practice of making those paths execute on purpose, when you are watching, with a bounded blast radius. It is not "break things randomly." The distinction is that it is an experiment, and an experiment has a hypothesis.
The four steps:
Define steady state as a measurable property of the system — an SLI, not an intuition. "99% of checkout requests succeed within 400 ms" (Module O-2).
Hypothesise that steady state holds under a specific fault: "if the recommendations service returns 500s for all requests, checkout's success rate and latency remain within their SLO."
Inject the fault in the smallest blast radius that can produce a signal.
Compare. If steady state held, you have evidence rather than hope. If it broke, you have found a real weakness without a customer incident — which is the entire value proposition.
Two framings worth keeping. The goal is confidence, not breakage: you are buying evidence about the system's behaviour under conditions you cannot otherwise observe. And an experiment that finds nothing is a success — it is a validated claim, and treating "nothing happened" as a wasted afternoon is how the practice gets abandoned.
Analogy: a fire drill. Nobody runs one because they want a fire; they run one because the alarm, the routes, the muster point and the person with the register are all untested assumptions, and the day of an actual fire is the worst possible time to discover the rear door is chained shut. A drill has a hypothesis ("the building empties in four minutes"), a bounded scope (one floor, announced, at 10 a.m.), an abort condition, and a debrief. And crucially: a drill that reveals the chained door was a good drill, and the person who reports it must not be the person who gets blamed.
Prerequisites, Stated Honestly
Most teams attempt chaos too early, and the result is outages with no learning. You need four things first:
1. Observability (Module O-1). You cannot run an experiment you cannot measure. If you have no degraded-serve-rate metric, injecting a dependency failure teaches you nothing — the fallback might be working perfectly or not at all, and both look identical on an error-rate dashboard.
2. A defined steady state (Module O-2). Without an SLI, "did it still work?" is a matter of opinion, and there is no automated abort criterion.
3. A way to stop the experiment, instantly and reliably. A fault you cannot remove is an incident you caused.
4. Recovery you have already exercised manually. Do not discover your failover procedure inside an experiment; the experiment should validate a procedure you believe in, not substitute for writing one (Module O-7).
If you lack the first two, build them first. Chaos on an unobservable system is just an outage with extra steps.
Fault Types, and Why Latency Beats Errors
Fault
Validates
Bug it typically finds
Latency injection
timeouts, bulkheads, deadlines (A-6, A-7)
a missing timeout; a shared pool exhausted by one slow dependency
Error injection (5xx at a rate)
retries, breakers, fallbacks
breaker thresholds wrong; a fallback that throws
Dependency unavailable
degradation (A-7)
a fallback that depends on the failed thing
Instance / pod termination
statelessness, draining (F-9, A-5)
no drain delay → 502s on every deploy
Resource exhaustion (CPU/memory/disk)
limits, OOM behaviour
an unbounded queue or cache
Network partition
split-brain handling (A-1)
two primaries; no witness
Packet loss / DNS failure
discovery, retries (A-5)
a connection pool holding a stale address
Clock skew
leases, tokens, ordering (A-1)
a lease evaluated against a wall clock; JWTs rejected as not-yet-valid
Certificate expiry
renewal automation
the failure that takes someone down every year
Latency injection is the highest-value fault, and it is worth saying why. Most systems handle errors tolerably — an exception propagates, a retry fires, a fallback runs. Almost no system handles slowness well, because slowness is not an event: nothing throws, nothing is caught, and the effects accumulate. Adding 500 ms to one dependency reveals, in one experiment, whether you have a total timeout (Module A-6), whether that dependency has its own concurrency bulkhead (Module A-7), and whether the request path holds resources while it waits (Module P-4's pool). A dependency returning errors is a rehearsed scenario; a dependency that is merely slow is the one that takes systems down.
The clock skew row is worth a note too: it is easy to inject, almost never tested, and Module A-1 spent a section on why its failures are silent — a lease that is extended or expired by a clock adjustment, tokens rejected as not-yet-valid, ordering inverted between regions.
Blast-Radius Control, and the Automated Abort
This is the discipline that separates an experiment from an incident. Six dimensions to bound:
Dimension
Start
Escalate to
Environment
staging
production
Scope
one instance
one AZ, then a region
Traffic
1% (or one test tenant)
5%, 25%, all
Duration
30 seconds
minutes
Timing
low-traffic hours, team present
any time, unannounced
Fault intensity
100 ms latency
5 s, or full unavailability
Move along one dimension at a time, and only after the previous step held.
The automated abort is mandatory, and it is what makes a production experiment defensible. The experiment must halt itself the moment steady state degrades — driven by the SLO burn rate (Module O-2), not by a human noticing a dashboard:
text
Without this, you are relying on an engineer's attention as a safety control during the one activity where you have deliberately introduced a fault. With it, the worst case is a bounded, brief, self-limiting degradation — which is a defensible risk to take against the value of knowing whether your resilience works.
Three timing rules that are not negotiable. Never run an experiment during an active incident, during a launch or a change freeze, or when the error budget is already exhausted — because the budget is literally the currency you are spending, and spending it deliberately when you have none left is the case the policy exists to prevent (Module O-2).
Announce, then stop announcing. Early experiments should be announced, so that a surprise does not become an incident response and so the team learns the tooling. Mature practice runs some experiments unannounced, because that is the only way to test detection, paging, escalation and the on-call response — which is usually where the real weakness is.
GameDays: Testing the Humans
Automated chaos tests the system. A GameDay is a scheduled exercise where a team responds to an injected or simulated failure end to end, and it tests things no automated experiment can:
Is there a runbook, and is it correct? Most are stale within a quarter (Module O-7).
Are the dashboards useful under pressure, or is the relevant graph buried three clicks deep?
Does the escalation path work at 3 a.m. on a Sunday, including for the person who is new?
Does everyone have the access they need, or does the fix require a credential only one person has?
The single-point-of-knowledge problem: if one named individual is required, that is a finding worth more than any code change.
Format: state the scenario and the hypothesis, inject, let the on-call respond without shortcuts, observe without helping, and hold a blameless review afterwards. The blamelessness is load-bearing rather than cultural decoration: an exercise where a discovered weakness reflects badly on somebody is an exercise nobody volunteers for again, and the practice dies within two rounds.
Proving the Claims: an Experiment per Mechanism
Module O-1 listed the metric that makes each mechanism visible. This is the companion list: the experiment that proves each mechanism works. Every row is a claim made earlier in this course that is a hypothesis until tested.
Claim (module)
Experiment
What it usually finds
The breaker opens under failure (A-7)
make a dependency return 100% errors
thresholds too high; 4xx counted as failures
The breaker closes again (A-7)
restore the dependency, watch it recover
half-open floods and reopens; never recovers
The fallback works (A-7)
make the dependency unavailable
the fallback throws, or hits the thing that failed
Losing the cache is survivable (A-7, P-12)
kill Redis
95% hit ratio becomes 20× database load → second outage
Retries are bounded (A-6)
inject errors and measure offered load downstream
3× or 27× amplification; no retry budget
Deadlines propagate (A-6)
add latency at depth, watch upstream give up
downstream keeps working on abandoned requests
Shedding fires before collapse (A-8)
drive load past the knee
unbounded queues; timeouts instead of 503s
The system recovers by itself (A-8)
overload, then remove the load
metastable — needs a human to drain queues
Readiness does not amplify (A-5)
make the database briefly unavailable
every instance fails readiness → total outage
Draining works (A-5)
terminate an instance under load
502s because nothing waits for deregistration
Failover happens within RTO (O-7, P-5)
kill the primary, time it
replica never promoted; app never reconnects
Two regions cannot split-brain (A-9, A-1)
partition the regions
both promote; no witness exists
Idempotency prevents double charges (P-16)
duplicate and delay requests
key minted per attempt; read-then-write race
Fan-out lag is bounded (A-14)
pause fan-out workers, resume
no alert on queue age; feed silently hours behind
Leases survive clock adjustment (A-1, A-2)
skew a node's clock
lock held by two owners; lease expires under a live holder
Two rows deserve special mention because they are the highest-value first experiments for most teams.
Kill Redis. It is fast, reversible, and it tests Module A-7's most common broken fallback: a cache with a high hit ratio failing does not merely remove the cache, it redirects the entire hit rate onto the database. Most teams discover here that their degraded path is a second outage.
Make the database briefly unavailable and watch your readiness probes. Module A-5 described the anti-pattern — a readiness check that queries a shared dependency fails on every instance simultaneously, removing every backend and turning a two-second blip into a total outage. It is trivially testable, and finding it in an experiment is enormously cheaper than finding it during a failover.
How to Inject: Application, Platform, or Managed
Application-level. A fault-injection middleware, ideally triggered by a header so it can be targeted at a single request path:
ts
Precise, expressive, and it requires code — which means it can drift, and it must be impossible to enable accidentally in production for real users.
Platform-level. A service mesh can inject latency and aborts declaratively (Module A-5), which is genuinely one of a mesh's better features: no application change, applied uniformly, targetable by route and percentage. Below that, tc/netem for packet loss and latency, container termination for instance faults, and toxiproxy — which is the practical recommendation for local and CI use, because it sits between your app and a dependency and can add latency, drop connections or slow a stream with no code changes at all.
Managed platforms (AWS FIS, Gremlin, Chaos Mesh, LitmusChaos) provide the experiment framework, scheduling, blast-radius controls and — importantly — the abort integration, which is the part you would otherwise build.
The trade: application-level injection expresses business-meaningful faults ("the payment provider is slow specifically on captures"), while platform-level injection needs no code and cannot be forgotten in a release. Use both.
Anti-Patterns, and the Maturity Ladder
Anti-patterns, each of which discredits the practice:
Chaos without observability — causing outages and learning nothing.
Staging only — proves nothing about production's data volume, traffic mix, configuration or scale (Module O-3's parity limit). Staging is where you learn the tooling.
No hypothesis — a fault whose result you cannot interpret, because you did not say in advance what should have happened.
No abort criterion — relying on human attention as the safety control.
Running against an exhausted error budget.
Blaming someone for a discovered weakness — the fastest way to guarantee there is never another experiment.
The ladder, in order:
Fault injection in CI. Integration tests where a dependency returns errors or hangs, using toxiproxy or a stub. Cheap, deterministic, and it makes the fallback path covered by tests — which is where the null-pointer error in the fallback gets caught, permanently, before it ever matters.
Manual experiments in staging, to learn the tooling and the abort mechanics.
Small, announced, automated production experiments with hard blast-radius limits.
Continuous production experiments, run on a schedule as regression tests for resilience.
Unannounced GameDays, which test detection and human response.
Step 1 is the one most teams skip and the one with the best return: a test asserting "when the recommendations client throws, the page still renders the popular-items list" costs ten minutes and prevents an entire class of Module A-7 failure permanently.
Why this matters in production: the failures chaos engineering prevents are the ones this course has repeatedly described as silent — a fallback that throws, a breaker that never closes, a readiness probe that amplifies, a cache whose loss doubles as a database outage, a system that cannot exit overload without a human, a failover that has never been timed. All of them are cheap to discover deliberately and expensive to discover at 3 a.m. And the practice's own failure mode is equally predictable: it is abandoned after one experiment that caused a real incident because nobody set an abort criterion, or after one that found something embarrassing and became a blame event.
Where This Shows Up in Your Stack
Redis: the highest-value first experiment for most teams — kill it and observe whether the application degrades or falls over. Expect to discover Module A-7's stampede: a 95% hit ratio disappearing means roughly 20× read load arriving at the database, so the fix is a bulkhead plus request coalescing on the degraded path, not merely a try/catch.
PostgreSQL: kill the primary and time the failover against your claimed RTO (Module O-7); use pg_terminate_backend to test how the application handles a killed connection mid-transaction; pause WAL replay on a replica to simulate lag and see whether read-your-own-writes breaks (Modules P-5, P-11); and hold a long transaction open while running a migration to reproduce Module O-4's lock-queueing outage in a controlled setting.
Kubernetes: delete pods to test draining and readiness (Module A-5), drain a node to find the missing PodDisruptionBudget that would have evicted a quorum majority (Module O-3), and cordon a zone to test whether zonal volumes make a StatefulSet pod unschedulable.
Service mesh: declarative latency and abort injection per route and percentage (Module A-5) — no code, uniform, and targetable, which makes it the easiest path to routine platform-level experiments.
toxiproxy: sits between the application and any TCP dependency and adds latency, timeouts, connection resets or bandwidth limits. The right tool for CI and local development, and the mechanism that turns "does the fallback work?" into a unit test.
Node.js: inject event-loop blocking (a deliberate synchronous busy-wait) to prove that CPU-bound work belongs in a worker (Module P-19) and that your health checks fail honestly when the loop is stalled.
Observability: Module O-1's dashboards are the instrument, and an experiment is also a test of them — if you cannot see the fault's effect, the finding is that your instrumentation is insufficient, which is a legitimate and common result.
Summary
Every failure-handling path is untested code that will first execute during an incident, and a fallback with a bug in it is worse than none because it creates false confidence. Chaos engineering makes those paths run on purpose, while you are watching, with a bounded blast radius.
It is an experiment, not breakage: define steady state as a measurable SLI, hypothesise it holds under a specific fault, inject in the smallest useful blast radius, and compare. The goal is confidence, and an experiment that finds nothing is a success — it is a validated claim.
Prerequisites are real: observability (or you learn nothing), a defined steady state (or there is no abort criterion), a reliable way to stop the fault, and recovery you have already exercised manually. Without the first two, chaos is an outage with extra steps.
Latency injection is the highest-value fault, because systems generally handle errors and hang on slowness — nothing throws, nothing is caught, and the effects accumulate. One 500 ms injection tests whether a total timeout exists, whether the dependency has a bulkhead, and whether resources are held while waiting.
Do not skip clock skew (leases, tokens, cross-region ordering) or certificate expiry, both of which fail silently and annually.
Bound the blast radius along six dimensions — environment, scope, traffic share, duration, timing, intensity — moving one at a time. The automated abort on error-budget burn is mandatory and is what makes a production experiment defensible; relying on a human watching a dashboard is not a safety control during an activity that deliberately introduces faults.
Never experiment during an incident, a launch or a freeze, or when the error budget is exhausted — the budget is the currency you are spending.
Announce early experiments, then stop announcing, because detection, paging, escalation and on-call response are usually where the real weakness lives.
GameDays test what automated chaos cannot: whether the runbook exists and is correct, whether the dashboards are usable under pressure, whether escalation works at 3 a.m., whether people have the access they need, and whether one named individual is a single point of knowledge. The blameless review is load-bearing, not decoration — one blame event ends the practice.
Every resilience claim in Phases 2 and 3 has a specific experiment that proves it, and the two with the best return are: kill Redis (which usually reveals that the degraded path is a second outage, since a 95% hit ratio disappearing sends ~20× read load to the database) and make the database briefly unavailable to watch the readiness probes (which reveals whether a shared-dependency health check turns a two-second blip into a total outage).
Inject at three levels: application-level middleware for business-meaningful faults (precise, requires code, must be impossible to enable for real users), platform-level via a mesh or tc/netem (no code, uniform, targetable), and managed platforms that supply scheduling and abort integration. Use both of the first two.
The anti-patterns each discredit the practice: no observability, staging-only, no hypothesis, no abort criterion, running against an exhausted budget, and blaming someone for a finding.
Climb the ladder in order, and do not skip step one: fault injection in CI with toxiproxy or stubs makes fallback paths test-covered permanently — a ten-minute test asserting the page still renders when the recommendations client throws prevents an entire class of failure — then staging, then small announced production experiments, then continuous ones, then unannounced GameDays.
The next module, Disaster Recovery: RTO and RPO, deals with the failures that are too large for any of this to absorb — a region lost, a database deleted, a ransomware event, a bad migration that corrupted data three hours ago. It covers backup tiers, point-in-time recovery, the standby spectrum, immutable and air-gapped copies, and the rule that governs the whole subject: an unrestored backup is not a backup.
Knowledge Check
A team's first chaos experiment makes their Redis cache return errors for all requests, expecting a modest latency increase since every read has a database fallback. Within forty seconds the database is saturated and the experiment is aborted manually after user-visible errors. What did the experiment successfully discover, and what does the module say about how it was conducted?
A platform team wants to validate resilience but has no SLOs, and their only dashboards are aggregate error rate and CPU. They propose beginning with weekly randomised pod terminations in production. What does the module advise?
After six months of automated experiments — latency injection, instance termination, dependency failures — a company has an unplanned incident in which the on-call engineer takes 50 minutes to escalate, cannot find the runbook, and eventually needs a colleague on holiday because only they have the required access. Every automated experiment had passed. What does the module say this reveals?
Test your knowledge with more question sets
Sign in to access a wider variety of questions and get notified when new practice sets are added to this module.
abort if: error-budget burn rate > 2× for 60s
or p99 latency > SLO threshold for 60s
or the abort switch is thrown by anyone, no questions asked
// Enabled only in specific environments, or for a specific tenant/header.// Business-meaningful faults are possible here and nowhere else.if(chaos.enabled(req))await chaos.maybe(req.header("x-chaos"),{"latency:500":()=>sleep(500),"error:503":()=>{thrownewServiceUnavailable();},"payment:slow":()=>sleep(12_000),// exceed the PSP timeout deliberately});