Module O-2 — SLIs, SLOs, and Error Budgets

The dependency ceiling

Your SLO cannot exceed what your dependencies compose to, unless you engineer around them. Module A-4's arithmetic:

Four hard dependencies at 99.9% each, in series:
    0.999⁴ ≈ 99.6%   →  a 99.95% SLO is not achievable, whatever you do to your own code

So a 99.99% target on top of a managed database with a 99.9% SLA requires either multi-region redundancy (Module A-9) or degradation that lets you serve while the dependency is unavailable (Module A-7). Which produces the strongest architectural argument for graceful degradation in this whole course:

A hard dependency's SLO caps yours. A soft, degradable dependency does not. Every dependency you can serve without is a dependency removed from the multiplication.

That is the sentence to bring to a design review when someone proposes a synchronous call to a new service on the critical path. The decision is not "is this service reliable" but "does adding it to the product of my dependencies still leave me inside my promise."

Choose a rolling window, typically 28 or 30 days, rather than a calendar month. A calendar window has a reset cliff: a bad first week poisons the whole month regardless of subsequent behaviour, and the first of the month grants a false fresh start on the day after an incident. A rolling window means the budget recovers gradually and continuously, which is both fairer and better behaved as a trigger.

---

The Error Budget Is Permission to Take Risk

error budget = (1 − SLO) × valid events
99.9% over 30 days  →  0.1% of requests, or ~43 minutes of total unavailability

The reframing that makes this useful is not that failure is allowed. It is that the budget is a quantity of risk you are entitled to spend:

• Budget remaining? Ship faster. Deploy more often. Run the experiment. Migrate the database. Do the risky-but-valuable thing, because you have room for it to go wrong.
• Budget exhausted? Stop feature work and spend on reliability, until it recovers.

This converts "should we prioritise reliability this quarter?" from an argument into a policy triggered by a number — and it works in both directions, which is the part that makes engineers trust it. The budget does not only restrain shipping; it authorises it.

The policy must be agreed in advance and must have consequences. Write down what actually happens when the budget is gone: non-essential deploys freeze, the next sprint is reliability-only, an escalation to a named person. A budget with no consequence is a dashboard, and a dashboard changes nobody's plan.

And an unused budget is also a signal. Consistent 100% attainment means one of two things, both worth acting on: the SLO is too loose to constrain anything (raise it), or you are over-investing in reliability relative to what users need (spend the surplus on velocity). A team that has never come close to its budget is not necessarily excellent; it may be measuring something that cannot fail, or paying for reliability nobody asked for.

---

Alert on Burn Rate, Not on a Threshold

The naive alert — "error rate above 1% for 5 minutes" — fails in two opposite directions:

It fires on things that do not matter. A 90-second spike to 3% consumes about 0.2% of a monthly budget. Paging a human at 3 a.m. for that is how a team learns to ignore its pager.

It misses things that do. A steady 0.5% error rate never crosses a 1% threshold, and over three days it consumes an entire 99.9% monthly budget. The alert never fires; the SLO is gone.

The fix is to alert on how fast you are consuming the budget:

burn rate = observed error rate / budget error rate
 
burn rate  1  → you will exhaust the budget exactly at the end of the window
burn rate  2  → exhausted in half the window
burn rate 14.4 → exhausted in ~2 days

Then use multiple windows at multiple burn rates, so alert urgency matches consequence:

Two details make this work in practice. The short window is a confirmation, requiring the burn to be ongoing now — without it, an alert based on a one-hour window keeps firing for an hour after a spike has ended, which is the second-largest source of alert fatigue after cause-based alerting. And fast burn pages while slow burn tickets, because a slow burn is real and does not need anyone woken up.

The property this buys is the important one: alert volume scales with consequence rather than with anomaly count. You are paged when your promise is genuinely at risk, and not otherwise.

Symptoms page; causes ticket

• Page on symptoms: an SLI burning fast. That is user impact, and it is bounded by your handful of SLOs.
• Ticket on causes: disk 80% full, a certificate expiring in 14 days, a circuit breaker open, replication lag rising, a dead-letter queue non-empty (Module O-1's table).

The reason is arithmetic. There are a hundred distinct ways to be slow, so cause-based paging multiplies without limit and most of those pages have no user impact. Symptom-based paging is capped by the number of SLOs and is user-impacting by construction.

And every page must be actionable and rare. If a page arrives that a human cannot act on, delete it or convert it to a ticket. Alert fatigue is not a discipline problem to be solved by exhorting people to care; it is a design problem caused by too many alerts, and the only fix is fewer. A page should also carry what it needs to be acted on: the user impact, the current burn rate, and a link to the runbook (Module O-7).

---

SLOs for Things That Are Not HTTP Requests

Most SLO writing assumes a request/response service, which leaves the majority of a real system unmeasured. The formulations:

The pattern for anything asynchronous is that the SLI is a freshness or a latency-to-start, not an availability — because these systems rarely "fail," they fall behind. And the corresponding budget is intuitive: "we may be more than two minutes stale for up to 7 hours a month."

Reporting honestly matters as much as measuring. Publish attainment over time, budget remaining, and — the most useful view — the top consumers of budget, so an incident review has the arithmetic in front of it. And keep the definition stable: retroactively reclassifying events as invalid to rescue an SLO destroys the only property that made the number worth having.

Why this matters in production: the two failure modes of reliability practice are both cultural and both caused by the same technical mistake. A team with cause-based threshold alerts gets paged constantly for things with no user impact, learns to acknowledge without reading, and misses the real incident inside the noise. A team with no quality or freshness SLI has an error rate that stays flat while the product degrades, and finds out from customers. Multi-window burn-rate alerts on a small number of user-facing SLIs — including a degraded-serve SLI and a freshness SLI — fix both, and the work is a day of PromQL rather than a reorganisation.

---

Where This Shows Up in Your Stack

• Prometheus: express SLIs as recording rules producing good and valid counters, then compute burn rate as a ratio over multiple windows. The practical trap: histogram bucket boundaries must include your SLO threshold — a 300 ms latency SLO requires a 300 ms bucket boundary, or the proportion under 300 ms is not computable and you will be interpolating (Module O-1). Choose buckets when you choose the SLO, not afterwards.
• SLO tooling (Sloth, OpenSLO, or your vendor's): generates the multi-window burn-rate rules from a declarative SLO definition, which is worth adopting because hand-written burn-rate expressions are easy to get subtly wrong and hard to review.
• Load balancer / gateway metrics: the default measurement point, since they see queueing, TLS and retries and are not affected by an instance being unable to report on itself.
• Synthetic checks per region (Module A-9), because a dead region is a modest dip in a global average and a total outage for the users in it.
• Real-user monitoring: the honest number, and the only place you learn that your 200 ms server-side p99 is a 900 ms experience on a mid-range Android phone (Module F-5).
• The underlying signals come from Module O-1's table — pool wait time, queue age, replication and CDC lag, breaker state, degraded-serve rate. An SLO is a promise expressed over those; the promise is only as honest as the measurement beneath it.

---

Summary

• 100% is the wrong target — unachievable, and each nine costs roughly ten times the last with no ceiling — and no target is worse, because it makes every reliability-versus-features decision a contest of insistence rather than arithmetic.
• SLI is a measurement, SLO an internal target, SLA a contract. Keep the SLA looser than the SLO with real margin, or every internal miss becomes a legal event and you have no room to learn.
• Express an SLI as good events over valid events. Define "good" to include latency, and decide the "valid" exclusions before the incident — excluding 429s from your own limiter, or an inconvenient region, is self-deception, and retroactively redefining validity destroys the only property that made the number useful.
• State latency as a proportion within a threshold, not as a percentile. A proportion aggregates by addition across instances, regions and windows (percentiles cannot be averaged at all), converts directly into a budget, and composes across multiple thresholds.
• Add a quality SLI on the degraded-serve rate. Once graceful degradation exists, the error rate stops reflecting whether the product works — this is what makes a three-week-old open breaker visible.
• Measure at the load balancer by default (it includes queueing, TLS and retries), supplemented by real-user monitoring for the honest number and synthetic probes per region. And keep the number of SLOs small — a handful, on critical journeys; an SLO per endpoint is unmanageable and produces unactionable alerts.
• Set the target from measurement, not aspiration, and note that beyond roughly 99.99% the user cannot perceive the difference because their own network is less reliable than your service.
• The dependency ceiling caps what you can promise: four hard dependencies at 99.9% compose to 99.6%, so a tighter SLO requires redundancy or degradation. Hence the strongest architectural argument for degradation in this course — a hard dependency's SLO caps yours; a soft, degradable one does not — which is the question to ask of any proposed synchronous call on the critical path.
• Use a rolling 28–30 day window, not a calendar month, which has a reset cliff and grants a false fresh start the day after an incident.
• The error budget is permission to take risk, not an allowance for failure. Budget remaining means ship, experiment, migrate. Budget exhausted means stop feature work. It authorises as much as it restrains, which is why engineers trust it — and the policy must have consequences written down in advance, or it is a dashboard.
• An unused budget is a signal too: consistent 100% attainment means the SLO is too loose or you are over-investing in reliability relative to what users need.
• Threshold alerts fail in both directions — they page for a 90-second spike consuming 0.2% of the budget, and they never fire for a steady 0.5% error rate that consumes the whole budget in three days.
• Alert on burn rate over multiple windows: 14.4× over an hour and 6× over six hours page; 3× over a day and 1× over three days ticket. Require a short-window confirmation so an alert stops when the burn stops, and page only on fast burn. The property this buys is that alert volume scales with consequence rather than anomaly count.
• Page on symptoms, ticket on causes, because there are a hundred ways to be slow (so cause-based paging multiplies without limit) while symptom-based paging is capped by your handful of SLOs and is user-impacting by construction. Every page must be actionable and rare, and carry impact, burn rate and a runbook link — alert fatigue is a design problem fixed by fewer alerts, not by exhortation.
• Asynchronous systems need freshness and latency-to-start SLIs, not availability, because they do not fail — they fall behind. CDC lag, queue age, batch completion by deadline, fan-out latency, per-endpoint webhook delivery, reconciliation completeness, answer groundedness.
• Report attainment, budget remaining, and the top consumers of budget, so an incident review starts with arithmetic rather than impressions.

The next module, Infrastructure as Code, moves from measuring the system to building it reproducibly. It covers Terraform state and the drift that makes a plan lie, immutable infrastructure as the alternative to configuration management, and the one Kubernetes question that stateful system design forces — what a StatefulSet gives a database that a Deployment cannot.

---