• Immutability that even you cannot override. Object lock in compliance mode with a retention period, so no credential — including the account root — can shorten or delete the object before it expires (Module A-11). If an administrator can delete it, so can whoever has the administrator's session.
• A separate account or subscription for backup storage, with separate credentials and no trust relationship that lets the production account write or delete there. Blast-radius separation (Module A-13) applied to the one asset whose loss is unrecoverable.
• MFA-delete and versioning as additional layers.
• Retention longer than your plausible detection time. This is the requirement most often set wrong: if data was silently encrypted or corrupted three weeks ago and your retention is fourteen days, every copy you hold is bad. Dwell time for a compromise is commonly measured in weeks, and a slow data-corruption bug can go unnoticed for longer, so retention needs a long tail — a few daily copies plus weekly and monthly ones extending months back.
• Restore tests that use the immutable copy specifically, because that is the path you will actually use in the scenario the copy exists for, and it usually has different access mechanics from the convenient one.

---

The Standby Spectrum

Cost rises steeply and non-linearly along that list, which is why different data classes belong at different tiers — a payment ledger at hot standby and an analytics warehouse at backup-and-restore is a coherent architecture, while treating everything as tier 0 is an unfunded aspiration.

Two structural notes carried from Module A-9. A warm or hot standby only helps if the failover path is exercised, and an unrehearsed standby is a line item rather than a plan. And two sites cannot safely fail over automatically — neither can distinguish "the other is down" from "I am partitioned" — so either add a witness in a third failure domain or accept that the declaration is a human decision, which puts it squarely in the RTO's decide term.

---

An Unrestored Backup Is Not a Backup

This is the rule the entire subject reduces to. A backup job that reports success proves that bytes were written. It does not prove they are readable, complete, restorable with the tools you have, or usable by your application.

Restore tests, automated and scheduled. Monthly at minimum, weekly for tier 0, into a fresh isolated environment, with verification that goes beyond "the restore command exited zero":

• row counts against expected ranges,
• checksums or aggregate comparisons on key tables,
• a smoke test of the application against the restored data — which is the step that catches the problems the database alone cannot show you.

Measure the restore duration, because that is your real RTO. It is almost always longer than assumed, and it is bounded by physics: a 4 TB restore is limited by disk and network throughput, and replaying three days of WAL for a PITR can take hours because replay is largely single-threaded. A team claiming a one-hour RTO whose measured restore takes five hours does not have an RTO, it has a wish.

The discoveries a first restore test makes are remarkably consistent across organisations, and every one of them is cheap to find in a drill and expensive to find in a disaster:

• The backup was of the wrong database, or excluded a schema.
• The encryption key is not available in the recovery region or account — a classic, and total: an encrypted backup without its key is noise.
• The engine version needed to restore is no longer offered by the provider.
• A required extension (pgvector, postgis, pg_trgm) is unavailable in the recovery environment, so the restore fails at index creation.
• Roles and grants were not in the dump, so the restored database has objects nobody can access.
• The restore succeeded and the application could not connect, because an endpoint was hardcoded somewhere.
• The runbook references a person who left, or a tool nobody has installed.
• Nobody had the permissions to perform the restore except one person.

Automate the verification as a scheduled job, so it is a green or red check rather than a quarterly act of heroism. A restore test that depends on someone remembering will stop happening within two quarters.

---

The Runbook, and the Decision Somebody Must Be Authorised to Make

Name who declares a disaster. The decide term dominates RTO, and it is dominated by ambiguity about authority. A named on-call role empowered to declare and act, with a defined escalation, converts forty minutes of "should we?" into two minutes of "we are."

The runbook must contain preconditions, steps, verification and a fail-back plan. Failing back is frequently harder than failing over, because data written at the recovery site during the incident must be reconciled into the primary — and a plan that ends at "we are running in the DR region" leaves the hardest half undocumented.

And the data-loss decision is a real decision, not a technicality. With asynchronous replication, promoting the standby means accepting the loss of everything not yet replicated. Someone must be authorised to make that call, and the correct answer depends on the data class in a way that inverts the usual instinct:

• For a feed or a session store, promote immediately and accept the loss (Module A-14's availability-first posture).
• For a payment ledger, waiting may be correct — Module P-24 put correctness above availability, and losing forty seconds of committed financial transactions can be worse than an extra twenty minutes of downtime. That is a business decision made in advance, written down, not improvised at 3 a.m.

Communication is part of RTO from the customer's perspective. A status page, a defined update cadence, and pre-drafted messages, hosted somewhere that is not on the infrastructure that is down.

---

What Else Must Be Recoverable

The database is the part everyone plans for. The list that ends recoveries:

That last row is worth the emphasis it gets everywhere: keep an offline or independently-hosted copy of the recovery documentation, including credentials access paths, contact numbers and the decision authority.

And your RTO cannot beat your dependencies'. A managed database's own recovery SLA, your provider's regional recovery, a payment processor's outage — these bound what you can promise regardless of your engineering (Module O-2's dependency ceiling, applied to recovery). Shared fate matters too: a backup in the same account, region and provider as the primary shares a class of failure with it, and provider-wide incidents do happen.

---

Testing: the Ladder

1. Automated restore verification — the CI-style job described above. Non-negotiable, and the cheapest.
2. Failover drill in staging — the procedure, end to end, with the runbook in hand.
3. Scheduled production failover — actually run on the real system, announced, in a low-traffic window. This is the step that separates a real DR capability from a documented one, and it is the step most organisations never take.
4. Unannounced drill (Module O-6's GameDay), which tests detection, decision authority and access under realistic conditions.

Why this matters in production: the recurring disaster-recovery failures are not exotic. A team with three replicas and no backups, believing they were covered, until a bad migration corrupted a column. A backup regime that had run successfully for two years and had never been restored, whose first restore failed on a missing extension. A ransomware event that encrypted the backups in the same account. A retention window shorter than the time it took to notice a corruption bug. A four-hour "one hour RTO" because nobody had timed a 4 TB restore. And a recovery that stalled because the KMS key existed only in the region that was gone. Each is prevented by something in this module, and every one of those preventions is exercised rather than documented.

---

Where This Shows Up in Your Stack

• PostgreSQL: physical base backups plus continuous WAL archiving (pgBackRest, WAL-G, Barman, or the provider's equivalent) for PITR, and alert on archive failures and the age of the newest archived WAL segment — a broken archive_command silently ends your recovery window while filling the primary's disk. Keep periodic pg_dump output for portability and single-table restores, remembering that roles need pg_dumpall --globals-only. Verify that the recovery environment has your extensions and a compatible engine version. And repeat the rule: a streaming replica is not a backup, because it applies your DELETE faithfully.
• Object storage: the backup destination, with versioning, object lock in compliance mode, MFA-delete, and a separate account with no write or delete path from production credentials (Modules A-11, A-13). Lifecycle rules for the long tail — daily for a fortnight, weekly for a quarter, monthly for a year — chosen so retention exceeds plausible detection time.
• KMS: replicate or mirror keys into the recovery region and test decryption there, because this is the single most common total-failure discovery in a DR drill.
• Redis: decide per keyspace. A cache is tier 3 — do not back it up, document the rebuild. Sessions, queues, rate-limiter state used for billing, or a presence store may be real data, in which case AOF persistence plus snapshots to a separate account is the answer (Modules P-14, P-18).
• Kafka: retention is not backup — it is a window, and a compacted topic is not an archive. Decide per topic whether it is a source of truth (back it up, or ensure the producer's store is) or a transport for rebuildable data. And note that offsets do not translate across clusters (Module P-23), so a recovery plan must say what consumers do about position.
• IaC and CI/CD: state files, module versions, pipeline definitions and registry images are part of the recovery path (Module O-3). An immutable-infrastructure practice pays off here specifically, because rebuilding is the thing you already do daily.
• Derived stores — search indexes, vector indexes, feed timelines, warehouse tables: document and time the rebuild rather than backing them up (Modules P-20, P-21, P-22, A-14).

---

Summary

• DR is not HA, and replication is not backup — replication copies a DELETE FROM orders to every standby in milliseconds. HA protects against things happening to your infrastructure; DR protects against things happening through it, and the most common causes of catastrophic loss are human error and malice, not hardware.
• RPO is how much data you can lose; RTO is how long you can be down — and RTO = detect + decide + execute + verify, with the first two usually dominating. A five-minute restore behind a forty-minute decision is a forty-five-minute RTO.
• Set both per data class, not once for the company. And know which data is a source of truth and which is derived — feed timelines, search and vector indexes, caches and warehouse tables are rebuildable, so the correct plan is a timed rebuild procedure rather than a backup, and restoring a stale derived copy alongside a current source creates inconsistency nobody expects.
• Logical dumps are portable and slow to restore, and omit roles (pg_dumpall --globals-only is the missing half). Physical backup plus continuous WAL archiving gives point-in-time recovery, which converts "we lost three hours" into "we lost forty seconds" and is what saves you from a bad migration.
• A broken WAL archive silently ends your PITR window while the primary keeps running and retaining WAL locally. Alert on archive success and the age of the newest archived segment.
• Snapshots are crash-consistent rather than application-consistent, and usually share the primary's account and region — hence its blast radius. And know your single-table restore story in advance: PITR restores a whole cluster, so recovering one table means restoring to a side instance and copying.
• 3-2-1-1-0: three copies, two locations, one offsite, one immutable or air-gapped, zero verified errors — because a ransomware operator has your credentials and deletes the backups first. That requires object lock even you cannot override, a separate account with no delete path from production, MFA-delete, and retention longer than your plausible detection time, since a fourteen-day window is useless against a three-week-old corruption.
• Test restores from the immutable copy specifically, because that is the path the scenario will use and its access mechanics differ from the convenient one.
• The standby spectrum runs from backup-and-restore (hours to days, cheapest) through pilot light and warm standby to hot/active-active (~0 RTO, a second full environment), with cost rising steeply — so different data classes belong at different tiers. An unrehearsed standby is a line item, not a plan, and two sites cannot safely fail over automatically without a witness in a third failure domain.
• An unrestored backup is not a backup. Automate scheduled restores into a fresh environment with row counts, checksums and a smoke test of the application against the restored data. Measure the restore duration — that is your real RTO, and it is bounded by throughput and largely single-threaded WAL replay.
• First restore tests reliably discover: the wrong database backed up, the encryption key unavailable in the recovery region, an engine version no longer offered, a missing extension, absent roles and grants, a hardcoded endpoint, a runbook naming someone who left, and one person holding the only sufficient permissions.
• Name who declares a disaster, because ambiguity about authority is what dominates the decide term. The runbook needs preconditions, steps, verification and a fail-back plan, since reconciling data written at the recovery site is usually the harder half.
• The data-loss decision is a business decision made in advance: promote immediately for a feed or session store, but for a payment ledger waiting may be correct, because losing forty seconds of committed financial transactions can be worse than twenty more minutes of downtime.
• Also recoverable, or the recovery ends: encryption keys, IaC state, secrets, DNS control, CI/CD and the registry, the observability stack, and the runbook itself — not hosted on the infrastructure that is down.
• Your RTO cannot beat your dependencies', and a backup sharing the primary's account, region and provider shares a class of failure with it.
• Climb the testing ladder: automated restore verification, a staging failover drill, a scheduled production failover — the step that separates a real capability from a documented one and the one most organisations never take — then an unannounced drill.

The next module, FinOps: Cost Architecture, treats money as a design constraint in the way this module treated recoverability. It covers cost per request and per tenant as first-class metrics, the egress tax that has appeared as a warning in six earlier modules, spot and reserved capacity, storage tiering, and the question worth asking of every scaling decision — whether you are buying capacity or paying to avoid fixing a design flaw.

---