What this module covers: Why the environment is part of the system's behaviour, and what it means that you cannot rebuild something that only exists as a sequence of clicks. Declarative state and reconciliation, and the difference between a tool that reconciles when you run it and one that reconciles continuously. Terraform state in detail — what it is, why it must be remote and locked, why it contains secrets, and how to split it so one mistake does not reach everything. Then drift: what causes it, why the dangerous moment is the next apply rather than the divergence itself, and the mature policy, which permits emergency console changes rather than pretending they will not happen. Immutable infrastructure, and the property that makes it worth the build pipeline — the deploy path and the disaster-recovery path become the same path. Policy as code, which turns security review into a build failure. And the one Kubernetes question stateful design forces: what a StatefulSet gives a database that a Deployment cannot, along with the more important thing it still does not give you.
The Environment Is Part of the System
A depressing share of production incidents have the same root cause: something was different in production, and nobody knew. A security-group rule, a timeout on a load balancer, an instance type with less memory, a feature flag, an environment variable, a missing index, a different Postgres minor version. The code was identical and tested; the environment was not.
Infrastructure as code exists because of four properties, and only the first is usually cited:
Reproducibility. The same definition produces the same environment, so staging is structurally the same as production.
Review. A change to a firewall rule goes through the same pull request, diff and approval as a change to a function.
Disaster recovery. You cannot rebuild what exists only as a history of console clicks — which makes IaC a prerequisite for Module O-7 rather than a nicety. "How do we recreate this environment in another region?" has an answer or it does not.
Auditability. Who changed the retention policy, when, and why is a git log rather than an archaeology project.
Analogy: a building's plans versus the building itself. You can construct a house without drawings — many were — and everything works until you need a second one, or an insurer asks where the load-bearing walls are, or the building burns down and you must rebuild it exactly. The drawings are not the building; they are the ability to reproduce the building, argue about it before pouring concrete, and prove what was intended. And the crucial failure mode is the same in both fields: someone knocks a wall through during an emergency and never updates the drawings, so the next contractor works from a document that is confidently wrong.
Declarative State and Reconciliation
Imperative infrastructure is a sequence of steps: create the network, then the subnets, then the instances. Scripts of this shape are not idempotent — running one twice creates two of everything, and running one against a partially-built environment fails or duplicates.
Declarative infrastructure describes the desired end state, and a reconciler computes the difference and applies it. Running it twice is a no-op, running it against a half-built environment completes it, and running it after a deletion recreates the missing part.
The distinction that matters between tools:
Reconciles
Consequence
Terraform / CloudFormation / Pulumi
when you run it
drift persists silently between runs
Kubernetes controllers
continuously
a deleted pod is recreated in seconds, unasked
GitOps (Argo, Flux)
continuously, from git
drift is corrected automatically, and that is also a risk
Continuous reconciliation removes the drift problem below and introduces its own: an operator's emergency manual change is reverted within seconds, sometimes mid-incident, by a controller doing exactly its job. Both models require a policy for emergency changes; they just fail differently in its absence.
State: What It Is, and Why It Is the Fragile Part
Terraform's state file is a mapping from the resource addresses in your code to real-world identifiers, plus a cache of their attributes:
text
It exists because the cloud API cannot answer "which of these forty subnets is private[2]?" Identity lives in the state file and nowhere else, which makes the state file the most important and most dangerous artefact in the system.
It must be remote, versioned and locked. Local state means one person can work, the history is on their laptop, and it will eventually be lost or diverge. A remote backend (S3 with DynamoDB locking, GCS, Terraform Cloud) gives shared access, versioning for recovery, and — critically — locking, because two concurrent applies produce a corrupted state that no longer matches reality. Expect the operational consequence: an apply killed mid-run leaves a stale lock, and clearing it requires a force-unlock that everyone is rightly nervous about.
It contains secrets in plaintext. A generated database password, a private key, an access token — any attribute the provider returns is cached in state, whether or not it is marked sensitive in the output. So: encrypt the backend, restrict read access as tightly as production database access, and prefer not to generate secrets in Terraform at all — create them out-of-band in a secrets manager and reference them by ARN or name, so the value never enters state (Module A-13).
Split state by lifecycle and ownership. One state file for everything gives you a slow plan, a lock the whole organisation queues behind, and one blast radius:
text
The cost is cross-state references — remote state data sources, or explicit inputs passed between layers — plus an ordering you now have to know. Worth it, because the alternative is a single apply that can destroy a database while adding a security-group rule.
Refactoring state is where accidents happen. Renaming a resource in code makes Terraform plan a destroy-and-create, because the address changed. Use moved blocks (or state mv) to tell it the resource is the same one; use import to bring an existing resource under management; and treat state rm with real caution, because it makes Terraform forget a resource without deleting it — leaving an unmanaged orphan that still costs money and still serves traffic.
And protect the irreplaceable. A prevent_destroy lifecycle rule on production databases, buckets and anything holding data is a one-line guard against a plan that a tired person approves at 2 a.m.
Drift: the Danger Is the Next Apply, Not the Divergence
Drift is reality diverging from code. The causes, in order of frequency:
A manual console change during an incident. Someone raised a timeout, opened a security group, or scaled a database at 3 a.m. to stop the bleeding. This is not misconduct; it is the correct thing to have done.
Another tool touching the same resource — an autoscaler adjusting capacity, a controller setting an annotation, a second Terraform workspace overlapping.
Provider or cloud defaults changing between versions, so a field you never set now has a different value.
Cloud-managed mutations — a managed service rotating something, adding a tag, or resizing storage.
The dangerous part is not the divergence. It is what happens next:
A later, unrelated apply silently reverts the emergency fix. Someone adds a tag, the plan includes "restore security group to declared state," and the mitigation that has been holding production together for two days disappears — as a side effect of a change nobody connected to it.
The second failure mode is a plan showing unexpected destroys, which a tired operator approves because the plan output is long and the important line is in the middle.
The fix is continuous visibility, not prohibition. Telling people never to touch the console fails, because during a severe incident the console is faster than a pull request and stopping the bleeding is correct. So:
Run a scheduled plan in CI (hourly or daily) and alert on any non-empty diff. Drift then has a mean time to detection of an hour rather than "until the next unrelated apply."
Adopt a break-glass procedure: emergency manual changes are permitted, are logged, automatically create a ticket, and must be codified within an agreed window — a day or two. The change becomes legitimate rather than hidden.
Use ignore_changes deliberately for fields another system genuinely owns (an autoscaler's desired count, a tag applied by a cost tool), which converts permanent noise into a documented boundary.
That policy is the mature position: assume drift will happen, make it visible immediately, and have a defined path back into code.
Immutable Infrastructure: Replace, Do Not Mutate
Two philosophies for getting software onto a machine:
Configuration management (Ansible, Chef, Puppet) converges a long-lived server towards a desired configuration. It works, and it accumulates history: a server that has been converged for three years has been touched by four generations of playbooks, a manual fix nobody documented, and a package installed for a debugging session in 2024. It becomes a snowflake, and the symptom is "it works on server 3."
Immutable infrastructure never modifies a running machine. You build an artefact — a container image or a machine image — and to change anything you build a new one and replace the instance.
What it buys:
No configuration drift between instances, because none of them is ever modified.
Rollback is redeploying the previous artefact, which is a known-good state rather than a reverse migration.
The build is testable: the artefact that passed CI is the artefact that runs, byte-identical.
And the property that matters most: the deploy path and the disaster-recovery path become the same path. Rebuilding an environment from scratch is what you do every day, so it works when you need it in an emergency (Module O-7). Recovery procedures that are only exercised during disasters do not work.
The costs are real: a build pipeline and image build time on every change, image storage and sprawl (with a lifecycle policy needed, or the registry bill grows quietly), and a longer path from a one-line config fix to production. And the significant exception: stateful components cannot simply be replaced, because the state has to survive. That exception is the whole of the next section.
The test for whether you have achieved it: can you terminate any instance right now, at random, with no coordination? If the answer is no for a stateless service, something is a pet. Module O-6 turns that question into a practice.
Environments, Configuration, and Honest Limits
Same code, different variables. Environments are the same modules instantiated with different inputs — not copy-pasted directories, which diverge within weeks and then differ in ways nobody can enumerate. The divergence is itself the incident cause the module opened with.
Separate config from secrets from code. Configuration in the repository, secrets by reference to a manager (never in a .tfvars committed to git, and remember that a committed secret must be rotated rather than deleted — Module A-13).
Ephemeral preview environments are the payoff. Once an environment is a function of code plus variables, a per-pull-request environment becomes possible, which is the largest practical benefit most teams get from IaC.
And be honest about the limit: IaC gives you structural parity, not behavioural parity. Staging has the same resources and not the same data volume, traffic shape, cardinality, cache warmth, or concurrency. A query that is fast on 10,000 rows is not fast on 400 million (Module P-22); a lock that never contends with three users contends with three thousand. Staging catches configuration and integration errors. It does not catch scale, which is what Module O-5 exists for.
Policy as Code: Security That Fails the Build
Module A-13 argued that a check each engineer must remember will be forgotten. The infrastructure version of the chokepoint is policy as code: rules evaluated automatically on every plan, in CI, blocking the merge.
text
Tools: Open Policy Agent/Conftest, Sentinel, tfsec, checkov, or a cloud provider's own guardrails. The value is not that these rules are clever; it is that they are enforced identically on every change, including the one made at 3 a.m. by someone who has been awake for nineteen hours. Security review by human attention does not scale; a failing pipeline does.
Note the two rows that other modules depend on: mandatory tags are what make per-service cost attribution possible at all (Module O-8), and mandatory backups plus deletion protection are what make an RPO claim true (Module O-7). Policy as code is where those become structural rather than aspirational.
Kubernetes: What a StatefulSet Gives a Database That a Deployment Cannot
This course treats Kubernetes as out of scope as a subject — but stateful system design forces exactly one question, so here it is.
A Deployment treats pods as interchangeable: random name suffixes, no ordering guarantees, replacement in any sequence, and either ephemeral storage or a volume shared by all replicas. That is correct and desirable for stateless services (Module F-9) — it is precisely what makes them replaceable.
Every one of those properties is wrong for a database, a broker, or a consensus member. A StatefulSet provides four things instead:
1. Stable network identity. Pods are db-0, db-1, db-2, and with a headless service each is addressable at db-0.db.namespace.svc.cluster.local — permanently. That identity is what a replication topology needs (a replica must know which host is its upstream, Module P-5) and what a quorum member needs (Raft membership is a list of specific peers, Module A-1). A Deployment's random, changing pod names cannot express either.
2. Stable per-pod storage. Each pod gets its own PersistentVolumeClaim that follows that ordinal across reschedules, so db-0's data is still db-0's data after it is rescheduled onto another node. A Deployment cannot express "each replica has its own durable disk."
3. Ordered startup and shutdown. Pods start 0, 1, 2 and terminate in reverse. This matters because a follower must not start before the primary it replicates from, and because removing quorum members abruptly and simultaneously is how you lose a majority (Module A-1's membership warning).
4. Ordered, controlled rolling updates, highest ordinal first, one at a time, with a readiness gate between — so an upgrade does not restart every broker at once.
And now the more important point, which is the one people miss:
A StatefulSet solves identity and storage. It does not solve consensus, leadership, or failover.
Kubernetes does not know which pod is the primary, will not promote a replica, and cannot tell a healthy standby from a stale one. db-0 being ordinal zero does not make it the leader. So running Postgres on a StatefulSet still requires Patroni plus a DCS (etcd/Consul) to hold the leader election (Module A-1), or an operator that encodes the same logic. Without one, you have durable storage and stable names attached to a database that nobody promotes when the primary dies.
The costs, which are worth stating because they surprise people:
Rollouts are slow by design (ordered, one at a time, with readiness gates).
Scale-down retains PVCs deliberately, so shrinking a StatefulSet leaves orphaned volumes that still cost money until someone deletes them — a recurring quiet line item.
Zonal volumes pin pods to zones. A cloud block device usually exists in one availability zone, so the pod bound to it can only be scheduled in that zone. If the zone fails, that pod cannot be rescheduled anywhere — which is a real availability limit that a "highly available Kubernetes cluster" does not remove.
PodDisruptionBudgets are mandatory, or a routine node drain evicts two of three quorum members at once.
The honest framing to end on: running a database on Kubernetes is a decision to operate a database. A managed database is usually the better trade, and the reasons to choose otherwise — a specific extension, a cost model, a regulatory constraint, an unusual topology — should be nameable. If they cannot be named, take the managed service.
Why this matters in production: the IaC failures that happen are drift reverting an emergency mitigation during an unrelated apply; a state rm that orphaned a resource nobody could find until the invoice arrived; a database password sitting in plaintext state in a bucket more people could read than anyone realised; an environment that could not be rebuilt in a second region because half of it was created by hand three years ago; and a StatefulSet-hosted database that came back after a node failure with stable storage, a stable name, and no leader. Each is prevented by something in this module, and each of those preventions is a small amount of setup work done in advance.
Where This Shows Up in Your Stack
Terraform: remote backend with locking from day one; state encrypted and access-restricted like production credentials; state split by lifecycle so a data-layer change and an app-layer change cannot share a blast radius; prevent_destroy on databases and buckets; moved blocks rather than rename-and-recreate; a scheduled plan in CI with drift alerting; and policy as code blocking merges. Prefer generating secrets outside Terraform so they never enter state.
PostgreSQL: manage the instance as code (parameter groups, versions, backup retention, deletion protection) so an RPO claim is enforced rather than assumed (Module O-7). Rotate database credentials out-of-band, precisely because anything Terraform generates is cached in state. And treat a major-version upgrade as an application-level migration with a rehearsal, not a Terraform attribute change.
Container images: the immutable artefact, tagged by content or commit rather than latest — a mutable tag reintroduces "which version is actually running?" and breaks rollback. Set a registry lifecycle policy, or image storage grows without anyone noticing.
Kubernetes: Deployments for stateless work, StatefulSets when identity and per-pod storage are required, PodDisruptionBudgets on anything with a quorum, and an operator or Patroni for the leadership decision a StatefulSet does not make. Remember zonal volumes constrain scheduling to one zone.
GitOps (Argo/Flux): continuous reconciliation, which removes silent drift and reverts manual changes within seconds — so the break-glass procedure becomes more important, not less, and needs a documented way to pause reconciliation during an incident.
Secrets managers: referenced from IaC by name, with dynamic short-lived credentials where available (Module A-13), so the infrastructure definition contains pointers rather than values.
Summary
A large share of incidents are "something was different in production and nobody knew." IaC exists for reproducibility, reviewable change, disaster recovery — you cannot rebuild what exists only as console clicks, which makes this a prerequisite for Module O-7 — and auditability.
Declarative beats imperative because reconciliation is idempotent: running it twice is a no-op and running it against a half-built environment completes it. But note the difference between reconciling when you run it (Terraform) and continuously (Kubernetes, GitOps) — the latter removes silent drift and reverts an operator's emergency change within seconds, so both models need an emergency-change policy.
State maps code addresses to real resource IDs, and identity lives nowhere else. It must be remote, versioned and locked, because concurrent applies corrupt it; expect stale locks after a killed apply.
State contains secrets in plaintext, so encrypt it, restrict it like production credentials, and prefer to create secrets outside Terraform and reference them, so the value never enters state.
Split state by lifecycle and ownership — network, data, platform, app-per-environment — so one apply cannot destroy a database while adding a tag. The cost is cross-state references and a known ordering.
Refactoring state is where accidents happen: a rename plans a destroy-and-create unless you use moved; import brings existing resources under management; and state rm leaves an unmanaged orphan that still costs money and still serves traffic. Put prevent_destroy on everything irreplaceable.
Drift's danger is the next apply, not the divergence — an unrelated change silently reverts the emergency mitigation that has been holding production together for two days, and a long plan hides an unexpected destroy from a tired approver.
The fix is visibility, not prohibition: a scheduled plan in CI with alerting on any diff, plus a break-glass procedure where emergency console changes are permitted, logged, ticketed and codified within an agreed window — because during a severe incident the console is faster than a pull request and stopping the bleeding is correct. Use ignore_changes to document fields another system genuinely owns.
Immutable infrastructure replaces rather than mutates, eliminating snowflake servers, making rollback a redeploy of a known-good artefact, and — the property that matters most — making the deploy path and the disaster-recovery path the same path, so recovery works because you exercise it daily. Costs: a build pipeline, image sprawl needing a lifecycle policy, and a longer path for a one-line fix.
The test is whether you can terminate any instance at random right now. For a stateless service, "no" means something is a pet.
Environments are the same modules with different variables, never copy-pasted directories, which diverge within weeks — and IaC delivers structural parity, not behavioural parity: staging has the same resources and not the same data volume, traffic shape, cardinality or concurrency, which is what Module O-5 exists for.
Policy as code is the infrastructure chokepoint (Module A-13's argument): deny public buckets, unencrypted storage, open security groups and untagged resources on every plan. Its value is uniform enforcement including at 3 a.m. — and note that mandatory tags are what make cost attribution possible (Module O-8) while mandatory backups and deletion protection are what make an RPO claim true (Module O-7).
A StatefulSet gives a database four things a Deployment cannot: stable network identity (db-0.db.ns.svc, which replication topologies and quorum membership require), stable per-pod persistent volumes that follow the ordinal, ordered startup and shutdown, and ordered controlled rolling updates.
But a StatefulSet does not solve consensus, leadership or failover. Kubernetes does not know which pod is primary and will not promote a replica, so Postgres on a StatefulSet still needs Patroni plus etcd, or an operator, to hold the election (Module A-1). Otherwise you have durable storage, stable names, and nobody to promote anything.
StatefulSet costs: slow ordered rollouts, scale-down deliberately retaining PVCs as orphaned billable volumes, zonal volumes pinning a pod to one availability zone so it cannot be rescheduled if that zone fails, and PodDisruptionBudgets being mandatory or a node drain evicts two of three quorum members.
Running a database on Kubernetes is a decision to operate a database. A managed service is usually the better trade, and the reasons to choose otherwise should be nameable.
The next module, Deploying and Migrating Without Downtime, takes the immutable artefact this module produced and asks how to put it into production without users noticing — blue-green and canary, feature flags as the decoupling of deploy from release, expand/contract schema migrations, and the specific reason a multi-terabyte Postgres migration takes an outage, which is lock queueing rather than the rewrite everyone expects.
Knowledge Check
During a severe incident, an engineer widens a security group and raises a load-balancer idle timeout via the cloud console, stabilising the service. Three days later a colleague merges an unrelated change adding tags to some resources, and production breaks again in the same way. What does the module identify, and what is the recommended policy?
A team migrates their Postgres primary and two replicas from managed hosting onto Kubernetes, using a StatefulSet with per-pod PersistentVolumeClaims and a headless service. A node fails; the pod is rescheduled, comes back with its data intact and its stable hostname, and the database remains unavailable for writes until an engineer intervenes manually. Why?
A team keeps all infrastructure in one Terraform state file with a remote backend and locking. They generate database passwords with random_password and pass them to the RDS resource. A security review and a near-miss incident produce two findings: the plan for a routine change included destroying the production database, and the state file was readable by everyone with access to the deployment bucket. What does the module prescribe?
Test your knowledge with more question sets
Sign in to access a wider variety of questions and get notified when new practice sets are added to this module.
network/ rarely changes, everything depends on it
data/ databases, buckets — long-lived, destructive changes forbidden
platform/ clusters, shared services
app-prod/ fast-moving, deployed often
app-staging/
DENY a storage bucket with public access
DENY an unencrypted volume or database
DENY a security group allowing 0.0.0.0/0 on anything but 80/443
DENY a resource without owner and cost-centre tags ← Module O-8 depends on this
WARN an instance type outside the approved set
DENY a database without deletion protection and backups ← Module O-7 depends on this