Compliance, Privacy, and Data Sovereignty30 min read
Module O-9·30 min read
GDPR, HIPAA, CCPA, and DPDP as design constraints; PII tokenisation and vaulting; and crypto-shredding as the answer to deletion over an immutable log.
Module O-9 — Compliance, Privacy, and Data Sovereignty
What this module covers: Privacy regulation reduced to the four engineering capabilities it actually demands — know where personal data is, limit where it goes, retrieve or delete it everywhere on request, and prove all three. Why none of that can be bolted on: an architecture either supports the obligations or it does not. The unglamorous prerequisite of data inventory and classification, including the places unmapped copies actually live. Minimisation and purpose limitation as design rules, with consent modelled as data that pipelines query rather than a banner. Tokenisation and vaulting as the same scope-collapsing move Module P-24 made with card numbers. Pseudonymisation versus anonymisation, and why the first is still personal data. Residency, where the usual violation is your telemetry pipeline and where "access is processing" makes an engineer's debugging session a cross-border transfer. And then the problem six earlier modules deferred to this one: how a deletion request reaches an immutable backup, an append-only log and a warehouse — for which the answer is to delete a key rather than a record.
Four Engineering Capabilities, Not a Paperwork Exercise
Regulations differ in wording and overlap heavily in what they require you to be able to do:
Know where personal data is — every store, pipeline, index, cache, log and third party.
Limit where it goes — residency, minimisation, purpose limitation, retention.
Act on a subject's request — access, portability, rectification and erasure, reaching every copy, within a legal deadline.
Prove it — an audit trail of access and of the actions you took.
Each of those is an engineering capability with an architectural precondition, which is why compliance cannot be added at the end:
A system that copied personal data into eleven places without recording that it did cannot satisfy an erasure request, regardless of how much legal effort is applied afterwards.
The practical consequence is that the cheap moment to build these capabilities is when the data model is designed, and the expensive moment is when the first request arrives.
Analogy: a library's obligation to remove a book from circulation. If there is a catalogue recording every copy, every branch, every reading list and every scan, removal is a procedure. If copies were lent freely, photocopied, and shipped between branches without record, the library cannot honestly claim it has removed the book — and the failure is not one of will or of legal advice, it is that nobody wrote down where the copies went. Notice also that "we shredded the copy on the shelf" is not compliance if the microfilm archive is unchanged.
The Regimes, in Engineering Terms
Not legal advice — the point is what each one asks you to build.
Regime
Distinctive engineering demand
GDPR / UK GDPR
a lawful basis recorded per purpose; minimisation and storage limitation; subject requests (access, erasure, portability, rectification) within ~1 month; breach notification in 72 hours; processor agreements; a transfer mechanism for data leaving the EEA
CCPA / CPRA
opt-out of sale/sharing, honoured including the Global Privacy Control browser signal — a header your code must respect; deletion and disclosure on request
HIPAA
PHI safeguards, business-associate agreements, audit of who viewed a record, and "minimum necessary" access
DPDP (India)
consent notice and withdrawal machinery, data-fiduciary duties, breach notification
PCI DSS
scope minimisation is the whole game (Module P-24) — never touch a card number and most requirements do not apply to you
Two demands from that table are easy to miss and are pure engineering. The GPC signal means a browser header must change what your backend records, which is code rather than policy. And auditing reads, not just writes, is a requirement in health contexts and good practice everywhere — "who looked at this customer's record" is the question that gets asked, and Module A-13's audit log is where it is answered.
Inventory and Classification: the Prerequisite Nobody Enjoys
You cannot do any of the four capabilities without knowing what you hold and where it flows. That means two artefacts, both maintained:
A field-level classification. Which columns are personal data, which are special category (health, biometrics, religion, sexual orientation, trade union membership — stricter handling), and which are neither. Make it machine-readable and enforce it:
sql
Then a CI check that any new column has a classification, so the inventory cannot silently rot — the same chokepoint argument as Module A-13 and Module O-3, applied to metadata.
A data-flow map. Where each classified field goes, which is a longer list than teams expect:
Destination
Usually forgotten?
Primary database and its replicas
no
Backups and snapshots (Module O-7)
yes — and they are immutable by design
The warehouse and its derived tables (P-22)
sometimes
Search index (P-20), vector store (P-21)
yes
Caches (P-12)
yes
Event log / Kafka topics (P-15)
yes — and it is append-only
CDC pipelines and their sinks (P-23)
yes
Application logs, traces and prompt logs (O-1, A-12)
The rows in bold are where the unmapped copies live. Logs are the most common: a request body in a log line is a PII store, replicated to a vendor, retained for months, and often shipped to a single region regardless of residency rules.
And two classifications that surprise engineers. Pseudonymous identifiers — device IDs, cookie IDs, IP addresses — are generally personal data under GDPR, so "we only store a device ID" is not an exemption. And an embedding is derived personal data: a vector computed from a person's text is not anonymous, since text can be partially reconstructed from embeddings, which means Module P-21's vector store is in scope for both residency and erasure.
Minimisation, Purpose, and Consent as Data
The cheapest compliance strategy is not collecting the data. Each step down this ladder removes obligations wholesale:
Do not collect it.
If you must collect it, do not store it (use it in the request and discard).
If you must store it, store less of it — a hash where you only need equality, an age band rather than a date of birth, a truncated IP, a token rather than a card number.
If you must store it, do not keep it — a retention policy that actually deletes, enforced by a job rather than by intention.
Concretely, four rules that remove most incidental exposure: do not log request bodies, allowlist the fields you log rather than denylisting the ones you redact (Module O-1), strip EXIF from uploads (Module A-11), and separate identity from behaviour so analytics can work on a pseudonymous key.
Purpose limitation has a specific engineering consequence. Data collected for billing cannot be repurposed for ad targeting without a basis — which means the purpose must be recorded alongside the data, and pipelines must be able to filter by purpose. That is what turns consent from a banner into an enforced property.
So model consent as data that systems query:
sql
Append-only, so the history of what someone agreed to and when is reconstructable — the same immutability argument as Module P-24's ledger, for the same reason: you will be asked to prove it.
And withdrawal must propagate. A revoked consent has to reach the warehouse (so a segment stops including them), the email platform, the ad platform, the recommendation model's training set, and any partner you shared with. If revocation only updates a row in your database, you have built a preference centre rather than a consent system.
Tokenisation and Vaulting: Collapse the Scope
The move that does the most work, and it is the same one Module P-24 made with card numbers: replace the sensitive value with a token, keep the mapping in one vault.
text
Now the great majority of your systems handle tokens, and the compliance surface is one vault instead of eleven stores. Access to real values is a narrow, audited, revocable path rather than an ambient property of the platform.
Two variants with real trade-offs. Format-preserving tokens let legacy systems and validation logic keep working. Deterministic tokens (the same input always yields the same token) preserve the ability to join across systems — at the cost that they are vulnerable to frequency analysis and correlation, so they are pseudonymisation, not anonymisation.
Pseudonymisation is not anonymisation
This distinction decides whether data is in scope at all:
Pseudonymised — re-identifiable with additional information (a token, a hashed email, an account ID). Still personal data. Still subject to erasure, residency and access rules.
Anonymised — not re-identifiable by any reasonable means. Out of scope, which is why it is attractive, and genuinely hard to achieve.
The hardness is worth being honest about. Aggregates leak: a count of one identifies a person; a unique combination of quasi-identifiers (postcode, birth year, job title) identifies a person even without a name; and repeated queries against a filtered dataset can isolate an individual. The practical controls are small-cell suppression (do not report groups below a threshold), generalisation of quasi-identifiers, and — where the stakes justify it — differential privacy, which adds calibrated noise so that any individual's presence cannot be inferred. A "we removed the names" dataset is pseudonymised, not anonymised, and treating it otherwise is the standard way this goes wrong.
Residency and Sovereignty
Requirements come in three escalating shapes, and the difference matters enormously:
Requirement
Means
Difficulty
Storage in-region
data at rest stays in the jurisdiction
moderate
Processing in-region
computation happens there too — including support and engineering access
hard
No access by a foreign entity
affects provider choice and key custody
hardest
The architectural consequences are Module A-9's, and they are structural: jurisdiction becomes the partition key, so there is no global table; a cross-region secondary index may itself be a violation; the disaster-recovery site must stay in-jurisdiction, which means residency and DR are designed together (Module O-7); analytics pipelines that ingest everything leak by design unless partitioned or tokenised before the data crosses; and — the one that catches nearly everyone — the telemetry pipeline is the usual violation, because logs and traces contain IPs, user IDs, emails in URLs and request bodies, and are shipped to one central region by default.
Two further items that sit outside the diagram:
Subprocessors. Your compliance is bounded by your vendors'. That requires a maintained list of every SaaS tool that touches customer data, where each stores it, and a processing agreement with each. The support desk and the analytics tool are usually the surprises.
Access is processing. An engineer in another country querying production to debug an issue is a cross-border transfer of the records they saw. Which produces three concrete engineering requirements: restrict production data access to a small audited group, provide masked or synthetic data for debugging so the ordinary path does not touch real records, and log access — Module A-13's audit-of-reads, which is also what makes a breach's scope determinable.
Deletion: the Problem Six Modules Deferred to Here
An erasure request must reach every copy. The full list, from the flow map: the primary database, its replicas, backups and snapshots, the warehouse and derived tables, the search index, the vector store, caches, the append-only event log, every CDC sink, logs, third parties, and any model trained on the data.
Three of those cannot be handled by deleting a record:
Immutable backups are immutable deliberately — that is the ransomware protection from Module O-7, and editing them to remove a subject would destroy the property that makes them worth having.
An append-only log (Module P-15) cannot be rewritten. Compaction with a tombstone removes a key eventually and only for keyed topics, and retention is a window rather than a deletion mechanism.
Object storage with versioning keeps prior versions when you "delete" an object, and object lock actively prevents deletion until the retention period expires (Module A-11). Both are correct security controls that conflict with erasure.
And chasing copies is unbounded in principle: every new sink adds another place to reach, and you must be able to enumerate them all.
Crypto-shredding
The mechanism that resolves all three, and the answer this course has referred forward to five times:
text
This is envelope encryption (Module A-13) used as a deletion primitive, and its properties are what make it the standard answer: erasure is a single operation with a bounded, verifiable effect, it works on media you cannot modify, and it does not compromise the immutability that protects you against ransomware.
The requirements and caveats, which must be designed in rather than discovered:
The key vault becomes critical infrastructure with its own availability and recovery story (Module O-7) — and a subtlety that matters: the vault's own backups must not resurrect deleted keys. A key restored from last week's backup un-deletes the subject's data everywhere. Key deletion must be propagated to, or excluded from, the vault's backups deliberately.
There must be no plaintext copies anywhere. The discipline is upstream: if a log line, a warehouse column or an analytics event holds the plaintext, shredding the key does not reach it. This is why the flow map and the classification exist.
It does not help for data you must retain anyway (see below).
Derived aggregates may still identify — a count of one, or a model that memorised a training example. Shredding the source does not un-train a model, which is an argument for keeping personal data out of training sets or for training on tokenised inputs.
Three companion practices. Soft deletes break propagation — if the application never issues a real DELETE, the CDC stream carries an UPDATE setting deleted_at and downstream consumers, unless written to interpret it, retain the data (Module P-23). Tombstones plus read-time filtering is the correct fast path for materialised views (Module A-14's lesson: filter at read rather than correcting millions of copies), with the actual key deletion behind it. And erasure must be verifiable and bounded in time — you have about a month, and you will be asked to demonstrate it happened, which means the deletion is itself an audited event.
Statutory retention beats erasure, and only partly
Module P-24 raised this conflict: a customer requests erasure of data you are legally required to keep for seven years as a financial record. These do not cancel out, and the resolution is a scoping exercise:
Financial-record obligations override erasure for the transaction data itself — amounts, dates, references, the ledger.
They do not extend to everything else — marketing preferences, device fingerprints, support-chat history, behavioural analytics.
Which produces a day-one design rule: keep the ledger free of personal data beyond a subject key, hold the personal data in a separately-encrypted store, and satisfy erasure by shredding that store's key. The transaction record survives, unreadable as to identity but intact as to amounts and dates. Designing this at the start is nearly free; retrofitting it means rewriting an append-only ledger, which its own immutability forbids.
Access Requests, Breaches, and Proving It
A subject access or portability request is the inventory problem in reverse: assemble everything you hold about one person, in a portable format, within the deadline. A team that has the flow map can automate it as a query per store; a team without one performs archaeology, and the first such request costs weeks. Build the export path when you build the second store, not when the request arrives.
Breach notification in 72 hours is an engineering requirement disguised as a legal one, because the clock starts when you become aware and the notification must state what was exposed and whose data it was. Answering that requires the classification (which fields), the flow map (which systems), and the audit log of access (which records were actually read). A team without those cannot scope a breach, and an unscoped breach must be reported at its worst plausible extent.
And prove it, throughout. An append-only, tamper-evident audit log of security- and privacy-relevant events — who accessed what, on whose behalf, and which erasures and exports were performed (Module A-13). An audit log that a compromised account can edit proves nothing, which is why write-once storage or a hash chain matters here specifically.
Why this matters in production: the compliance failures that happen are engineering failures. Request bodies in logs shipped to a vendor in another jurisdiction. A warehouse ingesting everything, so erasure never reaches it. A search index still serving a deleted person's documents. Embeddings computed from personal text, treated as anonymous. A soft delete that downstream consumers never interpreted. A "we removed names" dataset shared with a partner and re-identified from three quasi-identifiers. A key vault backup that resurrected shredded keys. And a first subject-access request that took three engineers two weeks because nobody knew where the copies were. Each is prevented by an artefact — a classification, a flow map, a token vault, a per-subject key — that costs little to establish early and cannot be recovered cheaply later.
Where This Shows Up in Your Stack
PostgreSQL: column-level classification in comments plus a CI check; row-level security for access control (Module A-13); envelope-encrypted fields for special-category data; partition by jurisdiction where residency applies. And know the crucial limitation: a DELETE does not remove the row from the WAL, the archive, or any backup, so DELETE alone is not erasure — which is exactly why crypto-shredding exists.
Kafka: retention is a window, not deletion, and compaction with a tombstone only helps for keyed topics and only when compaction runs. Treat topics carrying personal data as ciphertext with a per-subject key, and shred (Module P-15).
Object storage: versioning means a "delete" retains prior versions, and object lock prevents deletion by design (Module A-11) — both correct controls that make record-level erasure impossible, and both fully compatible with crypto-shredding.
Warehouse: the largest unmapped copy in most companies (Module P-22). Either tokenise before load (an ETL rather than ELT decision, Module P-23), partition by jurisdiction, or accept that erasure must reach it — for which a table format with row-level deletes (Iceberg, Delta) is the mechanism.
Search and vector stores: deleting a document is straightforward; remembering that the store exists is the hard part. An embedding is derived personal data (Modules P-20, P-21), so vectors are in scope for both residency and erasure.
Logs, traces and prompt logs: allowlist logged fields, scrub at the collector, keep retention short, and regionalise the pipeline — because this is the most common residency violation and the most common unmapped PII store (Modules O-1, A-12). LLM prompt logs contain both user input and retrieved documents.
Third parties: maintain the subprocessor list, hold processing agreements, and know where each vendor stores data. Support desks and analytics tools are the usual surprises.
KMS and the key vault: the foundation of crypto-shredding, with its own DR story — and a backup policy that does not resurrect deleted keys (Module O-7).
Summary
Regulation reduces to four engineering capabilities: know where personal data is, limit where it goes, act on subject requests everywhere within a deadline, and prove all of it. None can be bolted on — a system that copied data to eleven places without recording it cannot honestly satisfy an erasure request.
Two demands are pure code: honouring a Global Privacy Control browser signal, and auditing reads rather than only writes, because "who viewed this record" is the question that gets asked.
Classification and a data-flow map are the prerequisite. Make classification machine-readable and enforce it in CI, and map every destination — the ones usually forgotten being backups, caches, search and vector stores, the append-only event log, ML training sets, and above all logs, traces and prompt logs.
Pseudonymous identifiers (device IDs, cookies, IPs) are generally personal data, and an embedding is derived personal data, so a vector store is in scope for residency and erasure.
Minimise down the ladder: do not collect, do not store, store less (hash, age band, truncated IP, token), do not keep. Four rules remove most incidental exposure: no request bodies in logs, allowlist logged fields, strip EXIF, and separate identity from behaviour.
Purpose limitation means the purpose must be recorded with the data and pipelines must filter by it — that is what makes consent enforced rather than decorative. Model consent as append-only data that systems query, and make withdrawal propagate to the warehouse, the email and ad platforms, training sets and partners; otherwise you have a preference centre.
Tokenise and vault: replace sensitive values with tokens so most systems hold non-sensitive data and the compliance surface is one audited vault — the same scope collapse as never touching a card number. Format-preserving tokens keep legacy validation working; deterministic tokens allow joins at the cost of frequency-analysis correlation.
Pseudonymised data is still personal data; anonymised data is out of scope and genuinely hard. Aggregates leak — a count of one, a unique combination of quasi-identifiers, repeated filtered queries — so use small-cell suppression, generalisation, and differential privacy where the stakes justify it. "We removed the names" is pseudonymisation.
Residency comes in three shapes: storage in-region (moderate), processing in-region including support and engineering access (hard), and no foreign-entity access affecting provider and key custody (hardest). Consequences: jurisdiction as partition key, cross-region indexes possibly in scope, the DR site must stay in-jurisdiction, analytics pipelines leaking by design, and the telemetry pipeline as the usual violation.
Access is processing: an engineer debugging with production data in another country is a cross-border transfer — so restrict access, provide masked or synthetic data for debugging, and log reads. Maintain a subprocessor list, because your compliance is bounded by your vendors'.
Deletion cannot be done by deleting records in three places by design: immutable backups (the ransomware protection), an append-only log, and versioned or object-locked storage. Crypto-shredding resolves all three: encrypt each subject's personal fields with a per-subject data key and delete the key, making every copy everywhere unreadable with no rewrite anywhere.
Crypto-shredding's requirements: the key vault becomes critical infrastructure with its own DR — and its backups must not resurrect deleted keys; there must be no plaintext copies anywhere, which is why the classification and flow map exist; it does not cover data you must legally retain; and it does not un-train a model, so keep personal data out of training sets or train on tokenised input.
Soft deletes break propagation — downstream consumers see an UPDATE and keep the data — and tombstones plus read-time filtering is the right fast path for materialised views, with the key deletion behind it. Erasure must be verifiable, audited, and completed within the deadline.
Statutory retention overrides erasure for financial records and nothing more. So on day one: keep the ledger free of personal data beyond a subject key, hold personal data in a separately-encrypted store, and shred that key — the transaction survives, the identity does not. Retrofitting this means rewriting an append-only ledger, which its immutability forbids.
A subject access request is the inventory problem in reverse, cheap with a flow map and a multi-week archaeology project without one. Breach notification in 72 hours is an engineering requirement: scoping "what was exposed and whose" needs the classification, the flow map and the audit log of reads — and an unscoped breach must be reported at its worst plausible extent.
The next module is the final capstone: the system design interview. It closes the course by turning everything in it into a repeatable 45-minute framework, then works three designs end to end — chat with presence, ride-hailing, and video streaming — each chosen because it exercises a different half of the curriculum, with the trade-offs an interviewer actually probes.
Knowledge Check
A company receives its first erasure request. They delete the user's rows from Postgres and confirm completion. An audit later finds the person's data still present in: nightly backups retained for a year, a Kafka topic with 30-day retention, the analytics warehouse, the search index, and application logs held for 90 days by a third-party vendor. What does the module prescribe?
A team must keep EU customer data in the EU. They partition their database by jurisdiction, host EU tenants in an EU region, and set their disaster-recovery target to a US region for cost reasons. Their observability stack ships all logs and traces to a single US-hosted vendor. An auditor raises two findings. What are they?
A payments product receives an erasure request from a customer whose transactions it must retain for seven years under financial-record rules. The engineering lead proposes anonymising the ledger rows by replacing the customer ID with a random value. What does the module say?
Test your knowledge with more question sets
Sign in to access a wider variety of questions and get notified when new practice sets are added to this module.
Application, warehouse, logs, search, caches → tok_9f86d0818
The vault (strict access, full audit, encrypted) → tok_9f86d0818 = "alice@example.com"
Per subject: DEK (data encryption key), unique
stored in a key vault, itself encrypted by a KMS-held KEK
Every record containing that subject's personal fields stores CIPHERTEXT,
in every system — database, warehouse, Kafka, logs, backups, search index.
To erase: DELETE THE SUBJECT'S DEK.
Every copy, everywhere, becomes permanently unreadable —
with no rewrite of any store, log or backup.