What this module covers: Why a distributed lock is a lease rather than a mutex, and why no lock with a timeout can guarantee mutual exclusion — a garbage-collection pause is indistinguishable from a dead process, so the holder can wake up past its expiry and keep writing. The fencing token as the only actual safety mechanism, and the uncomfortable requirement it carries: the resource has to check it, which most third-party resources will not. Redlock, the critique of it, and the classification that resolves the argument — efficiency locks versus correctness locks. What real lock services (ZooKeeper, etcd) give you that a Redis key does not: consensus, sessions, and monotonic tokens for free. The implementation details that turn a working lock into an incident: releasing someone else's lock, renewing an expired one, and a session-scoped advisory lock leaking through a connection pooler. Then the most useful section — the six ways to design the lock away entirely, because the best distributed lock is the one you did not need.
A Distributed Lock Is a Lease, and That Changes Everything
An in-process mutex has a property you rely on without noticing: the holder and the lock live or die together. If the thread holding it crashes, the process crashes, and the lock is irrelevant. The runtime knows, with certainty, whether the holder is alive.
Across a network, none of that holds. The lock service and the holder are separate processes, so the service must decide "is the holder still alive?" using the only tool available — a timeout (Module P-10). Which means every distributed lock is really a lease: exclusive use for a bounded period, after which the service reclaims it whether or not the holder has finished.
Two consequences follow directly, and both are safety issues rather than inconveniences:
The lease can expire while the holder still believes it holds the lock. Nothing informs the holder. It carries on working.
The holder can be paused, partitioned, or descheduled and be unable to tell the difference between "0.2 seconds passed" and "40 seconds passed."
Here is the failure in full, and it is not exotic:
text
No component behaved incorrectly. The lock service honoured its contract. Client A never received an error. And the invariant the lock existed to protect is broken.
This is worth internalising as a general statement: no lock with a timeout can guarantee mutual exclusion, because guaranteeing it would require knowing that the holder is not merely slow — which Module A-1 established you cannot know. Longer leases do not fix it; they make the window rarer and the recovery from a genuinely dead holder slower, which is a trade between two bad outcomes rather than a solution.
Analogy: a hotel key card programmed to stop working at 11 a.m. on checkout day. The lock on the door is honest about the time, but it has no idea whether you are still inside. If you fell asleep, the card stops working while you are in the room, and the next guest's card opens the door — and you are both in there. The card system cannot fix this. What fixes it is the room refusing to accept anything from a guest whose booking number is older than the current occupant's, which is a change to the room, not to the card.
Fencing Tokens Are the Actual Safety Mechanism
The fix is not a better lock. It is to make the protected resource reject work from a stale holder.
A fencing token is a monotonically increasing number issued with each lock grant. The holder carries it into every operation on the resource, and the resource remembers the highest token it has seen and refuses anything lower.
text
The overlap still happens. What changes is that it is now harmless, because the second writer is fenced out at the resource. Note also that this converts a silent correctness failure into a visible error on client A — which is exactly the trade you want.
Implemented against your own database, a fencing token is a column and a predicate — and it is the same compare-and-set shape as Module P-24's state machine and Module P-17's version guard:
sql
And here is the crux, the reason most "distributed lock" implementations are unsafe: the resource has to participate. You can fence:
your own database rows (a version, epoch or fence column plus a guarded update),
a service you control (accept the token as a parameter and validate it),
Kafka (the leader epoch is precisely this, rejecting writes from a zombie leader),
object storage if it offers conditional writes (S3 now supports If-None-Match and conditional puts, which makes some previously-unfenceable patterns fenceable — worth checking, because this changed relatively recently).
You cannot fence:
a third-party payment API that has never heard of your token,
an email or SMS provider,
a filesystem or legacy service with no conditional-write primitive,
anything where the effect is external and irreversible.
For the second list, a lock cannot deliver correctness at all, and the honest alternatives are the ones from Module P-16: make the operation idempotent, give the downstream a key it will deduplicate on, or accept the possibility and reconcile.
Redlock, the Critique, and the Classification That Settles It
Redlock is Redis's proposed algorithm for locks across N independent Redis masters (typically 5, with no replication between them). Acquire the same key with SET NX PX on each; if you obtain a majority within a small fraction of the TTL, you hold the lock, with a validity period of TTL − elapsed − clock drift allowance.
Martin Kleppmann's critique is the standard reference and its objections are structural rather than nitpicks:
It uses timing assumptions to guarantee a safety property. Its correctness depends on bounded clock drift and bounded process pauses. Neither is bounded in reality (Module A-1), so the guarantee is conditional on assumptions the environment does not honour.
It provides no fencing token. Even a perfectly-implemented Redlock cannot prevent the GC-pause scenario above, because nothing downstream can tell the stale writer apart.
A restarted node forgets it granted the lock. With appendfsync everysec, a crashed-and-restarted Redis node loses recent writes, so it will happily grant the same lock again — and with enough nodes restarting, two clients hold a majority each.
The cost/benefit is inverted. If you only need efficiency, one Redis instance is sufficient and far simpler. If you need correctness, Redlock does not provide it, so the extra complexity buys nothing you needed.
Salvatore Sanfilippo's response is essentially that Redlock targets the efficiency case with best-effort correctness, and that fencing should be used where available. Which produces the classification that actually resolves the debate, and that you should apply to every lock in your system:
Efficiency lock
Correctness lock
Purpose
avoid doing the same work twice
prevent a wrong outcome
Cost of a double grant
wasted CPU, a duplicated email nobody minds, a thumbnail regenerated
double charge, corrupted state, two leaders
Adequate mechanism
one Redis SET NX EX
a lock is not sufficient at all
What you actually need
nothing more
fencing tokens, or idempotency, or a single-writer design
The whole practical value of this module is that second column's third row. If a double grant would be a correctness failure, do not solve it with a lock. Solve it with a fence at the resource, an idempotency key, or a design where only one writer can exist. A lock is a performance optimisation that happens to look like a safety mechanism, and mistaking one for the other is how systems acquire bugs that appear once a quarter and cannot be reproduced.
Real Lock Services Give You Three Things a Redis Key Does Not
ZooKeeper and etcd are consensus systems (Module A-1) with lock primitives built on top, and the difference is not marketing.
Consensus-backed durability. A grant is committed to a majority before it is acknowledged, so no single node's restart can forget it. Objection 3 above disappears.
Sessions rather than bare TTLs. The client maintains a heartbeat; the service tracks liveness rather than only wall-clock expiry. In ZooKeeper an ephemeral znode vanishes when the session ends, which means a crashed client's lock is released promptly rather than after a full TTL — better detection and a shorter unavailable window than a TTL alone can give you.
A monotonic token, for free. ZooKeeper's zxid (and the sequence number on a sequential znode) and etcd's revision are already monotonically increasing, cluster-wide. You do not have to invent a fencing token; the service hands you one. This is the single strongest argument for using a real lock service over a Redis key, because it makes the correct pattern the easy one.
The canonical etcd shape:
ts
Two details in that snippet matter as much as the acquire. The transaction is a compare-and-swap, not a read-then-write, for the reason Module P-16 gave about claims. And the lost handler exists because a client that learns its session ended must stop working immediately rather than finishing — the one thing a holder can do about the expiry problem, even though it is not a guarantee.
One more classic: the herd effect. If a thousand waiters all watch the lock key, releasing it wakes a thousand clients who all attempt to acquire, and 999 fail. The ZooKeeper recipe is for each waiter to create a sequential node and watch only its immediate predecessor, so a release wakes exactly one client. It is Module P-12's thundering herd in coordination clothing, and the fix has the same shape: do not broadcast an event that only one recipient can act on.
The Implementation Details That Become Incidents
A lock without a TTL is a deadlock waiting for a crash. Same lesson as Module P-19's job lease and Module P-16's idempotency claim: a finally block does not run when a process is killed, so every ownership must expire on its own. A permanent lock on a nightly job is discovered a week later, when someone asks why the reports stopped.
Releasing must verify ownership, or you release someone else's lock.
ts
The broken version produces a lovely cascading failure: A's lease expires, B acquires, A finishes and deletes B's lock, C acquires while B is still working, and now the lock is meaningless for everyone. It is worth checking every lock helper you own for this, because the two-line version is what people write first.
A renewal after expiry is not a renewal. Extending a lease must be conditional on still holding it — the same compare-and-set — and if the conditional extend fails, the correct action is to abandon the work, not to re-acquire and continue as though nothing happened. Re-acquisition means someone else may have already started and partially completed the work.
Measure the lease locally with a monotonic clock. You acquired a 10-second lease; how long have you held it? Not "what does the server's clock say" — Module A-1's rule applies, so use process.hrtime.bigint() and treat the lease as expired well before its nominal end, because network time to the service, your own scheduling delays, and clock error all eat into it. A generous safety margin is the only defence available to the holder.
Most implementations are not reentrant, and lock ordering still matters. Acquiring the same lock twice in a call stack will deadlock against yourself. And a system taking two locks in inconsistent orders will deadlock, distributed-systems-scale, with no detector — a database gives you 40P01 deadlock detection, and etcd does not.
Granularity is a real trade. One coarse lock is simple and serialises everything (a throughput ceiling of one). Per-entity locks give concurrency and multiply the number of things that can be held, leaked, or acquired out of order. Prefer the finest granularity that still protects the invariant, and prefer no lock at all — which is the next section.
Design the Lock Away
Most distributed locks in production exist because a lock was the first mechanism that came to mind, not because coordination was required. Six alternatives, roughly in order of how often they apply:
1. Make the operation idempotent, and concurrency stops mattering. If running it twice is harmless, you do not need to prevent running it twice (Module P-16). This dissolves more locks than every other technique combined.
2. Use a uniqueness constraint as the lock. The database is already a correct, consensus-backed, crash-safe lock manager:
sql
One statement, no TTL to tune, no release to get wrong, and it is transactional. Module P-19's dedupe_key on a scheduled slot is this pattern.
3. Compare-and-set on a version column — optimistic concurrency. Rather than locking a row while you think, read it, compute, and write conditionally on the version you read. Losing means retrying, not blocking, and no lock is held across a network call. This is the mechanism behind Module P-24's state transitions and it composes with fencing naturally, because the version is a fence.
4. Partition the work so a single writer exists by construction. The strongest technique in the list. Assign shards or keys to workers — Kafka's consumer group protocol (Module P-15), consistent hashing (Module P-7), a shard-assignment table — so that only one worker is ever responsible for a given key. There is nothing to lock because there is no contention. This is why Kafka needs no per-message locks and why "one consumer per partition" is an architecture rather than a coincidence.
5. Serialise through a queue keyed per entity. SQS FIFO message groups, or a Kafka partition keyed on the entity ID, guarantee that operations on one entity are processed one at a time by one consumer. You get mutual exclusion as a property of the transport.
6. Use the database's own locks when everyone talks to one database. A "distributed" lock between five app instances that all connect to the same Postgres is not distributed — it is a row lock or an advisory lock, and it is transactionally correct in a way no external lock service can be, because the lock and the data commit together.
sql
And when the goal is a singleton task rather than per-item exclusivity, elect a leader once (etcd, or an advisory lock) and let that instance own the task, instead of taking a lock per item.
If you need to…
Use
Avoid duplicate work, correctness not at stake
one Redis SET NX EX
Prevent a wrong outcome
fencing token at the resource, or idempotency — not a lock
Run a scheduled job once
uniqueness constraint on the scheduled slot (Module P-19)
Mutate one row safely
compare-and-set on a version column
Guarantee one writer per key
partition assignment; queue keyed per entity
Coordinate within one database
advisory lock or SELECT … FOR UPDATE
Elect a leader
etcd / ZooKeeper session, or a session advisory lock
Coordinate across genuinely independent systems
etcd/ZooKeeper plus a fence
Why this matters in production: lock bugs are among the worst to diagnose because they are timing-dependent, low-frequency, and leave no error. The symptoms are two workers having processed the same job, a counter that is occasionally wrong, a duplicated outbound email, or a nightly job that has silently not run for eleven days because a killed process left a lock behind. None of these produce a stack trace, none reproduce under test, and all of them are found by a human noticing an inconsistency. Which is the strongest argument for the sections above: a design where the lock is unnecessary has no such failure mode, and a fenced resource turns the failure into a loud rejection instead of a quiet corruption.
Where This Shows Up in Your Stack
PostgreSQL: advisory locks are excellent within a single database — pg_try_advisory_xact_lock for transaction scope (auto-released at commit, which is what you almost always want) and pg_advisory_lock for session scope. The pooler hazard is severe: with PgBouncer in transaction pooling mode, a session-scoped advisory lock stays attached to a server connection that is then handed to a different client, so locks leak between unrelated requests and can be released by someone else entirely. Use transaction-scoped locks with a pooler, always. SELECT … FOR UPDATE SKIP LOCKED is the lock-free alternative for work distribution (Module P-19) and is usually the better answer than locking. And row locks plus a version column give you both fencing and transactionality in one mechanism.
Redis:SET key token NX PX 10000 is the correct acquire, and release must be the Lua compare-and-delete — a bare DEL deletes whoever's lock is currently there. Remember that Redis failover is not consensus-safe (Module A-1), so an acknowledged lock can be lost on promotion. Use it for efficiency locks and never as the sole mechanism for a correctness lock. Redlock adds complexity that does not change that classification.
etcd / ZooKeeper: the right choice for leader election and for correctness-adjacent coordination, because they give consensus-backed grants, session-based liveness rather than bare TTLs, and a monotonic revision or zxid you can use directly as a fencing token. Use the predecessor-watch recipe to avoid the herd. Handle session loss by stopping work, not by re-acquiring.
Kafka: the best illustration of designing the lock away. Partition assignment means exactly one consumer owns a partition, so per-key mutual exclusion is structural. And the leader epoch is a fencing token in production use: a zombie leader's requests carry a stale epoch and are rejected by followers, which is precisely the mechanism this module argues for.
Node.js: an in-process async-mutex coordinates within one event loop and does nothing across instances — and single-threadedness gives no protection whatsoever between two pods. Also beware that a long synchronous operation blocks your lease renewal timer, so the very work you are protecting can cause your lease to lapse; that is an argument for worker_threads or chunking (Module P-19).
Summary
A distributed lock is a lease, not a mutex. The lock service can only detect liveness with a timeout, so the lease can expire while the holder is still working, and the holder cannot tell a 0.2-second pause from a 40-second one.
No lock with a timeout can guarantee mutual exclusion. A GC pause, VM steal, paused container or 30-second DNS timeout lets the holder wake past its expiry and write alongside the new holder, with every component behaving correctly and no error raised anywhere. Longer leases trade a rarer window for slower recovery from real failures.
The fencing token is the actual safety mechanism: a monotonically increasing number issued with the grant, carried into every operation, with the resource rejecting anything lower than the highest it has seen. The overlap still occurs; it becomes harmless, and the stale writer gets a visible error instead of causing silent corruption.
Fencing requires the resource to participate, which is why most distributed locks are unsafe. You can fence your own database rows, a service you control, Kafka's leader epoch, and object storage with conditional writes. You cannot fence a third-party payment API, an email provider, or any irreversible external effect — for those, the answer is idempotency or reconciliation (Module P-16), not a lock.
Redlock's problems are structural: it uses timing assumptions to guarantee safety, provides no fencing token, and a restarted node forgets it granted a lock. If you need efficiency, one Redis instance suffices; if you need correctness, Redlock does not provide it — so the added complexity buys nothing you needed.
Classify every lock as efficiency or correctness. An efficiency lock prevents wasted work and a single SET NX EX is fine. For a correctness lock, a lock is not sufficient at all — use fencing, idempotency, or a single-writer design. Mistaking one for the other produces bugs that appear quarterly and never reproduce.
Real lock services give three things a Redis key cannot: consensus-backed grants that survive a restart, sessions with heartbeats so a dead client's lock is released promptly rather than after a full TTL, and a monotonic revision or zxid that is your fencing token — which makes the correct pattern the easy one.
The implementation details that become incidents: a lock without a TTL deadlocks on a crash; a bare DEL release deletes whoever's lock is present and cascades into the lock meaning nothing for everyone; a renewal must be conditional on still holding it, and a failed renewal means abandon the work rather than re-acquire; measure the lease locally with a monotonic clock and a generous margin; most locks are not reentrant, and inconsistent lock ordering deadlocks with no detector.
Watch only your predecessor, not the lock key, or releasing wakes a thousand clients so that one can proceed.
Design the lock away, in roughly this order: make the operation idempotent so concurrency stops mattering; use a uniqueness constraint as the claim; compare-and-set on a version column so nothing is held across a network call; partition the work so a single writer exists by construction (the strongest technique — this is why Kafka needs no per-message locks); serialise through a queue keyed per entity; and use the database's own locks when every participant talks to one database, because then the lock and the data commit together. When you need a singleton task, elect one leader rather than taking a lock per item.
With a pooler, use transaction-scoped advisory locks only. A session-scoped lock under PgBouncer transaction pooling stays with a server connection that gets handed to another client, so locks leak between unrelated requests and can be released by someone else.
Lock bugs leave no stack trace. Two workers processing one job, an occasionally wrong counter, a nightly job silently not run for eleven days — all found by a human noticing an inconsistency, which is the strongest argument for designs that need no lock and for fences that turn corruption into a loud rejection.
The next module, Distributed Transactions, takes the same problem up a level. This module asked how to stop two actors touching one resource; A-3 asks how to make several resources change together when there is no single transaction to hold them — two-phase commit and why its blocking problem is fatal, sagas with compensating actions, and the question sagas are usually taught without: what you do when the compensation itself cannot succeed because the email has already gone out.
Knowledge Check
A team protects a critical "apply monthly interest" job with a Redis lock (SET NX PX 30000), holding it for the duration of the run. Twice in six months, interest has been applied twice to a subset of accounts. Logs show no errors, and the lock helper is correctly written with a Lua compare-and-delete release. What is happening, and what is the fix?
An engineer replaces a Redis lock with pg_advisory_lock('rebuild:' || tenant) for a per-tenant rebuild job, reasoning that Postgres gives consensus-free, transactionally-correct locking. The application connects through PgBouncer in transaction pooling mode. Shortly after deploy, rebuild jobs for unrelated tenants begin interfering with each other, and one job's lock is released by a completely different request. What went wrong?
A team is designing a system where each of 5,000 tenants has a background sync that must never run twice concurrently. Their proposal is a lock service call per tenant per run, with careful lease renewal and fencing tokens. What does the module suggest they consider first, and why?
Test your knowledge with more question sets
Sign in to access a wider variety of questions and get notified when new practice sets are added to this module.
t=0 Client A acquires lock, lease = 10s
t=1 Client A begins work
t=2 ── Client A stalls: a 15s stop-the-world GC pause
(or: VM steal time, a paused container, a swap storm,
a suspended laptop, a 30s DNS timeout inside a library)
t=12 Lease expires. The lock service reclaims it — correctly, by its own rules.
t=13 Client B acquires the lock and begins the same work.
t=17 Client A wakes up. It has no idea any time passed.
It completes its write. TWO CLIENTS HAVE NOW WRITTEN.
t=0 A acquires lock → token 33
t=12 lease expires
t=13 B acquires lock → token 34
t=14 B writes with token 34 → resource records 34, accepts
t=17 A wakes, writes with token 33 → 33 < 34, REJECTED
UPDATE shard_assignments
SET owner = $1, updated_at =now(), fence = $2WHERE shard_id = $3AND fence < $2;-- ← the fence. Zero rows means you are stale. Stop.
// A lease with a TTL, keepalive'd while working; the key is bound to the lease,// so it disappears if the process dies. The revision is your fencing token.const lease =await client.lease(10);// 10s TTLconst lock =await lease.grant();const ok =await client.if('locks/reindex','Create','==',0)// acquire iff absent.then(client.put('locks/reindex').value(myId).lease(lock)).else(client.get('locks/reindex')).commit();const fence = ok.header.revision;// ← carry this into every writelease.on('lost',()=>abortWorkImmediately());// ← and stop when it is gone
// BROKEN: by the time you DEL, your lease may have expired and B may hold it.await redis.del("lock:reindex");// CORRECT: compare-and-delete, atomically, against your own token.constRELEASE=`if redis.call('get', KEYS[1]) == ARGV[1]
then return redis.call('del', KEYS[1]) else return 0 end`;await redis.eval(RELEASE,1,"lock:reindex", myToken);
INSERTINTO daily_report_runs (report_date)VALUES(current_date)ON CONFLICT DO NOTHING RETURNING report_date;-- zero rows = someone else has it
SELECT pg_try_advisory_xact_lock(hashtext('rebuild:tenant:42'));-- released at COMMIT