Module A-2·30 min read

Redlock and its critics, etcd and ZooKeeper, fencing tokens as the actual safety mechanism, and how to design the lock away entirely.

JJS
Written by Jatin Jain Saraf · Senior Software Engineer

Module A-2 — Distributed Locks and Coordination

What this module covers: Why a distributed lock is a lease rather than a mutex, and why no lock with a timeout can guarantee mutual exclusion — a garbage-collection pause is indistinguishable from a dead process, so the holder can wake up past its expiry and keep writing. The fencing token as the only actual safety mechanism, and the uncomfortable requirement it carries: the resource has to check it, which most third-party resources will not. Redlock, the critique of it, and the classification that resolves the argument — efficiency locks versus correctness locks. What real lock services (ZooKeeper, etcd) give you that a Redis key does not: consensus, sessions, and monotonic tokens for free. The implementation details that turn a working lock into an incident: releasing someone else's lock, renewing an expired one, and a session-scoped advisory lock leaking through a connection pooler. Then the most useful section — the six ways to design the lock away entirely, because the best distributed lock is the one you did not need.


A Distributed Lock Is a Lease, and That Changes Everything

An in-process mutex has a property you rely on without noticing: the holder and the lock live or die together. If the thread holding it crashes, the process crashes, and the lock is irrelevant. The runtime knows, with certainty, whether the holder is alive.

Across a network, none of that holds. The lock service and the holder are separate processes, so the service must decide "is the holder still alive?" using the only tool available — a timeout (Module P-10). Which means every distributed lock is really a lease: exclusive use for a bounded period, after which the service reclaims it whether or not the holder has finished.

Two consequences follow directly, and both are safety issues rather than inconveniences:

  1. The lease can expire while the holder still believes it holds the lock. Nothing informs the holder. It carries on working.
  2. The holder can be paused, partitioned, or descheduled and be unable to tell the difference between "0.2 seconds passed" and "40 seconds passed."

Here is the failure in full, and it is not exotic:

text

No component behaved incorrectly. The lock service honoured its contract. Client A never received an error. And the invariant the lock existed to protect is broken.

This is worth internalising as a general statement: no lock with a timeout can guarantee mutual exclusion, because guaranteeing it would require knowing that the holder is not merely slow — which Module A-1 established you cannot know. Longer leases do not fix it; they make the window rarer and the recovery from a genuinely dead holder slower, which is a trade between two bad outcomes rather than a solution.

Analogy: a hotel key card programmed to stop working at 11 a.m. on checkout day. The lock on the door is honest about the time, but it has no idea whether you are still inside. If you fell asleep, the card stops working while you are in the room, and the next guest's card opens the door — and you are both in there. The card system cannot fix this. What fixes it is the room refusing to accept anything from a guest whose booking number is older than the current occupant's, which is a change to the room, not to the card.


Fencing Tokens Are the Actual Safety Mechanism

The fix is not a better lock. It is to make the protected resource reject work from a stale holder.

A fencing token is a monotonically increasing number issued with each lock grant. The holder carries it into every operation on the resource, and the resource remembers the highest token it has seen and refuses anything lower.

text

The overlap still happens. What changes is that it is now harmless, because the second writer is fenced out at the resource. Note also that this converts a silent correctness failure into a visible error on client A — which is exactly the trade you want.

Implemented against your own database, a fencing token is a column and a predicate — and it is the same compare-and-set shape as Module P-24's state machine and Module P-17's version guard:

sql

And here is the crux, the reason most "distributed lock" implementations are unsafe: the resource has to participate. You can fence:

  • your own database rows (a version, epoch or fence column plus a guarded update),
  • a service you control (accept the token as a parameter and validate it),
  • Kafka (the leader epoch is precisely this, rejecting writes from a zombie leader),
  • object storage if it offers conditional writes (S3 now supports If-None-Match and conditional puts, which makes some previously-unfenceable patterns fenceable — worth checking, because this changed relatively recently).

You cannot fence:

  • a third-party payment API that has never heard of your token,
  • an email or SMS provider,
  • a filesystem or legacy service with no conditional-write primitive,
  • anything where the effect is external and irreversible.

For the second list, a lock cannot deliver correctness at all, and the honest alternatives are the ones from Module P-16: make the operation idempotent, give the downstream a key it will deduplicate on, or accept the possibility and reconcile.


Redlock, the Critique, and the Classification That Settles It

Redlock is Redis's proposed algorithm for locks across N independent Redis masters (typically 5, with no replication between them). Acquire the same key with SET NX PX on each; if you obtain a majority within a small fraction of the TTL, you hold the lock, with a validity period of TTL − elapsed − clock drift allowance.

Martin Kleppmann's critique is the standard reference and its objections are structural rather than nitpicks:

  1. It uses timing assumptions to guarantee a safety property. Its correctness depends on bounded clock drift and bounded process pauses. Neither is bounded in reality (Module A-1), so the guarantee is conditional on assumptions the environment does not honour.
  2. It provides no fencing token. Even a perfectly-implemented Redlock cannot prevent the GC-pause scenario above, because nothing downstream can tell the stale writer apart.
  3. A restarted node forgets it granted the lock. With appendfsync everysec, a crashed-and-restarted Redis node loses recent writes, so it will happily grant the same lock again — and with enough nodes restarting, two clients hold a majority each.
  4. The cost/benefit is inverted. If you only need efficiency, one Redis instance is sufficient and far simpler. If you need correctness, Redlock does not provide it, so the extra complexity buys nothing you needed.

Salvatore Sanfilippo's response is essentially that Redlock targets the efficiency case with best-effort correctness, and that fencing should be used where available. Which produces the classification that actually resolves the debate, and that you should apply to every lock in your system:

Efficiency lockCorrectness lock
Purposeavoid doing the same work twiceprevent a wrong outcome
Cost of a double grantwasted CPU, a duplicated email nobody minds, a thumbnail regenerateddouble charge, corrupted state, two leaders
Adequate mechanismone Redis SET NX EXa lock is not sufficient at all
What you actually neednothing morefencing tokens, or idempotency, or a single-writer design

The whole practical value of this module is that second column's third row. If a double grant would be a correctness failure, do not solve it with a lock. Solve it with a fence at the resource, an idempotency key, or a design where only one writer can exist. A lock is a performance optimisation that happens to look like a safety mechanism, and mistaking one for the other is how systems acquire bugs that appear once a quarter and cannot be reproduced.


Real Lock Services Give You Three Things a Redis Key Does Not

ZooKeeper and etcd are consensus systems (Module A-1) with lock primitives built on top, and the difference is not marketing.

Consensus-backed durability. A grant is committed to a majority before it is acknowledged, so no single node's restart can forget it. Objection 3 above disappears.

Sign in to keep reading

The rest of this module is free — sign in with Google to unlock it and track your progress.

Sign in & Register

Discussion

0

Join the discussion

Loading comments...

© 2026 Jatin Jain Saraf (JJS). All rights reserved.