A-5 — Reentrant Locks, Hierarchies, and Deadlock Prevention

Who this module is for: You have basic Redis locking working but are hitting edge cases — a function that holds a lock tries to call another function that also acquires the same lock (deadlock), or you need to acquire multiple locks without racing with another process doing the same. This module covers reentrant locks, lock hierarchies, and the patterns that prevent deadlock at scale.

The Reentrant Lock Problem

A basic Redis lock (SET NX PX) is not reentrant. If a function holds lock:resource and calls a subroutine that also tries to acquire lock:resource, the subroutine blocks forever — it is waiting for itself to release a lock it holds.

In single-threaded Node.js, this is actually a deadlock: the blocking await acquireLock prevents the event loop from running, which means the outer function can never release the lock, which means the inner acquireLock can never unblock.

Reentrant Lock Implementation

A reentrant lock tracks the holder's identity and a reentry count. The same holder can acquire the lock multiple times; it is released only when the reentry count reaches zero.

Use a Redis Hash to store both the holder token and the count:

Lock Hierarchies and Deadlock

Deadlock occurs when two processes each hold a lock the other needs:

The Consistent Ordering Solution

If all processes acquire multiple locks in the same globally consistent order, circular waits are impossible.

Rule: Always acquire locks in ascending order of a canonical key (alphabetical order, numeric ID order, or a predetermined priority order).

Regardless of which account is "from" and which is "to", both processes always acquire account:1001 before account:1002. If Process A and Process B are both transferring between 1001 and 1002, they will queue up — one waits for the other, no circular wait.

The ordering must be globally consistent across all lock sites. A mix of orderings (some code sorts alphabetically, other code sorts by timestamp) defeats the pattern.

Lock Timeout as Deadlock Prevention

A simpler approach: set an acquisition timeout. If you cannot acquire a lock within N milliseconds, fail fast instead of waiting indefinitely.

Lock timeouts convert deadlocks into timeouts — the operation fails with an error instead of hanging forever. This is the correct production behaviour: fail fast and let the caller retry or report an error.

Mutex vs Semaphore

The locks covered so far are mutexes — exclusive access, one holder at a time. Sometimes you need to allow N concurrent holders (e.g., limit to 5 concurrent database connections from a particular service).

A counting semaphore in Redis:

Limitation: The Set-based semaphore has the same GC-pause vulnerability as single-instance locks — a holder that pauses longer than the TTL loses its slot. For strict concurrency limits with crash recovery, combine with periodic TTL renewal.

The Watchdog Pattern for Long-Held Locks

When a lock must be held for an indeterminate duration (e.g., while streaming a large file), use a watchdog that extends the lock while work is ongoing:

The watchdog ensures the lock TTL is continuously renewed while the holder is alive. If the holder crashes, the watchdog stops, the lock expires naturally, and another process can acquire it.

Summary

• Reentrant locks — use a Hash storing holder UUID + reentry count; same holder can reacquire; fully released when count reaches zero
• AsyncLocalStorage in Node.js carries the holder token across async context for reentrant lock identity
• Deadlock from multiple locks — prevented by consistent ordering: always acquire locks sorted by a canonical key across all code paths
• Lock timeouts — set an acquisition timeout to convert potential deadlocks into fast failures
• Mutex vs Semaphore — a counting semaphore (Set-based) allows N concurrent holders for rate-limiting concurrency
• Watchdog pattern — background timer extends lock TTL while holder is alive; lock expires on crash without explicit release

Next: A-6 — The SupraScan Architecture: Coordinating 10+ Concurrent Scanner Instances — the real-world distributed coordination problem from building a production blockchain indexer at 2,000+ TPS.