Inside a single process, a lock is reliable. A mutex is held until the holder releases it, and if the process dies, the lock dies with it. Across machines, none of that is true, and the ways it fails are surprising enough that a lot of production code using “distributed locks” is less safe than its authors believe. The problems are worth understanding even if you never write a lock service yourself, because the same reasoning applies to leader election, scheduled jobs and any “only one worker should do this” logic.
Why every distributed lock is really a lease
Suppose worker 1 acquires a lock in Redis or a database row to process a monthly payout file. Then worker 1’s machine loses power. If the lock had no expiry, it would be held forever and no one could ever process the file. So distributed locks almost always come with a time limit: a lease. Worker 1 holds the lock for, say, 30 seconds and must renew it to keep it. If it disappears, the lock frees itself.
Leases fix the crash problem and create a new one. The lock service decides the lease has expired based on its own clock. The worker believes it still holds the lock based on its own view. These two views can disagree.
The paused worker
Here is the scenario that breaks naive locking. Worker 1 acquires the lock with a 30-second lease. It checks that it holds the lock and starts writing the payout file to storage. Just then, its JVM enters a long garbage collection pause, or the virtual machine is suspended by the hypervisor for live migration, or the process is starved of CPU. The pause lasts 40 seconds.
During the pause, the lease expires. Worker 2 acquires the lock, quite correctly, and starts processing the same file. Then worker 1 wakes up. From its perspective no time has passed; it already checked that it held the lock, and it continues writing. Now two workers are writing the same file, and the output is corrupted or payouts are duplicated.
Checking the lock again before each write does not fully help, because a pause can occur between the check and the write. Network delay creates the same effect: a write request sent while the lock was valid can arrive at storage after the lease has expired and someone else holds it. No amount of care on the worker’s side closes this gap, because the worker cannot know it was paused.
Fencing tokens: let the resource check
The fix, described clearly by Martin Kleppmann in his widely read 2016 analysis of distributed locking, moves the check to where it can actually be enforced. Every time the lock service grants the lock, it also returns a fencing token: a number that increases with each grant. Worker 1 gets token 33. When its lease expires and worker 2 acquires the lock, worker 2 gets token 34.
Every write to the protected resource includes the token. The resource remembers the highest token it has seen and rejects any write with a lower one. Worker 2 writes with token 34; storage accepts and records 34. Worker 1 wakes from its pause and writes with token 33; storage rejects it, because 33 is lower than 34. The stale holder is fenced off, no matter how long it slept or how slow the network was.
The idea has a cost: the protected resource must participate. A database can do this with a conditional update that only applies if the stored token is lower than the one supplied. Object storage with conditional writes can sometimes approximate it. An external API that knows nothing about your tokens cannot, and for those you need idempotency keys or other safeguards instead.
Where tokens come from
Lock services built on consensus provide suitable numbers naturally. ZooKeeper assigns increasing transaction ids and sequence numbers to the nodes used for locks. etcd attaches a revision number to every key change, and that revision increases monotonically across the cluster. These systems are designed to stay correct under network partitions and node failures.
A single Redis instance with a key set only if it does not already exist, with an expiry, is a common lightweight lock. It is simple and fast, but provides no fencing token by default and loses lock state if the instance fails over to a replica that had not received the latest write. Redlock, an algorithm that acquires the lock on a majority of independent Redis instances, was proposed to improve this, and Kleppmann’s analysis argued that it still depends on timing assumptions that do not hold under pauses and clock jumps. Redis’s author published a response disputing parts of that critique. The debate is worth reading in full; the practical takeaway most engineers draw is that when correctness truly depends on mutual exclusion, use fencing, whichever lock service you choose.
Efficiency locks versus correctness locks
A helpful question before choosing any design is: what happens if two holders occasionally act at once?
- Efficiency lock. The lock exists to avoid wasted work, such as two servers rebuilding the same cache entry or sending the same daily summary email. A rare double execution costs a little compute or one duplicate email. A simple lease-based lock is fine.
- Correctness lock. Two holders acting at once corrupts data or moves money twice. Here a lease alone is not safe. Use fencing tokens, or design the operation so the database enforces exclusivity directly through a unique constraint, a conditional update, or a single transaction.
Often the best distributed lock is no distributed lock. If every payout record has a status column, a worker can claim a batch with a conditional update from PENDING to PROCESSING; the database guarantees only one worker succeeds. If the work is a queue, a message broker already ensures each message goes to one consumer at a time. Partitioning work by key, so that each key is only ever handled by one consumer, removes the need for coordination entirely.
The general principle is that a process can never be sure it still holds a lock at the moment its action takes effect. Safety has to come from the resource refusing out-of-date actions, not from the actor believing it is allowed.
Written by the RP Strativa Team. Lease durations, pause lengths and token numbers are illustrative.
Leave a Reply