A-6 — The SupraScan Architecture: Coordinating 10+ Concurrent Scanner Instances

The indexer must maintain a global "frontier" — the highest contiguous block number that has been fully indexed. Blocks may be processed out of order (due to parallel consumers), so we need to track which blocks are complete and advance the frontier only when gaps are filled.

const COMPLETED_BLOCKS_KEY = 'indexer:completed:blocks';
const FRONTIER_KEY = 'indexer:frontier';
 
// After successfully indexing a block:
async function markBlockComplete(blockNumber: number) {
  // Add to the sorted set (score = blockNumber)
  await redis.zadd(COMPLETED_BLOCKS_KEY, blockNumber, String(blockNumber));
 
  // Try to advance the frontier
  await advanceFrontier();
}
 
// This is a read-compute-write over two keys — with 10+ workers calling
// markBlockComplete() concurrently, a plain GET-then-SET here is a real race:
// two workers can both read the same starting frontier, compute two different
// "next" values, and whichever SET lands last wins — even if it's the smaller
// value, silently regressing the frontier and losing progress. This is exactly
// the class of bug A-2 (Lua scripting) and A-3 (Redis Functions) exist to
// prevent, so the fix is to make the whole read-compute-write a single atomic
// Lua script instead of separate GET/ZRANGEBYSCORE/SET round-trips:
const ADVANCE_FRONTIER_SCRIPT = `
local frontierKey = KEYS[1]
local completedKey = KEYS[2]
local lookahead = tonumber(ARGV[1])
 
local frontier = tonumber(redis.call('GET', frontierKey) or '0')
local nextBlocks = redis.call('ZRANGEBYSCORE', completedKey,
  frontier + 1, frontier + lookahead, 'LIMIT', 0, lookahead)
 
local newFrontier = frontier
for _, blockStr in ipairs(nextBlocks) do
  local block = tonumber(blockStr)
  if block == newFrontier + 1 then
    newFrontier = block
  else
    break -- gap in sequence — stop advancing
  end
end
 
if newFrontier > frontier then
  redis.call('SET', frontierKey, tostring(newFrontier))
  redis.call('ZREMRANGEBYSCORE', completedKey, 0, newFrontier)
end
 
return newFrontier
`;
 
async function advanceFrontier() {
  // Redis executes the entire script as one atomic step — no other client's
  // commands can interleave between the read and the write, so two workers
  // racing here can no longer regress the frontier.
  const newFrontier = await redis.eval(
    ADVANCE_FRONTIER_SCRIPT,
    2, FRONTIER_KEY, COMPLETED_BLOCKS_KEY,
    10000, // look ahead 10,000 blocks
  );
  logger.info(`Frontier advanced to block ${newFrontier}`);
}

The Sorted Set holds in-flight and recently completed block numbers. The frontier advances when a contiguous run of completed blocks is detected. Completed blocks below the frontier are trimmed to prevent unbounded memory growth.

---

Crash Recovery: Reclaiming Orphaned Locks

When a worker crashes mid-processing, its lock expires (due to TTL). The block range is not marked complete. Another worker must pick it up.

The recovery mechanism: a background "reclaimer" process periodically scans for block ranges that have locks but are not in the completed set and not currently being processed by a healthy worker.

async function reclaimOrphanedRanges() {
  // Scan all block range locks
  const lockKeys: string[] = [];
  let cursor = '0';
  do {
    const [nextCursor, keys] = await redis.scan(cursor, 'MATCH', 'lock:block:range:*', 'COUNT', 100);
    lockKeys.push(...keys);
    cursor = nextCursor;
  } while (cursor !== '0');
 
  for (const lockKey of lockKeys) {
    const lockValue = await redis.get(lockKey);
    if (!lockValue) continue;  // already expired
 
    const ttl = await redis.pttl(lockKey);
    if (ttl > 30000) continue;  // lock is healthy (more than 30s remaining)
 
    // Lock is close to expiry — is the holder still alive?
    const [, workerId] = lockKey.split('lock:block:range:');
    const heartbeatKey = `heartbeat:${lockValue}`;
    const heartbeatExists = await redis.exists(heartbeatKey);
 
    if (!heartbeatExists) {
      // Worker is dead — reclaim this range for reprocessing
      logger.warn(`Reclaiming orphaned lock ${lockKey} (worker ${lockValue} is down)`);
      await redis.del(lockKey);
      // The range will be re-acquired by the next worker that processes it
    }
  }
}

Worker Heartbeats

Each worker publishes a heartbeat key with a short TTL. If the worker crashes, the heartbeat expires and the reclaimer can detect orphaned ranges:

// Each worker: publish heartbeat every 10 seconds
const HEARTBEAT_TTL = 30;  // 30 seconds
const heartbeatKey = `heartbeat:${workerId}`;
 
setInterval(async () => {
  await redis.set(heartbeatKey, Date.now().toString(), 'EX', HEARTBEAT_TTL);
}, 10000);
 
// On clean shutdown: remove heartbeat immediately
process.on('SIGTERM', async () => {
  await redis.del(heartbeatKey);
  process.exit(0);
});

---

The 6-Hour Replication Lag Incident

The most instructive production failure: a Redis replica fell 6 hours behind the primary due to network partition. After the partition healed and the replica caught up, approximately 6 hours of completed-block markers were replayed — causing the reclaimer to see blocks as "not completed" and requeue them for reprocessing.

Root cause: The reclaimer queried a replica (for read scaling), which had stale data. It saw block 500,000 as incomplete (because the completion marker had not yet replicated), even though it was fully indexed in PostgreSQL.

Consequences: Approximately 1M blocks were re-queued for reprocessing. The workers detected duplicate inserts via PostgreSQL's ON CONFLICT DO NOTHING and discarded them, but the extra processing load caused a 2-hour throughput degradation.

Fix:

1. Read coordination state from the primary only. Completion markers, frontier state, and lock state are read from the primary Redis connection. Replica reads are only used for non-critical data (metrics, display data).

2. Idempotent writes to PostgreSQL. All INSERT statements use ON CONFLICT DO NOTHING or ON CONFLICT DO UPDATE. Double processing produces the same result as single processing.

3. Replication lag monitoring. Alert when replica.lag > 5s. A 6-hour lag should have triggered alerts long before it caused data consistency issues.

// Separate client for coordination reads (always primary)
const primaryRedis = new Redis({ host: 'redis-primary.internal' });
const replicaRedis = new Redis({ host: 'redis-replica.internal' });
 
// All coordination reads from primary
const frontier = await primaryRedis.get(FRONTIER_KEY);
const lockExists = await primaryRedis.exists(rangeKey);
 
// Non-critical display data can use replica
const displayMetrics = await replicaRedis.hgetall('indexer:metrics');

---

Memory Management Under Load

At 500–1,000 TPS with 10 workers, Redis memory pressure was a real concern. Key management practices:

Bounded Sorted Set for completed blocks:

// Keep only the last 100,000 completed block numbers in the sorted set
// (frontier advancement removes everything below the frontier anyway)
await redis.zremrangebyrank(COMPLETED_BLOCKS_KEY, 0, -100001);

TTL on all coordination keys:

Every key written by the indexer has a TTL — lock keys (60s), heartbeat keys (30s), completion markers (24h). No orphan keys accumulate indefinitely.

Monitoring key count:

setInterval(async () => {
  const info = await primaryRedis.info('keyspace');
  const dbMatch = info.match(/db0:keys=(\d+)/);
  if (dbMatch) {
    metrics.gauge('redis.key_count', parseInt(dbMatch[1]));
  }
}, 60000);

---

Lessons for Your Own Distributed Systems

1. Read coordination state from the primary. Replica lag causes phantom failures and false recoveries. The performance cost is acceptable for coordination operations.

2. Make all writes idempotent. Double-processing must produce the same result as single-processing. Design your database writes with ON CONFLICT from the start, not as an afterthought.

3. TTL everything. Every Redis key written by background processes must have a TTL. An exception is a memory leak waiting to manifest under production load.

4. Heartbeats are not optional. Without heartbeats, you cannot distinguish a slow worker from a crashed one. Without that distinction, your reclaimer either ignores crashed workers (gap in processing) or aggressively reclaims slow workers (double processing under load).

5. Monitor replication lag as a first-class metric. A large lag is a data consistency crisis in slow motion. Alert at seconds, not minutes.

6. The sorted set frontier pattern scales. Using a Sorted Set to track out-of-order completions and advance a contiguous frontier is a general pattern — applicable to any system processing an ordered stream with parallel workers.

---

Summary

• SupraScan coordinated 10+ concurrent worker instances using Redis block-range locks with heartbeat extension and UUID-based holder identity
• The completed blocks Sorted Set + frontier key pattern tracked out-of-order processing and maintained strict sequential guarantees
• Crash recovery via lock expiry + heartbeat absence detection — a worker with a missing heartbeat is presumed dead; its ranges are reclaimed
• The 6-hour replication lag incident taught the fundamental lesson: always read coordination state from the primary; replica reads are for display data only
• Idempotent database writes are the safety net — double processing should produce the same result as single processing
• TTL every coordination key, monitor key counts, alert on replication lag above 5 seconds

Next: A-7 — Master-Replica Replication: PSYNC, Replication Buffer, and Lag — how Redis propagates writes to replicas, what happens when a replica falls behind, and how to measure and manage replication lag.

---