Module A-10·20 min read

Adding a node to a running cluster, assigning slots, CLUSTER SETSLOT MIGRATING/IMPORTING, the MIGRATE command, zero-downtime resharding with ASKING redirections, and why resharding time scales with key count.

JJS

Written by Jatin Jain Saraf · Senior Software Engineer

A-10 — Resharding, Node Addition, and Live Slot Migration

Q: During a live resharding operation, a Redis Cluster client requests a key that belongs to a hash slot currently migrating from Node A to Node B. The specific key has *already* been migrated to Node B. What sequence of events occurs?

Node A returns an `ASK` error pointing to Node B. The client sends an `ASKING` command to Node B, followed by the original request, but does *not* update its internal slot map for future requests. — This is the core mechanic of zero-downtime migrations. Because a slot migration takes time (keys are moved in batches), ownership of the slot hasn't officially changed yet. Node A is still the owner. Therefore, Node A issues a temporary `ASK` redirect instead of a permanent `MOVED` redirect. The client must ask Node B for this specific key (prefacing the request with `ASKING` to bypass Node B's normal slot-ownership checks), but the client must remember that Node A still owns the slot overall until the migration officially finishes.

Q: When migrating a massive dataset (e.g., 50 million keys) from one Redis Cluster node to another using `redis-cli --cluster reshard`, what is the most significant performance bottleneck you should anticipate?

The CPU and network overhead of serializing, transmitting, and restoring each key individually via the `MIGRATE` command. — Resharding is not instantaneous or free. Unlike adding a replica (which uses a highly optimized bulk RDB transfer), moving slots between primary nodes involves the `MIGRATE` command. `MIGRATE` serializes keys, sends them over the network, restores them on the target, and deletes them from the source. For millions of keys, this process is compute-heavy and network-intensive, often taking hours for large datasets. This is why resharding large clusters should be planned carefully, ideally during off-peak hours.

Q: You want to gracefully remove a primary node (`Node C`) from a running Redis Cluster. What is the correct sequence of operations?

Use `redis-cli --cluster reshard` to move all slots currently owned by Node C to other primary nodes. Once Node C owns 0 slots, use `redis-cli --cluster del-node` to remove it from the cluster topology. — You cannot simply delete a primary node that owns data; doing so would cause the cluster to enter a `fail` state because those hash slots would become unassigned. You must first drain the node by resharding all of its slots to other active primary nodes. Only when `CLUSTER INFO` or `redis-cli --cluster check` confirms the node owns exactly 0 slots is it safe to formally remove it from the cluster configuration using `del-node`.

Who this module is for: Your Redis Cluster is under capacity and you need to add nodes, or you need to rebalance slot distribution after node failures and recoveries. Resharding moves hash slots between nodes while serving live traffic. This module covers the mechanics of slot migration, the ASKING redirection that enables zero-downtime resharding, and how to use the redis-cli cluster tooling.

Why Resharding?

After initial cluster creation, hash slots are distributed evenly. But over time, the distribution may become uneven:

Capacity scaling: adding a new node requires migrating slots to it from existing nodes
Data imbalance: some slots hold more data than others due to access patterns
Node removal: removing a node requires migrating its slots elsewhere first
Hotspot correction: if one node handles most traffic, rebalancing moves some slots to less-loaded nodes

Resharding is live — traffic continues flowing during migration. Keys in migrating slots are briefly accessible from both the source and destination during the transition.

Slot Migration: The States

A slot can be in one of three states during migration:

text

Normal:         Slot 7638 is on Node A — all reads/writes go to Node A

MIGRATING:      Slot 7638 is migrating FROM Node A to Node B
                - Keys still in Node A answer requests
                - Keys already moved to Node B → Node A issues ASK redirection

IMPORTING:      Slot 7638 is being imported TO Node B
                - Node B accepts keys for this slot only if the ASKING command was sent
                - Normal requests for slot 7638 still go to Node A

The MIGRATE Command

The low-level operation that moves a single key from one node to another:

MIGRATE host port key destination-db timeout [COPY] [REPLACE] [KEYS key [key ...]]

text

# Move key "user:1001" from Node A to Node B
MIGRATE 10.0.1.52 6379 user:1001 0 5000

MIGRATE:

Dumps the key's value (serializes to RDB format)
Sends it to the destination node with RESTORE semantics
If the destination acknowledges: deletes the key from the source
Atomic from the perspective of each node (the key exists on exactly one node at any moment)

Bulk migration with KEYS option:

MIGRATE 10.0.1.52 6379 "" 0 5000 KEYS user:1001 user:1002 user:1003

Slot Migration Protocol

The full protocol for migrating slot 7638 from Node A to Node B:

bash

# Step 1: Prepare the destination (Node B) to accept the slot
CLUSTER SETSLOT 7638 IMPORTING <node-A-id>

# Step 2: Prepare the source (Node A) to migrate the slot
CLUSTER SETSLOT 7638 MIGRATING <node-B-id>

# Step 3: Get all keys in slot 7638 on Node A, in batches
CLUSTER GETKEYSINSLOT 7638 100   → returns up to 100 keys in this slot

# Step 4: For each batch of keys, MIGRATE to Node B
MIGRATE 10.0.1.52 6379 "" 0 5000 KEYS user:1001 session:abc123

# Repeat steps 3-4 until CLUSTER GETKEYSINSLOT returns empty

# Step 5: Finalize: tell all nodes the slot now belongs to Node B
CLUSTER SETSLOT 7638 NODE <node-B-id>
# This must be sent to ALL nodes in the cluster, not just A and B

After CLUSTER SETSLOT 7638 NODE <node-B-id> is propagated to all nodes, the migration is complete. Normal MOVED redirections take over for slot 7638 pointing to Node B.

The ASKING Redirection During Migration

While slot 7638 is in MIGRATING state on Node A:

Keys that haven't been migrated yet are still on Node A → served normally
Keys that have been migrated are on Node B → Node A returns -ASK 7638 10.0.1.52:6379

The client receiving an ASK redirection must:

Send ASKING to Node B (one-time flag — tells Node B "I know you're importing this slot, accept this command")
Resend the original command to Node B
Not update its slot map — slot 7638 is still officially on Node A until migration completes

text

Client → Node A: GET user:1001  (user:1001 already migrated)
Node A → -ASK 7638 10.0.1.52:6379

Client → Node B: ASKING
Client → Node B: GET user:1001  → "Jatin"
# Client's slot map: 7638 → Node A  (unchanged)

# Next request for a different key in slot 7638:
Client → Node A: GET user:1002  (still on Node A)
→ served normally

This is what makes resharding zero-downtime: every key is always accessible, just the route changes depending on whether the specific key has been migrated yet.

Using redis-cli for Resharding

The redis-cli tool automates the multi-step migration protocol:

Add a Node and Rebalance

bash

# Add a new empty primary node to the cluster
redis-cli --cluster add-node 10.0.1.60:6379 10.0.1.50:6379
# new-node-address:port  existing-cluster-node:port

# The new node has 0 slots — rebalance to distribute evenly
redis-cli --cluster rebalance 10.0.1.50:6379 --cluster-use-empty-masters

rebalance automatically calculates how many slots to move from each existing node to the new one and runs the migration.

Manual Resharding

bash

# Move 1000 slots to the new node
redis-cli --cluster reshard 10.0.1.50:6379 \
  --cluster-from <source-node-id> \
  --cluster-to <destination-node-id> \
  --cluster-slots 1000 \
  --cluster-yes   → skip confirmation prompt

reshard handles the full SETSLOT MIGRATING/IMPORTING protocol, GETKEYSINSLOT, MIGRATE, and SETSLOT NODE sequence automatically.

Adding a Node as a Replica

bash

redis-cli --cluster add-node 10.0.1.61:6379 10.0.1.50:6379 \
  --cluster-slave \
  --cluster-master-id <primary-node-id>

Remove a Node

You must first migrate all its slots away, then remove it:

bash

# Move all slots from the node to be removed to other nodes
redis-cli --cluster reshard 10.0.1.50:6379 \
  --cluster-from <node-to-remove-id> \
  --cluster-to <receiving-node-id> \
  --cluster-slots <all-slots-count>

# Then remove the empty node
redis-cli --cluster del-node 10.0.1.50:6379 <node-id>

Resharding Performance and Impact

Migration speed is limited by:

Key size: larger values take longer to serialize and transmit
Key count: more keys in the migrating slot = longer migration
Network bandwidth between source and destination
MIGRATE timeout: keys that cannot be migrated within the timeout are left in place and retried

During migration:

Read and write commands continue to work (via MOVED/ASK redirections)
Latency may increase slightly due to extra redirections
The source node's CPU and I/O load increases during active migration

To minimise impact:

Migrate during off-peak hours for large datasets
Use --cluster-pipeline option to batch MIGRATE commands
Monitor source node CPU and network during migration

bash

# Use pipeline option for faster migration (batch 10 keys per MIGRATE call)
redis-cli --cluster reshard 10.0.1.50:6379 \
  --cluster-from <source-id> \
  --cluster-to <dest-id> \
  --cluster-slots 1000 \
  --cluster-pipeline 10

Verifying Migration

bash

# Check cluster health and slot distribution
redis-cli --cluster check 10.0.1.50:6379

# Expected output:
# [OK] All nodes agree about slots configuration.
# [OK] All 16384 slots covered.

# Detailed slot distribution per node
redis-cli --cluster info 10.0.1.50:6379

text

127.0.0.1:6379> CLUSTER INFO
cluster_state:ok
cluster_slots_assigned:16384
cluster_slots_ok:16384
cluster_slots_pfail:0
cluster_slots_fail:0

Any value > 0 for cluster_slots_pfail or cluster_slots_fail indicates nodes in a degraded state.

When Resharding Time Is Proportional to Key Count

The most common surprise with resharding: it is slow for large datasets. Each key must be serialized, sent over the network, restored on the destination, and deleted from the source. A slot with 1 million keys takes much longer than a slot with 1,000 keys.

For very large datasets:

Estimate migration time before starting: measure MIGRATE speed (keys/second) on a test slot, extrapolate
Plan for migrations to take hours on multi-billion-key clusters
Consider creating the new node as a replica first, letting it sync fully, then resharding — the replica sync is faster than MIGRATE for bulk data transfer

Summary

Resharding moves hash slots between nodes while traffic continues — zero downtime via MOVING/IMPORTING slot states and ASKING redirections
MIGRATING on source + IMPORTING on destination: keys not yet moved served from source; keys moved served from destination via ASK
ASK redirection: temporary — send ASKING then the command; do not update slot map until CLUSTER SETSLOT NODE finalises
MIGRATE moves individual keys atomically — key exists on exactly one node at any moment during migration
redis-cli --cluster reshard automates the full migration protocol; --cluster rebalance distributes slots evenly after adding nodes
Migration speed = keys/second × key-count; large datasets require hours and should be planned for off-peak
After migration: redis-cli --cluster check to verify all 16,384 slots are covered and healthy

Next: A-11 — Gossip Protocol and Network Partition Handling — how cluster nodes discover each other, propagate topology changes, and handle split scenarios where portions of the cluster become unreachable.

Knowledge Check

During a live resharding operation, a Redis Cluster client requests a key that belongs to a hash slot currently migrating from Node A to Node B. The specific key has *already* been migrated to Node B. What sequence of events occurs?

When migrating a massive dataset (e.g., 50 million keys) from one Redis Cluster node to another using redis-cli --cluster reshard, what is the most significant performance bottleneck you should anticipate?

You want to gracefully remove a primary node (Node C) from a running Redis Cluster. What is the correct sequence of operations?

Test your knowledge with more question sets

Sign in to keep reading

The rest of this module is free — sign in with Google to unlock it and track your progress.

PreviousModule A-9: Redis Cluster: Hash Slots and Data Distribution Next Module A-11: Gossip Protocol and Network Partition Handling

Discussion

Join the discussion

Loading comments...