A-14 — Performance Benchmarking and Production Tuning

redis-benchmark: Establishing Baselines

redis-benchmark is the official Redis performance testing tool. Always benchmark before tuning — you need a baseline to know whether a change improved anything.

bash
# Basic benchmark: GET and SET throughput with 50 concurrent clients, 100K requests
redis-benchmark -h 127.0.0.1 -p 6379 -c 50 -n 100000

# Benchmark specific commands
redis-benchmark -h 127.0.0.1 -p 6379 -c 50 -n 100000 -t get,set,hset,lpush

# With authentication
redis-benchmark -h 127.0.0.1 -p 6379 -a password -c 50 -n 100000 -t get,set

# Test with different payload sizes (to understand memory bandwidth limits)
redis-benchmark -h 127.0.0.1 -p 6379 -n 100000 -d 1024   # 1KB values
redis-benchmark -h 127.0.0.1 -p 6379 -n 100000 -d 100000  # 100KB values

# Pipeline test (simulate batch operations)
redis-benchmark -h 127.0.0.1 -p 6379 -n 100000 -P 16   # 16 commands per pipeline

# Output in CSV format for analysis
redis-benchmark -h 127.0.0.1 -p 6379 -n 100000 --csv

Interpreting Results

====== SET ======
  100000 requests completed in 0.85 seconds
  50 parallel clients
  3 bytes payload
  keep alive: 1

99.68% <= 1 milliseconds
99.91% <= 2 milliseconds
100.00% <= 3 milliseconds
117647.06 requests per second        ← throughput

What these numbers mean:

• 117K ops/sec for SET with 50 clients and 3-byte payload is typical for a local Redis
• With 1KB payload: expect 50–80K ops/sec (memory bandwidth becomes the bottleneck)
• With 100KB payload: expect 5–10K ops/sec (dominated by serialisation and network bandwidth)

What redis-benchmark does not test:

• Your actual command mix (most workloads are not 100% GET or 100% SET)
• Real client libraries (ioredis has overhead compared to the raw C client benchmark uses)
• Production network topology (benchmark runs locally by default)

Always supplement with production metrics from INFO commandstats to understand your real-world baseline.

OS-Level Tuning

Redis performance is heavily influenced by OS configuration. These are the settings that matter most.

1. Disable Transparent Huge Pages (THP)

THP causes Redis latency spikes during BGSAVE because copy-on-write must duplicate 2MB pages instead of 4KB pages. Redis explicitly warns you if THP is enabled:

WARNING you have Transparent Huge Pages (THP) support enabled in your kernel.
This will create latency and memory usage issues with Redis.

Disable permanently:

bash
# Disable immediately
echo never > /sys/kernel/mm/transparent_hugepage/enabled

# Persist across reboots (add to /etc/rc.local or systemd unit)
echo 'never' > /sys/kernel/mm/transparent_hugepage/enabled
echo 'never' > /sys/kernel/mm/transparent_hugepage/defrag

Impact: Eliminates latency spikes of 10–100ms that occur during BGSAVE on write-heavy instances.

2. Increase System File Descriptor Limit

Each Redis client connection uses a file descriptor. The default limit (1024 on many Linux systems) is too low for production Redis:

bash
# Check current limits
ulimit -n

# Increase for the Redis process (add to /etc/security/limits.conf):
redis soft nofile 65536
redis hard nofile 65536

# Or set in the Redis systemd service:
[Service]
LimitNOFILE=65536

Also configure in redis.conf:

maxclients 10000   → Redis will set the OS fd limit accordingly

3. TCP Backlog

The TCP backlog queue holds pending connection requests. Under high connection rates, a small backlog causes connections to be silently dropped:

bash
# Increase OS TCP backlog
sysctl -w net.core.somaxconn=65535
sysctl -w net.ipv4.tcp_max_syn_backlog=65535

# Persist in /etc/sysctl.conf
net.core.somaxconn = 65535
net.ipv4.tcp_max_syn_backlog = 65535

Redis also warns when the OS TCP backlog is smaller than its configured backlog:

WARNING: The TCP backlog setting of 511 cannot be enforced because /proc/sys/net/core/somaxconn is set to the lower value of 128.

# redis.conf
tcp-backlog 511

4. Disable Swap (or Configure Swappiness)

Redis accesses data randomly across its entire working set. If any Redis data is swapped to disk, access times jump from microseconds to milliseconds.

bash
# Check if swap is in use
free -h
cat /proc/meminfo | grep Swap

# Disable swappiness (Redis-specific machines):
sysctl -w vm.swappiness=0

# Or reduce swappiness (shared machines):
sysctl -w vm.swappiness=1

Monitor for swapping: INFO memory → used_memory_rss < used_memory indicates Redis data is in swap.

5. CPU Pinning (High-Throughput Workloads)

For very high-throughput Redis (> 500K ops/sec), pin the Redis process to a specific CPU core to avoid context switching across cores:

bash
# Pin Redis to CPU 0
taskset -cp 0 $(pgrep redis-server)

# Or in the systemd service file:
CPUAffinity=0

I/O threads (Redis 6.0+) can be pinned to separate cores:

# redis.conf
server-cpulist 0               → main thread on CPU 0
bio-cpulist 1                  → BIO threads on CPU 1
io-threads-cpulist 2-3         → I/O threads on CPUs 2-3

Redis Configuration Tuning

1. hz: Event Loop Frequency

hz 10   → default: 10 cycles per second

The hz setting controls how often Redis runs its internal event loop for background tasks (active key expiry, client timeout checks, etc.).

• Higher hz (e.g., 100): more frequent expiry checks, faster TTL response, higher CPU usage at idle
• Lower hz (e.g., 10): less CPU at idle, slightly less responsive expiry

For latency-sensitive workloads with many expiring keys:

hz 100
dynamic-hz yes   → automatically adjust hz based on connected clients

2. Socket and TCP Options

tcp-keepalive 300   → send TCP keepalive every 300 seconds (detect dead connections)
timeout 0           → don't timeout idle clients (0 = never); set to 300 for idle clients

tcp-keepalive prevents dead connections from accumulating (clients that crashed without disconnecting properly). Without it, these connections linger until Redis's maxclients limit is hit.

3. Lazy Freeing

lazyfree-lazy-eviction yes   → async eviction (don't block main thread on large evictions)
lazyfree-lazy-expire yes     → async key expiry for large keys
lazyfree-lazy-server-del yes → UNLINK instead of DEL for server-side deletes (AOF rewrite, etc.)
lazyfree-lazy-user-del no    → DEL is synchronous for user-issued commands (default)
lazyfree-lazy-user-flush no  → FLUSHDB/FLUSHALL is synchronous by default

Enable lazy eviction and lazy expire — they offload the actual memory deallocation of large keys to a background thread, reducing main thread blocking.

4. AOF fsync Tuning for Write Throughput

For write-heavy workloads where appendfsync always is required:

no-appendfsync-on-rewrite yes   → disable fsync during AOF rewrite (reduces I/O contention)

During AOF rewrite (which itself calls fsync), disabling the regular fsync prevents double I/O contention on the disk. The risk: if the machine crashes during rewrite, the AOF file may be partially written — but the RDB snapshot provides a recovery point.

5. Replication Buffer Size

For instances with many replicas or replicas that occasionally fall behind:

repl-backlog-size 100mb   → larger backlog prevents full resyncs on brief disconnects

Tune based on your write rate × maximum tolerable disconnect time.

Identifying and Fixing Specific Performance Issues

Symptom: High Latency on Specific Commands

INFO commandstats → find commands with high usec_per_call

If HGETALL has usec_per_call=500 (500µs) while GET has usec_per_call=5:

• Hash is in hashtable encoding (large) — reduce fields or raise listpack threshold
• Hash has thousands of fields — never designed for HGETALL on large hashes

Symptom: Periodic Latency Spikes (Every N Minutes)

Correlate with:

• INFO persistence → rdb_bgsave_in_progress: 1 during spikes → BGSAVE fork causing CoW spikes
• LATENCY HISTORY fork → confirm fork latency is elevated
• Fix: ensure THP is disabled; consider reducing write rate during snapshot window

Symptom: High keyspace_misses (Low Hit Rate)

INFO stats → keyspace_hits and keyspace_misses

Low hit rate causes: TTLs too short, maxmemory too small, cache warming not working, key patterns not matching access patterns.

• Increase TTLs if data is not changing frequently
• Increase maxmemory if eviction is occurring (evicted_keys > 0)
• Profile which keys are being missed with MONITOR (briefly, on a replica)

Symptom: Growing Memory Despite Eviction

mem_fragmentation_ratio > 1.5

Enable active defragmentation:

CONFIG SET activedefrag yes
CONFIG SET active-defrag-threshold-lower 10
CONFIG SET active-defrag-threshold-upper 100

The 5 Configuration Changes With the Highest Impact

In order of typical impact:

1. Disable THP — eliminates BGSAVE-related latency spikes (10–100ms spikes → gone)
2. Set maxmemory and maxmemory-policy — prevents OOM kill; defines cache behaviour
3. Enable lazy freeing (lazyfree-lazy-eviction yes, lazyfree-lazy-expire yes) — reduces main-thread blocking on large key deletions
4. Increase replication backlog (repl-backlog-size 100mb) — prevents expensive full resyncs on brief replica disconnections
5. Enable activedefrag yes with appropriate thresholds — recovers memory from fragmentation without restarting

Summary

• redis-benchmark establishes throughput baselines; supplement with INFO commandstats for real-world command latency
• OS tuning that matters most: disable THP (eliminates BGSAVE latency spikes), increase file descriptor limit, increase TCP backlog, disable swap
• Redis config: hz 100 + dynamic-hz yes for faster expiry; tcp-keepalive 300 for dead connection cleanup; lazy freeing for async memory deallocation
• Identify slow commands via SLOWLOG and INFO commandstats; correlate latency spikes with LATENCY HISTORY fork
• Top 5 high-impact changes: disable THP, set maxmemory+policy, enable lazy freeing, increase replication backlog, enable activedefrag

Next: A-15 — Topology Decision Tree: Standalone, Sentinel, or Cluster — the final module synthesises everything into a decision framework for choosing the right Redis deployment topology for your specific requirements.