A-14 — Performance Benchmarking and Production Tuning

Who this module is for: Redis is slower than expected — commands are taking longer than they should, throughput is lower than the hardware should support, or latency spikes are occurring under load. This module covers redis-benchmark for baseline measurement, OS-level tuning that meaningfully impacts Redis performance, and the configuration changes that consistently improve throughput in production.

redis-benchmark: Establishing Baselines

redis-benchmark is the official Redis performance testing tool. Always benchmark before tuning — you need a baseline to know whether a change improved anything.

Interpreting Results

What these numbers mean:

• 117K ops/sec for SET with 50 clients and 3-byte payload is typical for a local Redis
• With 1KB payload: expect 50–80K ops/sec (memory bandwidth becomes the bottleneck)
• With 100KB payload: expect 5–10K ops/sec (dominated by serialisation and network bandwidth)

What redis-benchmark does not test:

• Your actual command mix (most workloads are not 100% GET or 100% SET)
• Real client libraries (ioredis has overhead compared to the raw C client benchmark uses)
• Production network topology (benchmark runs locally by default)

Always supplement with production metrics from INFO commandstats to understand your real-world baseline.

OS-Level Tuning

Redis performance is heavily influenced by OS configuration. These are the settings that matter most.

1. Disable Transparent Huge Pages (THP)

THP causes Redis latency spikes during BGSAVE because copy-on-write must duplicate 2MB pages instead of 4KB pages. Redis explicitly warns you if THP is enabled:

Disable permanently:

Impact: Eliminates latency spikes of 10–100ms that occur during BGSAVE on write-heavy instances.

2. Increase System File Descriptor Limit

Each Redis client connection uses a file descriptor. The default limit (1024 on many Linux systems) is too low for production Redis:

Also configure in redis.conf:

3. TCP Backlog

The TCP backlog queue holds pending connection requests. Under high connection rates, a small backlog causes connections to be silently dropped:

Redis also warns when the OS TCP backlog is smaller than its configured backlog:

4. Disable Swap (or Configure Swappiness)

Redis accesses data randomly across its entire working set. If any Redis data is swapped to disk, access times jump from microseconds to milliseconds.

Monitor for swapping: INFO memory → used_memory_rss < used_memory indicates Redis data is in swap.

5. CPU Pinning (High-Throughput Workloads)

For very high-throughput Redis (> 500K ops/sec), pin the Redis process to a specific CPU core to avoid context switching across cores:

I/O threads (Redis 6.0+) can be pinned to separate cores:

Redis Configuration Tuning

1. hz: Event Loop Frequency

The hz setting controls how often Redis runs its internal event loop for background tasks (active key expiry, client timeout checks, etc.).

• Higher hz (e.g., 100): more frequent expiry checks, faster TTL response, higher CPU usage at idle
• Lower hz (e.g., 10): less CPU at idle, slightly less responsive expiry

For latency-sensitive workloads with many expiring keys:

2. Socket and TCP Options

tcp-keepalive prevents dead connections from accumulating (clients that crashed without disconnecting properly). Without it, these connections linger until Redis's maxclients limit is hit.

3. Lazy Freeing

Enable lazy eviction and lazy expire — they offload the actual memory deallocation of large keys to a background thread, reducing main thread blocking.

4. AOF fsync Tuning for Write Throughput

For write-heavy workloads where appendfsync always is required:

During AOF rewrite (which itself calls fsync), disabling the regular fsync prevents double I/O contention on the disk. The risk: if the machine crashes during rewrite, the AOF file may be partially written — but the RDB snapshot provides a recovery point.

5. Replication Buffer Size

For instances with many replicas or replicas that occasionally fall behind:

Tune based on your write rate × maximum tolerable disconnect time.

Identifying and Fixing Specific Performance Issues

Symptom: High Latency on Specific Commands

If HGETALL has usec_per_call=500 (500µs) while GET has usec_per_call=5:

• Hash is in hashtable encoding (large) — reduce fields or raise listpack threshold
• Hash has thousands of fields — never designed for HGETALL on large hashes

Symptom: Periodic Latency Spikes (Every N Minutes)

Correlate with:

• INFO persistence → rdb_bgsave_in_progress: 1 during spikes → BGSAVE fork causing CoW spikes
• LATENCY HISTORY fork → confirm fork latency is elevated
• Fix: ensure THP is disabled; consider reducing write rate during snapshot window

Symptom: High keyspace_misses (Low Hit Rate)

Low hit rate causes: TTLs too short, maxmemory too small, cache warming not working, key patterns not matching access patterns.

• Increase TTLs if data is not changing frequently
• Increase maxmemory if eviction is occurring (evicted_keys > 0)
• Profile which keys are being missed with MONITOR (briefly, on a replica)

Symptom: Growing Memory Despite Eviction

Enable active defragmentation:

The 5 Configuration Changes With the Highest Impact

In order of typical impact:

1. Disable THP — eliminates BGSAVE-related latency spikes (10–100ms spikes → gone)
2. Set maxmemory and maxmemory-policy — prevents OOM kill; defines cache behaviour
3. Enable lazy freeing (lazyfree-lazy-eviction yes, lazyfree-lazy-expire yes) — reduces main-thread blocking on large key deletions
4. Increase replication backlog (repl-backlog-size 100mb) — prevents expensive full resyncs on brief replica disconnections
5. Enable activedefrag yes with appropriate thresholds — recovers memory from fragmentation without restarting

Summary

• redis-benchmark establishes throughput baselines; supplement with INFO commandstats for real-world command latency
• OS tuning that matters most: disable THP (eliminates BGSAVE latency spikes), increase file descriptor limit, increase TCP backlog, disable swap
• Redis config: hz 100 + dynamic-hz yes for faster expiry; tcp-keepalive 300 for dead connection cleanup; lazy freeing for async memory deallocation
• Identify slow commands via SLOWLOG and INFO commandstats; correlate latency spikes with LATENCY HISTORY fork
• Top 5 high-impact changes: disable THP, set maxmemory+policy, enable lazy freeing, increase replication backlog, enable activedefrag

Next: A-15 — Topology Decision Tree: Standalone, Sentinel, or Cluster — the final module synthesises everything into a decision framework for choosing the right Redis deployment topology for your specific requirements.