Module A-14·20 min read

redis-benchmark usage and interpreting results, OS-level tuning (transparent huge pages, TCP backlog, ulimit), slowlog analysis, latency percentile monitoring, and the 5 configuration changes that meaningfully improve throughput.

JJS

Written by Jatin Jain Saraf · Senior Software Engineer

A-14 — Performance Benchmarking and Production Tuning

Q: You notice severe, periodic latency spikes (10-100ms) on a write-heavy Redis production server. Checking the logs, you see that these spikes perfectly correlate with Redis initiating a `BGSAVE` operation for an RDB snapshot. What is the most likely operating system-level cause of this specific performance degradation?

Transparent Huge Pages (THP) is enabled in the Linux kernel, causing the Copy-on-Write (CoW) mechanism to duplicate memory in massive 2MB chunks instead of standard 4KB pages when data changes during the save. — This is a classic Redis performance gotcha. When Redis forks a background process to create an RDB snapshot (`BGSAVE`), it relies on the OS's Copy-on-Write mechanism. Normally, if a client writes to a key during the save, the OS only has to copy a tiny 4KB page of memory so the parent process can mutate it while the child process reads the old version. If THP is enabled, the OS must copy a massive 2MB page instead. On a busy server, copying hundreds of 2MB pages stalls the main Redis thread, resulting in severe latency spikes. Disabling THP is mandatory for Redis deployments.

Q: You have a Redis instance configured as an LRU cache. It is holding millions of large string payloads. Occasionally, an application update necessitates flushing specific namespaces of keys, or the cache reaches max memory and begins evicting old keys. During these eviction or deletion events, the main Redis thread blocks entirely, causing application timeouts. Which configuration changes will best mitigate this blocking behavior?

Enable `lazyfree-lazy-eviction yes` and `lazyfree-lazy-expire yes` to offload the actual memory deallocation to a background thread. — Because Redis is primarily single-threaded for command execution, deleting a massive key (e.g., a Set with 10 million elements, or a 500MB string) blocks the main thread while the OS reclaims that memory. By enabling the `lazyfree` family of settings, Redis will logically unlink the key from the dictionary instantly (making it invisible to clients) but will hand off the actual slow memory deallocation work to a background I/O thread, keeping the main thread highly responsive.

Q: When interpreting the results of a `redis-benchmark` run, you notice that throughput for `SET` commands with a 3-byte payload is 120,000 requests per second, but throughput for `SET` commands with a 100-kilobyte payload plummets to 8,000 requests per second. What is the primary bottleneck explaining this drop?

The performance is dominated by network bandwidth and the memory allocation/serialization overhead of moving large payloads, rather than CPU command processing speed. — Redis is incredibly fast at command execution. For small payloads, the bottleneck is often the CPU parsing the protocol or the sheer number of TCP packets. However, when you introduce large payloads (like 100KB), the sheer volume of data moving across the network interface and being allocated into RAM becomes the limiting factor. You simply hit the physical limits of network bandwidth (e.g., a 1 Gigabit link) or memory bandwidth long before you saturate the CPU's ability to execute the `SET` command logic.

Who this module is for: Redis is slower than expected — commands are taking longer than they should, throughput is lower than the hardware should support, or latency spikes are occurring under load. This module covers redis-benchmark for baseline measurement, OS-level tuning that meaningfully impacts Redis performance, and the configuration changes that consistently improve throughput in production.

redis-benchmark: Establishing Baselines

redis-benchmark is the official Redis performance testing tool. Always benchmark before tuning — you need a baseline to know whether a change improved anything.

bash

# Basic benchmark: GET and SET throughput with 50 concurrent clients, 100K requests
redis-benchmark -h 127.0.0.1 -p 6379 -c 50 -n 100000

# Benchmark specific commands
redis-benchmark -h 127.0.0.1 -p 6379 -c 50 -n 100000 -t get,set,hset,lpush

# With authentication
redis-benchmark -h 127.0.0.1 -p 6379 -a password -c 50 -n 100000 -t get,set

# Test with different payload sizes (to understand memory bandwidth limits)
redis-benchmark -h 127.0.0.1 -p 6379 -n 100000 -d 1024   # 1KB values
redis-benchmark -h 127.0.0.1 -p 6379 -n 100000 -d 100000  # 100KB values

# Pipeline test (simulate batch operations)
redis-benchmark -h 127.0.0.1 -p 6379 -n 100000 -P 16   # 16 commands per pipeline

# Output in CSV format for analysis
redis-benchmark -h 127.0.0.1 -p 6379 -n 100000 --csv

Interpreting Results

text

====== SET ======
  100000 requests completed in 0.85 seconds
  50 parallel clients
  3 bytes payload
  keep alive: 1

99.68% <= 1 milliseconds
99.91% <= 2 milliseconds
100.00% <= 3 milliseconds
117647.06 requests per second        ← throughput

What these numbers mean:

117K ops/sec for SET with 50 clients and 3-byte payload is typical for a local Redis
With 1KB payload: expect 50–80K ops/sec (memory bandwidth becomes the bottleneck)
With 100KB payload: expect 5–10K ops/sec (dominated by serialisation and network bandwidth)

What redis-benchmark does not test:

Your actual command mix (most workloads are not 100% GET or 100% SET)
Real client libraries (ioredis has overhead compared to the raw C client benchmark uses)
Production network topology (benchmark runs locally by default)

Always supplement with production metrics from INFO commandstats to understand your real-world baseline.

OS-Level Tuning

Redis performance is heavily influenced by OS configuration. These are the settings that matter most.

1. Disable Transparent Huge Pages (THP)

THP causes Redis latency spikes during BGSAVE because copy-on-write must duplicate 2MB pages instead of 4KB pages. Redis explicitly warns you if THP is enabled:

text

WARNING you have Transparent Huge Pages (THP) support enabled in your kernel.
This will create latency and memory usage issues with Redis.

Disable permanently:

bash

# Disable immediately
echo never > /sys/kernel/mm/transparent_hugepage/enabled

# Persist across reboots (add to /etc/rc.local or systemd unit)
echo 'never' > /sys/kernel/mm/transparent_hugepage/enabled
echo 'never' > /sys/kernel/mm/transparent_hugepage/defrag

Impact: Eliminates latency spikes of 10–100ms that occur during BGSAVE on write-heavy instances.

2. Increase System File Descriptor Limit

Each Redis client connection uses a file descriptor. The default limit (1024 on many Linux systems) is too low for production Redis:

bash

# Check current limits
ulimit -n

# Increase for the Redis process (add to /etc/security/limits.conf):
redis soft nofile 65536
redis hard nofile 65536

# Or set in the Redis systemd service:
[Service]
LimitNOFILE=65536

Also configure in redis.conf:

maxclients 10000   → Redis will set the OS fd limit accordingly

3. TCP Backlog

The TCP backlog queue holds pending connection requests. Under high connection rates, a small backlog causes connections to be silently dropped:

bash

# Increase OS TCP backlog
sysctl -w net.core.somaxconn=65535
sysctl -w net.ipv4.tcp_max_syn_backlog=65535

# Persist in /etc/sysctl.conf
net.core.somaxconn = 65535
net.ipv4.tcp_max_syn_backlog = 65535

Redis also warns when the OS TCP backlog is smaller than its configured backlog:

WARNING: The TCP backlog setting of 511 cannot be enforced because /proc/sys/net/core/somaxconn is set to the lower value of 128.

text

# redis.conf
tcp-backlog 511

4. Disable Swap (or Configure Swappiness)

Redis accesses data randomly across its entire working set. If any Redis data is swapped to disk, access times jump from microseconds to milliseconds.

bash

# Check if swap is in use
free -h
cat /proc/meminfo | grep Swap

# Disable swappiness (Redis-specific machines):
sysctl -w vm.swappiness=0

# Or reduce swappiness (shared machines):
sysctl -w vm.swappiness=1

Monitor for swapping: INFO memory → used_memory_rss < used_memory indicates Redis data is in swap.

5. CPU Pinning (High-Throughput Workloads)

For very high-throughput Redis (> 500K ops/sec), pin the Redis process to a specific CPU core to avoid context switching across cores:

bash

# Pin Redis to CPU 0
taskset -cp 0 $(pgrep redis-server)

# Or in the systemd service file:
CPUAffinity=0

I/O threads (Redis 6.0+) can be pinned to separate cores:

text

# redis.conf
server-cpulist 0               → main thread on CPU 0
bio-cpulist 1                  → BIO threads on CPU 1
io-threads-cpulist 2-3         → I/O threads on CPUs 2-3

Redis Configuration Tuning

1. hz: Event Loop Frequency

hz 10   → default: 10 cycles per second

The hz setting controls how often Redis runs its internal event loop for background tasks (active key expiry, client timeout checks, etc.).

Higher hz (e.g., 100): more frequent expiry checks, faster TTL response, higher CPU usage at idle
Lower hz (e.g., 10): less CPU at idle, slightly less responsive expiry

For latency-sensitive workloads with many expiring keys:

text

hz 100
dynamic-hz yes   → automatically adjust hz based on connected clients

2. Socket and TCP Options

text

tcp-keepalive 300   → send TCP keepalive every 300 seconds (detect dead connections)
timeout 0           → don't timeout idle clients (0 = never); set to 300 for idle clients

tcp-keepalive prevents dead connections from accumulating (clients that crashed without disconnecting properly). Without it, these connections linger until Redis's maxclients limit is hit.

3. Lazy Freeing

text

lazyfree-lazy-eviction yes   → async eviction (don't block main thread on large evictions)
lazyfree-lazy-expire yes     → async key expiry for large keys
lazyfree-lazy-server-del yes → UNLINK instead of DEL for server-side deletes (AOF rewrite, etc.)
lazyfree-lazy-user-del no    → DEL is synchronous for user-issued commands (default)
lazyfree-lazy-user-flush no  → FLUSHDB/FLUSHALL is synchronous by default

Enable lazy eviction and lazy expire — they offload the actual memory deallocation of large keys to a background thread, reducing main thread blocking.

4. AOF fsync Tuning for Write Throughput

For write-heavy workloads where appendfsync always is required:

no-appendfsync-on-rewrite yes   → disable fsync during AOF rewrite (reduces I/O contention)

During AOF rewrite (which itself calls fsync), disabling the regular fsync prevents double I/O contention on the disk. The risk: if the machine crashes during rewrite, the AOF file may be partially written — but the RDB snapshot provides a recovery point.

5. Replication Buffer Size

For instances with many replicas or replicas that occasionally fall behind:

repl-backlog-size 100mb   → larger backlog prevents full resyncs on brief disconnects

Tune based on your write rate × maximum tolerable disconnect time.

Identifying and Fixing Specific Performance Issues

Symptom: High Latency on Specific Commands

INFO commandstats tells you which command type is slow on average; SLOWLOG tells you the actual individual executions that crossed your latency threshold, with their exact arguments:

bash

CONFIG SET slowlog-log-slower-than 10000  # log commands over 10ms (microseconds)
SLOWLOG GET 10                             # most recent 10 slow entries
SLOWLOG LEN                                # how many entries are queued
SLOWLOG RESET                              # clear the log after investigating

Each SLOWLOG GET entry includes the command and its arguments, which commandstats' aggregate averages don't give you — that's what lets you find the specific HGETALL call on an oversized hash, not just that HGETALL is slow on average. P-11 (Monitoring and Observability) covers the full monitoring toolkit this fits into.

INFO commandstats → find commands with high usec_per_call

If HGETALL has usec_per_call=500 (500µs) while GET has usec_per_call=5:

Hash is in hashtable encoding (large) — reduce fields or raise listpack threshold
Hash has thousands of fields — never designed for HGETALL on large hashes

Symptom: Periodic Latency Spikes (Every N Minutes)

Correlate with:

INFO persistence → rdb_bgsave_in_progress: 1 during spikes → BGSAVE fork causing CoW spikes
LATENCY HISTORY fork → confirm fork latency is elevated
Fix: ensure THP is disabled; consider reducing write rate during snapshot window

Symptom: High `keyspace_misses` (Low Hit Rate)

INFO stats → keyspace_hits and keyspace_misses

Low hit rate causes: TTLs too short, maxmemory too small, cache warming not working, key patterns not matching access patterns.

Increase TTLs if data is not changing frequently
Increase maxmemory if eviction is occurring (evicted_keys > 0)
Profile which keys are being missed with MONITOR (briefly, on a replica)

Symptom: Growing Memory Despite Eviction

mem_fragmentation_ratio > 1.5

Enable active defragmentation:

text

CONFIG SET activedefrag yes
CONFIG SET active-defrag-threshold-lower 10
CONFIG SET active-defrag-threshold-upper 100

The 5 Configuration Changes With the Highest Impact

In order of typical impact:

Disable THP — eliminates BGSAVE-related latency spikes (10–100ms spikes → gone)
Set maxmemory and maxmemory-policy — prevents OOM kill; defines cache behaviour
Enable lazy freeing (lazyfree-lazy-eviction yes, lazyfree-lazy-expire yes) — reduces main-thread blocking on large key deletions
Increase replication backlog (repl-backlog-size 100mb) — prevents expensive full resyncs on brief replica disconnections
Enable activedefrag yes with appropriate thresholds — recovers memory from fragmentation without restarting

Summary

redis-benchmark establishes throughput baselines; supplement with INFO commandstats for real-world command latency
OS tuning that matters most: disable THP (eliminates BGSAVE latency spikes), increase file descriptor limit, increase TCP backlog, disable swap
Redis config: hz 100 + dynamic-hz yes for faster expiry; tcp-keepalive 300 for dead connection cleanup; lazy freeing for async memory deallocation
Identify slow commands via SLOWLOG and INFO commandstats; correlate latency spikes with LATENCY HISTORY fork
Top 5 high-impact changes: disable THP, set maxmemory+policy, enable lazy freeing, increase replication backlog, enable activedefrag

Next: A-15 — Topology Decision Tree: Standalone, Sentinel, or Cluster — the final module synthesises everything into a decision framework for choosing the right Redis deployment topology for your specific requirements.

Knowledge Check

You notice severe, periodic latency spikes (10-100ms) on a write-heavy Redis production server. Checking the logs, you see that these spikes perfectly correlate with Redis initiating a BGSAVE operation for an RDB snapshot. What is the most likely operating system-level cause of this specific performance degradation?

You have a Redis instance configured as an LRU cache. It is holding millions of large string payloads. Occasionally, an application update necessitates flushing specific namespaces of keys, or the cache reaches max memory and begins evicting old keys. During these eviction or deletion events, the main Redis thread blocks entirely, causing application timeouts. Which configuration changes will best mitigate this blocking behavior?

When interpreting the results of a redis-benchmark run, you notice that throughput for SET commands with a 3-byte payload is 120,000 requests per second, but throughput for SET commands with a 100-kilobyte payload plummets to 8,000 requests per second. What is the primary bottleneck explaining this drop?

Test your knowledge with more question sets

Sign in to keep reading

The rest of this module is free — sign in with Google to unlock it and track your progress.

PreviousModule A-13: Disaster Recovery, Backup, and Point-in-Time Restore Next Module A-15: Topology Decision Tree: Standalone, Sentinel, or Cluster

Discussion

Join the discussion

Loading comments...