P-4 — Memory Profiling and Optimization

Who this module is for: Redis is using more RAM than expected and you do not know why. Or you are designing a Redis schema and want to estimate memory costs before deploying. This module covers the full suite of Redis memory inspection tools and the optimizations that consistently recover the most RAM in production.

The Memory Audit Starting Point: INFO memory

Every Redis memory investigation starts here:

Interpreting the Key Fields

mem_fragmentation_ratio = used_memory_rss / used_memory

• < 1.0 → Redis is using swap (critical, investigate immediately)
• 1.0–1.2 → healthy
• 1.2–1.5 → moderate fragmentation (normal for dynamic workloads)

• 1.5 → high fragmentation — consider activedefrag or restart

used_memory_overhead = memory used by Redis's internal data structures (the global keyspace dict, expiry table, per-client buffers). If this is a large fraction of used_memory, you have very small values (overhead dominates) — consider consolidating keys into Hashes.

mem_clients_normal = memory used by client output buffers. If this is large (> 10MB), you may have slow clients receiving data faster than they can consume it.

MEMORY USAGE: Per-Key Cost

Returns the exact number of bytes allocated for a key and its value, including all internal structures (robj, SDS, listpack nodes, etc.).

For collections, SAMPLES controls how many elements are sampled to estimate total cost (default 5). Use SAMPLES 0 for exact measurement on small collections.

Using MEMORY USAGE to find expensive keys:

On a production instance with millions of keys, sample a representative subset:

MEMORY DOCTOR

Returns a human-readable diagnosis. Possible outputs:

Or for a healthy instance:

Not a substitute for INFO memory, but a quick sanity check.

MEMORY MALLOC-STATS

Dumps the full jemalloc allocator statistics — bin sizes, fragmentation per bin, active vs retained pages. Useful when you suspect allocator-level fragmentation rather than Redis-level issues.

Finding the Memory Culprits

Pattern 1: Large Hashes in hashtable Encoding

A Hash with > 128 fields (or any field > 64 bytes) switches from listpack to hashtable encoding. The memory cost jumps roughly 5x per element. Find them:

If you find 1,000 user Hashes in hashtable encoding that should be in listpack encoding (they have < 128 fields), your encoding threshold is wrong. Check:

If the threshold is already 128 but hashes have 50 fields in hashtable encoding, some field values exceed 64 bytes. Identify them with HGETALL on a sample key.

Pattern 2: Keys Without TTL (Orphaned Data)

Or in a Redis script:

If expires is much less than keys, most of your keys have no TTL. For a cache, this means eviction will eventually clear them — but you are paying for that memory until then. For an application database, this is expected.

Pattern 3: String Keys Storing JSON When Hashes Would Be Better

OBJECT ENCODING tells you if a String key is in raw encoding (large string). If it is storing a JSON blob, consider whether you update individual fields — if so, a Hash is more efficient and enables atomic partial updates.

Pattern 4: Sorted Set Keys in skiplist Encoding

Sorted Sets with > 128 members use skiplist encoding. A skiplist + hashtable for 1,000 members uses ~250KB; listpack for the same 1,000 members uses ~55KB. If you have many small-to-medium sorted sets exceeding the listpack threshold by a few members:

Active Defragmentation

When mem_fragmentation_ratio > 1.5, enable active defragmentation:

Active defragmentation runs a background scan, finding allocations that can be moved to compacted jemalloc pages. It uses 1–25% of a CPU core and can recover significant memory without restarting Redis.

When it is not enough: If mem_fragmentation_ratio > 2.0 and the instance has been running for months with heavy churn, active defragmentation may be slow to converge. A Redis restart (graceful shutdown → dump.rdb / AOF flush → restart → reload) resets memory layout and eliminates fragmentation instantly. Plan this during a low-traffic window.

Encoding Threshold Tuning

The single most impactful memory optimization is ensuring data structures use compact encodings.

Hash Thresholds

If your user Hashes have 50 fields with values averaging 30 bytes: raise to entries 256, value 64 to keep them in listpack. Memory reduction: ~5x per hash.

Sorted Set Thresholds

If your leaderboard sorted sets have up to 200 members under 40 bytes each: raise to entries 256.

Set Thresholds (integer sets)

If you are storing user IDs (integers) in Sets: intset is the most compact encoding. Ensure all members are integers to keep the intset encoding.

Testing Threshold Changes

After changing thresholds:

1. New keys will use the new thresholds
2. Existing keys will NOT automatically convert (they were already promoted to the larger encoding)
3. To convert existing keys: DUMP + RESTORE or use redis-cli --pipe to reload the data

The safest approach for large deployments: change thresholds and let natural key churn (TTL expiry + re-creation) gradually adopt the new encoding.

The Hash-for-Small-Objects Pattern

For reference, here is the memory comparison that drives this pattern:

The per-key overhead (robj, SDS for key, hash table entry) adds ~80 bytes per key. When values are small, this overhead dominates. Grouping small objects under a single Hash key eliminates most of it.

Implementation: Instead of SET user:1001:name "Jatin", use HSET users 1001:name "Jatin" (or one Hash per user: HSET user:1001 name "Jatin" email "...".

This pattern has limits: a Hash cannot have per-field TTLs, and you cannot atomically query across multiple user Hashes. For most use cases, the memory savings outweigh these constraints.

Memory Optimization Checklist

• Run INFO memory — check mem_fragmentation_ratio, used_memory_overhead
• Find top 20 keys by memory: MEMORY USAGE + SCAN
• Check encoding of large collections: OBJECT ENCODING
• Identify Hashes in hashtable encoding that should be listpack: HLEN + OBJECT ENCODING
• Check for keys without TTL in a cache context: INFO keyspace (expires vs keys count)
• Review encoding thresholds: CONFIG GET hash-max-listpack-*, zset-max-listpack-*
• Consider active defragmentation if mem_fragmentation_ratio > 1.5
• Evaluate Hash-for-small-objects pattern for high-cardinality small-value datasets
• Set maxmemory and maxmemory-policy if not set (do not let Redis use unbounded RAM)

Summary

• INFO memory is the starting point: check mem_fragmentation_ratio, used_memory_dataset, mem_clients_normal
• MEMORY USAGE key gives exact per-key RAM cost including all internal structures
• MEMORY DOCTOR for a quick health check; MEMORY MALLOC-STATS for allocator-level detail
• Encoding inspection: OBJECT ENCODING reveals whether a key is in compact (listpack, intset) or large (hashtable, skiplist) encoding
• Tuning encoding thresholds (hash-max-listpack-entries, zset-max-listpack-entries) is the highest-leverage memory optimization
• Active defragmentation (activedefrag yes) recovers memory from fragmentation without restarting
• The Hash-for-small-objects pattern reduces per-key overhead for high-cardinality small datasets by 5–10x

Next: P-5 — Atomic Counters, Rate Limiters, and Sliding Windows — building lock-free counters, fixed and sliding window rate limiters, and the token bucket algorithm using Redis's atomic operations.