Preventing Cache Stampedes and Hot Keys

Caching significantly reduces database load, but poorly designed caches can become bottlenecks themselves. Two of the most common production problems are cache stampedes, where many requests simultaneously rebuild an expired value, and hot keys, where a single cache entry receives a disproportionate amount of traffic.

These issues can overwhelm databases, saturate Redis instances, increase latency, and trigger cascading failures. Preventing them requires controlling how requests rebuild cached data, distributing load across cache infrastructure, and designing keys that avoid traffic concentration.

Table of Contents

• Understanding the Problem
• Preventing Cache Stampedes
  • Request Coalescing
  • Stale-While-Revalidate
  • TTL Jitter
• Handling Hot Keys
  • Replication
  • Key Sharding
  • Local Caching
• Strategy Comparison
• Production Python Examples
• Production Design Example
• Common Mistakes
• Production Checklist
• Conclusion
• More Articles to Read

Understanding the Problem

A cache normally protects the database by serving repeated requests. However, when many requests simultaneously miss the cache or target the same key, the cache itself becomes the source of system pressure.

Two failure patterns occur repeatedly in production:

• Cache stampede — many requests rebuild the same expired entry.
• Hot key — one cache key receives an extremely large percentage of traffic.

Both situations reduce cache effectiveness and can overload downstream systems.

Normal operation

1000 requests
      |
Redis
      |
990 cache hits
10 database queries


Cache stampede

1000 requests
      |
Expired key
      |
1000 database queries

A cache hit ratio may remain high overall while one popular key repeatedly overloads the database whenever it expires.

Production systems usually combine several techniques because no single approach eliminates every failure mode.

Preventing Cache Stampedes

A cache stampede occurs when many requests attempt to rebuild the same expired value simultaneously.

Cache expires

      |

1000 requests

      |

1000 cache misses

      |

1000 database queries

      |

Database overload

The objective is to ensure that only one request rebuilds the cache while the remaining requests either wait or temporarily use stale data.

Request Coalescing

Request coalescing allows one request to rebuild a missing cache entry while other concurrent requests wait for the result.

Cache miss
      |
Acquire lock
      |
+----------------------+
|                      |
| first request        |
|                      |
| Database query       |
|                      |
| Populate cache       |
+----------------------+
      |
Waiting requests
      |
Read newly cached value

This technique dramatically reduces database traffic during cache misses because only one expensive query executes.

Advantages

• Prevents duplicate database queries
• Protects backend services
• Simple for individual cache keys
• Works well with cache-aside

Disadvantages

• Waiting requests experience slightly higher latency
• Distributed locking is more complex than local locking
• Long rebuild operations increase wait time

When to Use

• expensive SQL queries
• pricing calculations
• recommendation engines
• external API responses
• large object reconstruction

A local implementation uses one lock per cache key.

from __future__ import annotations

import asyncio


class LockRegistry:

    def __init__(self):
        self._locks = {}

    def lock(
        self,
        key: str,
    ) -> asyncio.Lock:

        return self._locks.setdefault(
            key,
            asyncio.Lock(),
        )

Inside one application instance, only one coroutine rebuilds the cache. Distributed deployments usually require Redis locks or another coordination mechanism.

Stale-While-Revalidate

Instead of forcing every request to wait for fresh data, stale-while-revalidate temporarily serves the previous cached value while one request refreshes it in the background.

Expired cache
      |
Return stale value
      |
Background refresh
      |
Replace cache

This keeps latency low even while the cache is rebuilding.

Advantages

• Very low user latency
• Only one refresh operation
• Prevents request spikes
• Works well for public content

Disadvantages

• Users temporarily receive stale data
• Not suitable for strongly consistent workflows
• Refresh failures require retries

When to Use

• public APIs
• documentation
• blogs
• product catalogs
• search indexes

Many CDNs implement this behavior using stale-while-revalidate.

from fastapi import Response


def apply_headers(
    response: Response,
):

    response.headers["Cache-Control"] = (
        "public,"
        " max-age=300,"
        " stale-while-revalidate=60"
    )

TTL Jitter

If every cache entry expires after exactly five minutes, many keys disappear simultaneously.

No jitter

300
300
300
300
300


With jitter

274
286
305
319
291

Random expiration spreads cache rebuilds across time and significantly reduces synchronized database traffic.

Advantages

• Very simple implementation
• Reduces synchronized cache misses
• No infrastructure changes required
• Works with every cache strategy

Disadvantages

• Does not solve hot keys
• Popular keys can still stampede individually
• Requires reasonable TTL selection

When to Use

• almost every distributed cache

import random


def ttl_with_jitter(
    ttl: int,
) -> int:

    spread = ttl // 10

    return ttl + random.randint(
        -spread,
        spread,
    )

Handling Hot Keys

A hot key receives much more traffic than every other cache entry.

Examples include:

• homepage configuration
• global feature flags
• popular products
• currency exchange rates
• landing pages

Even when Redis responds quickly, millions of requests to one key can saturate CPU, network bandwidth, or a single cluster shard.

Normal traffic

product:1
product:2
product:3
product:4


Hot key

homepage

^^^^^^^^^^^^^^^^^^^^^^
millions of requests

Replication

One approach is to replicate the hot key across several Redis replicas or cache nodes so requests are distributed instead of concentrated.

Advantages

• Reduces pressure on one node
• Simple for read-heavy workloads
• Works well with static data

Disadvantages

• Updates become more expensive
• Replication lag may appear
• Higher infrastructure cost

Key Sharding

Another technique stores several identical copies of the same value using different keys.

homepage:1
homepage:2
homepage:3
homepage:4

The application randomly selects one key for reads, distributing traffic across multiple cache entries.

This technique is appropriate only for read-heavy values that change infrequently because every update must refresh every shard.

Local Caching

Frequently accessed shared values can also be cached inside each application instance. After the first Redis request, repeated reads are served directly from process memory.

Client
   |
Application
   |
Local Memory
   |
+--------+
|  HIT   |----> Return
+--------+
   |
MISS
   |
Redis
   |
Database

This approach dramatically reduces Redis traffic because every application instance serves repeated requests locally.

Advantages

• Microsecond read latency
• Reduces Redis CPU and network traffic
• Simple implementation
• Ideal for frequently reused reference data

Disadvantages

• Each application instance stores its own copy
• Local caches require invalidation or short TTLs
• Memory usage grows with application replicas

When to Use

• feature flags
• configuration
• country lists
• currencies
• carrier metadata
• small lookup tables

A lightweight in-memory cache can sit in front of Redis.

from cachetools import TTLCache


LOCAL_CACHE = TTLCache(
    maxsize=5000,
    ttl=30,
)

Local TTLs are usually shorter than Redis TTLs so stale values naturally disappear even if an invalidation event is missed.

Strategy Comparison

Stampede prevention and hot-key mitigation solve different problems. One protects backend systems during cache rebuilds, while the other distributes traffic away from overloaded cache nodes.

These techniques are complementary rather than exclusive. A high-traffic service often combines TTL jitter, request coalescing, stale-while-revalidate, and local caching simultaneously.

Production Python Examples

The following cache-aside implementation combines local caching, Redis, request coalescing, and TTL jitter.

from __future__ import annotations

import asyncio
import json
import random
from dataclasses import asdict, dataclass

from cachetools import TTLCache
from redis.asyncio import Redis


LOCAL_CACHE = TTLCache(
    maxsize=5000,
    ttl=30,
)


@dataclass(frozen=True)
class Product:
    product_id: str
    title: str
    price: float
    version: int


class ProductCache:

    def __init__(
        self,
        redis: Redis,
        repository,
    ):
        self.redis = redis
        self.repository = repository
        self.locks = {}

    async def get(
        self,
        product_id: str,
    ):

        if product_id in LOCAL_CACHE:
            return LOCAL_CACHE[product_id]

        cached = await self.redis.get(product_id)

        if cached:

            product = Product(
                **json.loads(cached)
            )

            LOCAL_CACHE[product_id] = product

            return product

        lock = self.locks.setdefault(
            product_id,
            asyncio.Lock(),
        )

        async with lock:

            cached = await self.redis.get(
                product_id
            )

            if cached:

                product = Product(
                    **json.loads(cached)
                )

                LOCAL_CACHE[product_id] = product

                return product

            product = await self.repository.get(
                product_id
            )

            if product is None:
                return None

            ttl = 300 + random.randint(
                -30,
                30,
            )

            await self.redis.set(
                product_id,
                json.dumps(
                    asdict(product)
                ),
                ex=ttl,
            )

            LOCAL_CACHE[product_id] = product

            return product

This implementation provides:

• local-memory hits for repeated requests
• Redis shared cache
• one database query per application instance during a cache miss
• randomized expiration

Distributed deployments usually replace the local lock with a distributed coordination mechanism.

from redis.asyncio import Redis


class DistributedLock:

    def __init__(
        self,
        redis: Redis,
    ):
        self.redis = redis

    async def acquire(
        self,
        key: str,
    ):

        return await self.redis.set(
            f"lock:{key}",
            "1",
            ex=10,
            nx=True,
        )

If the lock cannot be acquired, the request can briefly wait, retry the cache, or return a stale value depending on the workload.

Production Design Example

Consider a global e-commerce platform serving millions of requests per hour.

The homepage, best-selling products, feature flags, and pricing rules receive significantly more traffic than ordinary products. At the same time, cache entries expire continuously as new inventory and prices become available.

Users
   |
Browser Cache
   |
CDN
   |
Reverse Proxy
   |
Application
   |
Local Cache
   |
Redis Cluster
   |
PostgreSQL

The architecture combines several techniques:

• browser and CDN caching for static assets
• reverse-proxy request coalescing
• Redis cache-aside for shared business objects
• local in-memory caching for feature metadata
• TTL jitter for every Redis entry
• distributed locks for expensive cache rebuilds

This combination protects both Redis and PostgreSQL during traffic spikes while keeping latency consistently low.

Failure Scenarios

A popular cache entry expires during peak traffic. Without protection, thousands of requests immediately reach the database. Request coalescing ensures that only one request rebuilds the value while the remaining requests wait or receive a temporarily stale response.

Redis becomes overloaded by a single key. CPU utilization reaches 100%, increasing latency for unrelated keys. Local caching, key replication, or key sharding spreads traffic before it reaches Redis.

A deployment starts with an empty cache. Every application instance rebuilds popular objects simultaneously. Progressive warm-up, request coalescing, and preloading frequently accessed keys reduce cold-start pressure.

Many cache entries expire at the same second. The database receives a large burst of requests. TTL jitter distributes expiration over time so rebuilds occur gradually instead of simultaneously.

The request rebuilding a cache entry crashes. Waiting requests should retry the cache after the lock expires. Distributed locks require expiration to prevent permanent deadlocks.

Redis becomes temporarily unavailable. Local cache entries continue serving recent values. New cache misses fall back to PostgreSQL using bounded concurrency and rate limits to avoid overwhelming the database.

A hot key changes frequently. Key replication becomes inefficient because every update must refresh every copy. In this case, replication should be replaced with local caching or a different cache hierarchy.

Traffic suddenly increases after a marketing campaign. CDN caching absorbs public traffic, while local memory prevents millions of identical Redis requests for shared metadata.

Monitoring

Stampedes and hot keys often develop gradually before becoming visible as outages. Monitoring should identify growing traffic concentration before databases or Redis clusters become saturated.

Useful request tracing includes the layer that served each request.

X-Cache-Layer: LOCAL
X-Cache-Layer: REDIS
X-Cache-Layer: DATABASE

Combining cache-layer metrics with request latency makes it easier to identify where bottlenecks originate. A sudden increase in database-served requests after TTL expiration usually indicates a developing stampede.

Common Mistakes

Production Checklist

• Use request coalescing for expensive cache rebuilds.
• Add TTL jitter to every distributed cache.
• Protect databases with bounded fallback concurrency.
• Monitor the most frequently accessed cache keys.
• Use local caching for small shared metadata.
• Keep distributed-lock expiration shorter than rebuild timeouts.
• Use stale-while-revalidate for latency-tolerant workloads.
• Warm caches progressively after deployments.
• Load-test cache expiration during peak traffic.
• Alert on Redis CPU, rebuild latency, and hot-key traffic.
• Track cache source (local, Redis, database) in traces.
• Limit replication to stable read-heavy values.
• Retain finite TTLs even with explicit invalidation.
• Document acceptable stale windows for each workload.
• Regularly review hot-key distribution as traffic patterns change.

Conclusion

Cache stampedes and hot keys are common scaling challenges that appear only under production traffic. Request coalescing, stale-while-revalidate, and TTL jitter protect databases during cache rebuilds, while local caching, replication, and key sharding distribute traffic away from overloaded cache nodes.

The most resilient architectures combine several techniques instead of relying on one solution. Each mechanism addresses a different failure mode, and together they maintain low latency while protecting both cache infrastructure and authoritative storage.

Key Takeaway: Design caches for failure, not only for cache hits. Prevent duplicate rebuilds, distribute traffic away from popular keys, stagger expiration times, and ensure backend systems remain protected even when cache layers experience heavy load or temporary failures.

More Articles to Read

• Caching Explained: Improving Performance Without Overloading Databases
• Cache-Aside vs Write-Through vs Write-Behind
• CDN vs Reverse Proxy vs Application Cache
• Designing Multi-Level Caching Architectures
• Cache Invalidation Strategies for Production Systems
• Caching Best Practices for Distributed Applications