What Is a Hot Key?
A hot key is a cache, database, or distributed-storage key that receives disproportionately high traffic compared with other keys. Even when the overall system has enough capacity, one heavily accessed key can overload the specific node, partition, shard, or resource responsible for it.
Hot keys are common in systems with uneven access patterns: viral content, popular products, global configuration, trending posts, celebrity profiles, counters, leaderboards, or any resource that suddenly attracts a large percentage of total traffic.
Table of Contents
- How Hot Keys Happen
- Why Hot Keys Are Dangerous
- Hot Key vs Cache Stampede
- Detecting Hot Keys
- Handling Hot Read Keys
- Handling Hot Write Keys
- Partitioning and Hot Keys
- Expiration and Hot Keys
- Failure Scenarios
- Production Design Example
- Common Hot Key Mistakes
- Production Checklist
- Frequently Asked Questions
- Conclusion
How Hot Keys Happen
Distributed systems typically spread keys across multiple nodes so traffic and storage are distributed.
key:A ──> Node 1
key:B ──> Node 2
key:C ──> Node 3
key:D ──> Node 4This works well when traffic is reasonably balanced. Four nodes handling 40,000 requests per second might each receive approximately 10,000 requests per second.
Real workloads are rarely perfectly uniform.
product:101 → 1,200 req/s
product:102 → 900 req/s
product:103 → 1,100 req/s
product:999 → 30,000 req/sIf product:999 maps to Node 3, adding unused capacity to the other nodes does not necessarily help:
Node 1 Node 2 Node 3 Node 4
15% 18% 98% 14%
^
|
product:999The cluster appears lightly utilized on average, but one node is saturated.
This is the central hot-key problem: traffic distribution is determined not only by the number of keys, but also by how frequently each key is accessed.
Why Hot Keys Are Dangerous
A hot key creates a localized bottleneck. The affected resource reaches its capacity before the overall system reaches its theoretical capacity.
Depending on the architecture, the overloaded resource might be:
- a Redis node;
- a database partition;
- a DynamoDB partition;
- a storage shard;
- a message broker partition;
- a lock protecting one resource;
- a single application worker responsible for a key.
The immediate symptoms often include higher latency, connection saturation, CPU spikes, throttling, timeouts, and uneven node utilization.
Hot Key
|
v
One partition overloaded
|
v
Latency increases
|
v
Requests accumulate
|
v
Timeouts
|
v
Retries
|
v
Even more trafficRetries can make the problem significantly worse. A partition that is already overloaded may receive additional attempts from clients responding to increased latency.
This can eventually turn a localized hot-key problem into a wider service failure.
Hot Key vs Cache Stampede
Hot keys and cache stampedes are closely related, but they describe different failure modes.
| Problem | What Happens | Primary Bottleneck |
|---|---|---|
| Hot key | One key receives disproportionately high traffic | Node or partition serving the key |
| Cache stampede | Many requests regenerate the same missing or expired key | Backend dependency |
A hot key can exist while the cache is operating normally:
100,000 requests
|
v
Hot Key
|
v
One cache nodeA cache stampede happens when the cached value disappears:
Hot Key expires
|
v
Many cache misses
/ | | \
v v v v
DB DB DB DBThis makes hot keys particularly dangerous around expiration. A key already receiving extreme traffic can suddenly redirect that traffic toward a database or downstream service.
Preventing Cache Stampedes and Hot Keys covers the combined production problem and mitigation strategies in more depth.
Detecting Hot Keys
Cluster-wide averages are often insufficient for detecting hot keys. A cluster with average CPU utilization of 25% can still contain one node running at 100%.
Metrics should therefore be examined at multiple levels:
| Metric | What It Can Reveal |
|---|---|
| Requests per key | Individual hot keys |
| Requests per partition | Uneven traffic distribution |
| CPU per node | Localized compute saturation |
| Network throughput per node | Read-heavy hot spots |
| Latency per partition | Overloaded partitions |
| Throttled operations | Capacity limits being reached |
| Cache misses per key | Hot-key expiration or eviction problems |
For very large keyspaces, collecting metrics for every key may be too expensive. Sampling, top-N heavy-hitter tracking, approximate frequency algorithms, or provider-specific hot-key diagnostics can be used instead.
A useful operational question is:
Which 10 keys are responsible
for the most traffic right now?Another important signal is partition skew:
Partition A 12k req/s
Partition B 11k req/s
Partition C 79k req/s ← investigate
Partition D 13k req/sMonitoring only aggregate throughput can hide this imbalance.
Handling Hot Read Keys
Read-heavy hot keys are usually easier to mitigate because the same value can often be replicated or cached in multiple places.
The goal is to stop every request from reaching one shared resource.
Local Caching
A small in-process cache can absorb repeated reads before they reach a distributed cache.
Clients
|
v
Application Instances
| | |
v v v
L1 L1 L1
Cache Cache Cache
\ | /
\ | /
RedisSuppose 20 application instances each receive requests for the same product. Without local caching, every request reaches Redis. With a short local TTL, each application instance may query Redis only occasionally.
A simple implementation could look like:
import time
local_cache = {}
def get_local(key: str):
entry = local_cache.get(key)
if entry is None:
return None
value, expires_at = entry
if expires_at <= time.time():
local_cache.pop(key, None)
return None
return value
def set_local(key: str, value, ttl: float):
local_cache[key] = (
value,
time.time() + ttl,
)Local caching reduces distributed-cache traffic but introduces another layer of stale data and invalidation complexity.
For systems where multiple cache layers are useful, Designing Multi-Level Caching Architectures covers the broader architecture.
Replication
Read traffic can sometimes be spread across multiple copies of the value.
Hot Value
/ | \
v v v
Replica Replica Replica
1 2 3
\ | /
\ | /
ReadsThis works best for read-heavy workloads where updates are relatively infrequent.
Replication introduces consistency questions. If the value changes, all copies may not become current simultaneously. The design must define whether temporary staleness is acceptable.
Request Coalescing
If a hot key is missing or needs regeneration, request coalescing prevents many requests from performing the same expensive operation.
R1 ──+
R2 ──+
R3 ──+──> One refresh ──> Backend
R4 ──+ |
R5 ──+ v
ResultThis does not reduce normal read traffic for an existing hot value, but it protects the backend during refresh and expiration.
It is therefore commonly combined with local caching, replication, or stale-while-revalidate behavior.
Handling Hot Write Keys
Write-heavy hot keys are harder because writes usually need coordination around a single logical value.
Consider a counter for a viral post:
post:481:views
+1
+1
+1
+1
+1
...
Millions of updatesIf every request updates the same storage key, that key can become the system's serialization point.
Write Sharding
Instead of maintaining one physical counter, writes can be distributed across several keys:
Logical counter:
post:481:views
Physical counters:
post:481:views:0
post:481:views:1
post:481:views:2
post:481:views:3
post:481:views:4Each update selects a shard:
import random
SHARDS = 16
def counter_key(post_id: str) -> str:
shard = random.randrange(SHARDS)
return f"post:{post_id}:views:{shard}"The logical value is calculated by summing the shards:
Shard 0 = 18,240
Shard 1 = 17,991
Shard 2 = 18,402
Shard 3 = 18,103
-------
Total = 72,736This trades simpler reads for scalable writes.
Write sharding works particularly well for commutative values such as counters where partial results can be combined.
Batching and Aggregation
Another option is to avoid writing every individual update directly to the final key.
Events
|
v
Buffer / Queue
|
v
Aggregate
|
v
Periodic UpdateInstead of 50,000 counter increments per second reaching the database, application workers can aggregate increments and periodically write a larger delta.
+1 +1 +1 +1 +1 +1 +1
|
v
aggregate
|
v
+7The trade-off is that the stored value may temporarily lag behind real-time activity.
Partitioning and Hot Keys
Partitioning spreads data across multiple nodes, but it does not automatically spread traffic for one key.
With hash-based partitioning:
hash(key) → partitionEvery request for the same key generally maps to the same partition.
Even a well-balanced key distribution can therefore have an unbalanced traffic distribution:
Keys per partition:
P1 ██████████
P2 ██████████
P3 ██████████
P4 ██████████
Traffic per partition:
P1 ██
P2 ██
P3 ██████████████████
P4 ██Adding more partitions can help general capacity but does not necessarily split one hot key.
This distinction is important when designing partition keys. What Is Consistent Hashing? explains how keys can be distributed across changing sets of nodes, but a hot key still requires traffic-aware mitigation.
Expiration and Hot Keys
Expiration is a particularly dangerous moment for read-heavy hot keys.
Consider a cached value receiving 20,000 requests per second:
20,000 req/s
|
v
Cached Key
|
v
Backend sees almost nothingWhen the key expires, concurrent requests can suddenly attempt to reconstruct it.
Hot Key expires
|
v
20,000 req/s
|
v
Cache misses
|
v
Backend overloadThe system has moved from a hot-key problem to a cache stampede.
Common protection includes:
- request coalescing;
- stale-while-revalidate;
- refresh locks;
- proactive refresh;
- TTL jitter for groups of related keys.
The production techniques for this failure mode are covered in Caching Best Practices for Production Systems.
Failure Scenarios
Hot-key mitigation should define behavior when individual protection mechanisms fail.
Local cache expires everywhere simultaneously. Application instances can suddenly send a synchronized burst to the distributed cache. Adding small TTL jitter helps spread refreshes.
Distributed cache node fails. A hot key may move to another node, transferring its heavy traffic and potentially overloading the replacement.
Read replica becomes unhealthy. Redirecting all traffic to fewer replicas can make the remaining copies hot.
Write shards become uneven. Poor shard selection can simply move the hotspot from one key to one shard.
Traffic suddenly changes. Yesterday's normal key can become today's hot key after a viral post, promotion, breaking event, or customer behavior change.
Static assumptions about traffic distribution are therefore insufficient. Hot-key protection needs runtime visibility and capacity limits.
Production Design Example
Consider a news platform where one article becomes viral and receives 100,000 requests per second.
The application uses Redis as a distributed cache:
Clients
|
v
Load Balancer
|
+----------+----------+
| | |
v v v
API-1 API-2 API-3
| | |
+----------+----------+
|
v
Redis Cluster
/ | \
v v v
Node A Node B Node C
^
|
article:9182The viral article maps to Node B. Most other traffic remains evenly distributed, but Node B approaches its network and CPU limits.
Adding more Redis nodes alone may not solve the immediate problem because the same key still has one primary location.
A multi-level caching strategy can reduce the pressure:
100,000 req/s
|
v
Application Instances
|
v
Short-TTL Local Cache
|
| remaining requests
v
Redis Cluster
|
| occasional misses
v
DatabaseSuppose there are 50 application instances and each keeps the article locally for five seconds.
Instead of every request reaching Redis, each instance needs to refresh its local copy only periodically. The distributed cache traffic for the hot key can drop dramatically.
The local TTL can include jitter:
import random
def local_ttl() -> float:
return 5 + random.uniform(-1, 1)This prevents all application instances from refreshing the local value at exactly the same instant.
For expiration, the system can keep a stale copy available while one request refreshes the value:
Local value becomes stale
|
+---- return stale value
|
+---- one refresh request
|
v
Redis
|
v
Local cache updatedThe article view counter requires a different strategy because it is write-heavy.
Instead of one key:
article:9182:viewsthe system can use multiple counter shards:
article:9182:views:0
article:9182:views:1
article:9182:views:2
...
article:9182:views:31Writes are distributed across the shards, while an asynchronous aggregation process calculates the displayed total.
This illustrates an important distinction:
| Hot-Key Workload | Typical Strategy |
|---|---|
| Repeated reads | Replication, local caching, multi-level caching |
| Expensive regeneration | Request coalescing, stale-while-revalidate |
| Counter writes | Sharded counters, batching |
| Strict serialized writes | Capacity control, redesign, or explicit serialization |
Useful production metrics include per-key request frequency, top hot keys, traffic per cache node, cache hit rate, local-cache hit rate, throttling, node CPU, network utilization, and backend queries per hot key.
Common Hot Key Mistakes
- Looking only at cluster averages. One node can be saturated while average utilization remains low.
- Assuming hashing guarantees balanced traffic. Hashing balances keys, not request popularity.
- Adding partitions without analyzing the hot key. A single key may still map to one partition.
- Treating reads and writes identically. Replication helps reads but does not automatically solve write contention.
- Using unlimited retries. Retries amplify traffic toward an already overloaded resource.
- Ignoring key expiration. A hot key can become a severe cache stampede when it expires.
- Adding local caching without considering staleness. Multiple application instances may temporarily serve different values.
- Using identical local-cache TTLs. All instances can refresh the same hot key simultaneously.
- Sharding counters without a merge strategy. Reads become expensive or inaccurate if aggregation is poorly designed.
- Hard-coding known hot keys. Traffic patterns change and new hot keys appear dynamically.
Production Checklist
- Measure request distribution across keys.
- Monitor traffic per node and partition.
- Track top-N high-frequency keys.
- Alert on severe partition skew.
- Separate read-hot and write-hot workloads.
- Use local caching for extremely popular read-only or read-heavy values when appropriate.
- Define acceptable local-cache staleness.
- Add TTL jitter across application instances.
- Use request coalescing during regeneration.
- Consider stale-while-revalidate for suitable data.
- Replicate read-heavy values where the storage model allows it.
- Shard high-frequency counters when exact immediate aggregation is unnecessary.
- Batch high-volume writes where possible.
- Protect overloaded dependencies with concurrency limits.
- Use bounded retries and backoff.
- Define behavior when a hot-key node fails.
- Test sudden traffic concentration on one key.
- Test hot-key expiration under load.
- Test cache-node failure while a key is hot.
- Monitor whether mitigation shifts the hotspot to another layer.
Frequently Asked Questions
Hot keys are fundamentally about uneven workload distribution. The best solution depends on whether the key is read-heavy, write-heavy, cached, replicated, partitioned, or required to maintain strict consistency.
Are Hot Keys Only a Cache Problem?
No. Hot keys can occur in caches, databases, key-value stores, message brokers, storage systems, and other partitioned architectures.
Any design where one logical key or partition receives disproportionately high traffic can experience a hotspot.
Does Consistent Hashing Solve Hot Keys?
No. Consistent hashing helps distribute keys across nodes and minimizes key movement when nodes are added or removed.
It does not automatically split the traffic for one extremely popular key across multiple nodes. A single hot key can still overload the node responsible for it.
Do More Replicas Solve a Hot Key?
Replicas can help read-heavy hot keys if requests can be distributed across them. They are less useful for a write-heavy key when writes still need to reach a single leader or maintain strict coordination.
The consistency model also determines how safely reads can be spread across replicas.
Can One Key Really Overload a Large Cluster?
Yes. Cluster capacity is useful only when workload can be distributed across that capacity.
If one key maps to one partition and receives more traffic than that partition can handle, unused capacity elsewhere in the cluster does not automatically help.
Can Hot Keys Be Prevented Completely?
Not always. User behavior and traffic patterns can change unexpectedly, making previously ordinary keys hot.
A resilient design therefore combines prevention with detection: distribute predictable hotspots where possible, monitor workload skew, and have mechanisms that can react when new hot keys appear.
Conclusion
A hot key occurs when one key receives disproportionately high traffic and overloads the node, partition, or resource responsible for it. The overall system can have substantial unused capacity while one localized hotspot determines performance.
Read-heavy hot keys can often be mitigated with local caching, replication, multi-level caching, and request coalescing. Write-heavy hot keys may require sharded counters, batching, aggregation, or a redesign that distributes writes across multiple physical keys.
Key Takeaway: distributing data does not guarantee that traffic is distributed. Monitor workload skew, identify high-frequency keys, and choose mitigation based on whether the hotspot comes from reads, writes, regeneration, or partitioning.
Comments (0)