Caching Best Practices for Production Systems

Caching improves latency and reduces pressure on databases, APIs, storage systems, and expensive computations by keeping frequently needed data closer to where it is consumed. In production, however, adding a cache also creates another copy of data that can become stale, inconsistent, overloaded, or unavailable.

Effective caching is therefore not about caching as much data as possible. It is about choosing what to cache, where to cache it, how long it may remain stale, how it is invalidated, and what happens when the cache fails.

Table of Contents

• Cache the Right Data
• Choose the Right Cache Layer
  • Local In-Memory Cache
  • Distributed Cache
  • Edge and CDN Cache
• Choose a Caching Pattern
  • Cache-Aside
  • Read-Through Cache
  • Write-Through and Write-Behind
• Design Cache Keys Carefully
• Set TTL Based on Acceptable Staleness
• Invalidate Cache Safely
  • Invalidate After the Database Commit
  • Version Cache Keys When Useful
• Prevent Cache Stampedes
• Protect Against Cache Penetration
• Avoid Hot-Key Bottlenecks
• Design for Cache Failures
• Plan Capacity and Eviction
• Monitor Cache Effectiveness
• Common Production Mistakes
• Production Checklist
• Conclusion
  • More Articles to Read

Cache the Right Data

The first caching decision is not which technology to use. It is whether the workload benefits from caching at all.

Good cache candidates are usually expensive to obtain and frequently reused. Examples include product records, configuration, user permissions, search results, rendered pages, database aggregations, exchange-rate snapshots, and frequently requested API responses.

Consider an endpoint that performs a database aggregation taking 180 ms and receives 5,000 requests per second while the underlying data changes once every few minutes. Caching the result for even 10 seconds can eliminate tens of thousands of repeated calculations.

By contrast, caching a record that is requested once and never reused adds a cache read and write without avoiding meaningful backend work.

A useful rule is: cache because repeated backend work is expensive, not because a cache is available.

Choose the Right Cache Layer

Applications can cache at several layers. The closer cached data is to the consumer, the lower the lookup latency, but coordination and invalidation usually become harder.

Local In-Memory Cache

A local cache lives inside an application process. It can provide extremely low latency because no network request is required.

from functools import lru_cache


@lru_cache(maxsize=1000)
def get_country_configuration(country_code: str) -> dict:
    return load_country_configuration(country_code)

This works well for small, frequently reused, slowly changing data such as configuration or reference information.

The disadvantage appears after horizontal scaling. Ten application instances now contain ten independent copies, and invalidating one does not invalidate the others.

Local caches should therefore be used when temporary inconsistency between instances is acceptable or when a reliable invalidation mechanism exists.

Distributed Cache

A distributed cache such as Redis gives many application instances access to shared cached state.

Application Instances → Distributed Cache → Database

This improves consistency between application instances and allows cache capacity to scale independently from application memory.

The trade-off is network latency and another infrastructure dependency. A local dictionary lookup may take microseconds, while a distributed-cache operation requires network communication, serialization, and server processing.

Distributed caches are a strong fit for database objects, sessions, rate-limit counters, expensive query results, and other state shared across application instances.

Edge and CDN Cache

Public content can often be cached before a request reaches the application at all.

CDNs are especially effective for static assets, public HTML, images, downloadable files, and cacheable API responses.

Edge caching reduces both application latency and origin traffic because requests are served geographically closer to clients.

Cache keys must account for dimensions that change the response. If content varies by language, authentication, device type, query parameter, or another request attribute, an incomplete cache key can return the wrong representation to a client.

Choose a Caching Pattern

The relationship between the application, cache, and database determines how cache misses and writes behave.

Cache-Aside

Cache-aside is one of the most common application caching patterns.

1. Read from the cache.
2. If the value exists, return it.
3. Otherwise read from the database.
4. Store the result in the cache.
5. Return it.

import json


def get_product(product_id: int, cache, repository) -> dict | None:
    key = f"product:{product_id}"

    cached = cache.get(key)
    if cached is not None:
        return json.loads(cached)

    product = repository.find(product_id)
    if product is None:
        return None

    cache.setex(
        key,
        300,
        json.dumps(product),
    )

    return product

Cache-aside keeps the cache optional: the database remains authoritative and missing cache entries can be reconstructed.

The main challenges are stale values and concurrent cache misses.

Read-Through Cache

With read-through caching, the application reads through a caching abstraction that loads missing values from the backing store.

This simplifies application code and centralizes caching behavior, but the cache layer becomes more tightly coupled to data loading.

Read-through is useful when many services need the same predictable loading behavior and the caching infrastructure supports it cleanly.

Write-Through and Write-Behind

With write-through, a write updates the cache and persistent store as part of the write path. Reads then have a high probability of finding fresh data in cache.

The disadvantage is additional write latency and more coordination between storage layers.

With write-behind, writes are accepted into a fast layer and persisted asynchronously.

This can dramatically improve write throughput, but it changes the durability model. A cache failure before persistence can lose acknowledged data unless the intermediate layer itself is durable.

Write-behind should therefore be used only when its failure semantics are explicitly acceptable.

Design Cache Keys Carefully

A cache key is part of the application's data model. Poor key design causes collisions, difficult invalidation, accidental data exposure, and operational problems.

A useful key often includes namespace, entity, identifier, and version:

product:v3:48291
user-permissions:v2:841
search:v4:us:database:indexing

Multi-tenant applications should usually include tenant identity:

tenant:981:product:48291

Without it, two tenants using the same local object identifier could accidentally share cached data.

Do not include unnecessary high-cardinality values. For example, caching search responses by the complete raw request body can generate millions of entries that are never reused.

Keys should also remain reasonably short. Large keys consume memory and increase network payloads without improving cache effectiveness.

Set TTL Based on Acceptable Staleness

TTL should represent a business freshness requirement rather than an arbitrary default such as five minutes for every key.

Different data deserves different expiration policies:

Long TTLs increase hit ratio but increase the maximum duration of stale data. Short TTLs improve freshness but increase backend load.

TTL is therefore a consistency and capacity control, not merely a cleanup setting.

When many keys are created together, avoid giving all of them exactly the same expiration time. Otherwise they may expire simultaneously and create a burst of backend traffic.

Add expiration jitter:

import random


def cache_ttl(base_ttl: int, jitter: int = 30) -> int:
    return base_ttl + random.randint(0, jitter)

A 300-second TTL might therefore become 300–330 seconds across different entries, spreading refresh traffic over time.

Invalidate Cache Safely

Expiration limits how long stale values survive, but many systems need changes to become visible before the TTL expires.

Cache invalidation should be designed as part of the write path rather than added after stale-data bugs appear.

Invalidate After the Database Commit

For cache-aside, a common write pattern is:

1. Update the database.
2. Commit the transaction.
3. Delete the cached value.

Deleting before the database commit creates a race:

1. Writer deletes the cache.
2. Reader sees a cache miss.
3. Reader loads the old database value because the transaction has not committed.
4. Reader caches that old value again.
5. Writer commits.

The cache now contains stale data until another invalidation or expiration occurs.

Invalidating after commit reduces this window.

There is still a failure possibility between database commit and cache deletion. For important data, an event-driven invalidation mechanism or transactional outbox can make invalidation retryable.

Version Cache Keys When Useful

Some deployments can avoid mass deletion by changing the cache namespace.

For example:

catalog:v17:category:phones

After a major catalog change:

catalog:v18:category:phones

New requests immediately use the new namespace while old entries disappear naturally through expiration.

This works particularly well for deployments, static datasets, generated pages, and other groups of related cache entries.

Prevent Cache Stampedes

A cache stampede occurs when a popular key expires and many requests simultaneously discover the miss.

Suppose a cached analytics query normally receives 20,000 requests per second. When the key expires, thousands of requests may execute the same expensive database query at once.

The cache was intended to protect the database, but its expiration can suddenly overload it.

Several techniques reduce this risk.

• Expiration jitter. Prevent many related keys from expiring simultaneously.
• Request coalescing. Allow one request to rebuild a missing value while others wait briefly or use stale data.
• Stale-while-revalidate. Continue serving an older value while one worker refreshes it.
• Proactive refresh. Refresh important values before they expire.

A simplified single-flight pattern can use a short lock:

def get_expensive_report(report_id: str, cache, repository):
    key = f"report:{report_id}"
    lock_key = f"lock:{key}"

    cached = cache.get(key)
    if cached is not None:
        return cached

    acquired = cache.set(
        lock_key,
        "1",
        nx=True,
        ex=10,
    )

    if acquired:
        try:
            report = repository.generate(report_id)
            cache.setex(key, 60, report)
            return report
        finally:
            cache.delete(lock_key)

    return wait_for_cached_value(cache, key)

The lock should have a timeout so a crashed worker cannot permanently prevent refresh.

For latency-sensitive systems, serving slightly stale data is often safer than making thousands of requests wait behind one refresh.

Protect Against Cache Penetration

Cache penetration occurs when requests repeatedly ask for values that do not exist.

Consider an attacker or buggy client requesting millions of random product IDs. Every request misses the cache and reaches the database because there is no valid object to cache.

One solution is negative caching.

NOT_FOUND = "__NOT_FOUND__"


def find_product(product_id: int, cache, repository):
    key = f"product:{product_id}"

    cached = cache.get(key)

    if cached == NOT_FOUND:
        return None

    if cached is not None:
        return cached

    product = repository.find(product_id)

    if product is None:
        cache.setex(key, 30, NOT_FOUND)
        return None

    cache.setex(key, 300, product)
    return product

Negative entries should generally use shorter TTLs than existing objects because a missing resource may be created later.

Input validation and API rate limiting provide additional protection when random or malicious requests create excessive miss traffic.

Avoid Hot-Key Bottlenecks

A distributed cache can be horizontally partitioned and still suffer from one extremely popular key.

Suppose homepage:global receives 500,000 reads per second. Hash partitioning sends that key to one cache shard, so adding more unrelated shards does not reduce load on the node that owns it.

Possible mitigations include:

• small local caches in application instances;
• CDN caching for public content;
• replica reads where consistency permits;
• short-lived request-level caching;
• replicating immutable hot values under controlled conditions.

A two-level cache is particularly useful:

Application Memory → Distributed Cache → Database

The local cache absorbs extremely frequent reads while the distributed cache provides shared state when the local copy expires.

The trade-off is another consistency layer. Each local cache can temporarily contain a different version.

Design for Cache Failures

A cache should improve system reliability, not become a mandatory dependency that takes the application down whenever it is unavailable.

For cache-aside data, a cache outage can often fall back to the database:

try:
    cached = cache.get(key)
except CacheUnavailable:
    cached = None

if cached is not None:
    return cached

return repository.find(entity_id)

But unrestricted fallback can be dangerous. If a cache normally absorbs 95% of 100,000 requests per second, losing it could suddenly send nearly the entire workload to a database sized for only a small fraction of that traffic.

This creates a cache failure cascade.

Protection can include:

• rate limiting fallback traffic;
• load shedding non-critical requests;
• serving stale values;
• database query timeouts;
• request coalescing;
• gradual cache warming after recovery.

Failure isolation and load shedding are discussed further in Circuit Breaker vs Bulkhead vs Load Shedding.

Recovery also deserves attention. If a large cache restarts empty, immediately restoring full application traffic can create the same database overload as an outage. Warm important keys first or allow traffic to increase gradually.

Plan Capacity and Eviction

Cache capacity should be planned from actual object sizes, key cardinality, replication, allocator overhead, and expected growth rather than only the nominal payload size.

A simplified estimate is:

Memory ≈ active keys × average entry size × overhead × replication factor

If 20 million entries average 1 KB, the raw value data already requires roughly 20 GB before accounting for keys, metadata, fragmentation, replicas, and operational headroom.

When memory fills, the cache needs an eviction policy.

Common strategies include least recently used, least frequently used, TTL-based eviction, and policies that refuse new writes instead of evicting existing values.

The best policy depends on the workload. A cache containing millions of one-time search queries benefits from removing rarely reused entries, while a small set of expensive configuration objects may deserve explicit protection from eviction.

Eviction rate itself is an important signal. A sudden increase often means the working set has outgrown available memory or an application change has created unexpectedly high key cardinality.

Monitor Cache Effectiveness

A cache can be healthy from an infrastructure perspective while providing almost no application benefit.

CPU and memory alone do not answer whether caching is working.

Important metrics include:

• cache hit ratio by endpoint and data type;
• miss rate and backend requests caused by misses;
• p50, p95, and p99 cache latency;
• eviction rate;
• expiration rate;
• cache memory utilization;
• key count and growth rate;
• hot-key traffic;
• connection-pool utilization;
• cache errors and timeouts;
• backend latency on cache misses;
• database load during cache failures.

Hit ratio should be interpreted with workload context. A 95% hit ratio can still be poor if the remaining 5% represents thousands of expensive database queries per second.

Likewise, a 60% hit ratio can be extremely valuable when each hit avoids several seconds of computation.

The more useful question is: how much backend cost and latency does the cache actually eliminate?

Production monitoring strategies are covered more broadly in Observability Best Practices for Production Systems.

Common Production Mistakes

Many cache incidents come from treating caching as a transparent performance optimization when it actually changes application behavior.

Production Checklist

Before relying on a cache in a production request path, verify both its performance value and its failure behavior.

• Define acceptable staleness. Set TTL and invalidation behavior from the data's freshness requirement.
• Keep authoritative data elsewhere. Treat ordinary caches as rebuildable unless the system explicitly provides durability guarantees.
• Design cache keys deliberately. Include tenant, version, and response dimensions when they affect correctness.
• Protect cache misses. Prevent mass expiration from turning into uncontrolled backend traffic.
• Cache missing values when appropriate. Use short negative TTLs for repeated nonexistent lookups.
• Add expiration jitter. Spread refresh work instead of synchronizing it.
• Plan for hot keys. Confirm that the hottest object does not exceed one shard's capacity.
• Test cache outages. Verify that fallback traffic does not overload databases or downstream services.
• Plan cold-start recovery. Avoid sending full production traffic through an empty cache immediately after restart.
• Monitor the miss path. Measure the latency and infrastructure cost created when caching does not help.
• Track memory growth and eviction. Detect working-set changes before the cache spends most of its time replacing entries.

Conclusion

Production caching is a trade-off between latency, backend capacity, freshness, memory, and operational complexity. A high cache hit ratio is useful only when the cached data is correct enough and failures do not expose downstream systems to uncontrolled traffic.

The strongest designs cache expensive reusable data, choose the appropriate cache layer, define TTL from acceptable staleness, invalidate after durable writes, protect against stampedes and penetration, and treat hot keys and cache outages as expected failure scenarios.

The central principle is simple: a cache should remove expensive work without becoming a new correctness or reliability problem. Design the miss path, invalidation path, and failure path as carefully as the cache hit itself.

More Articles to Read

• Circuit Breaker vs Bulkhead vs Load Shedding
• Observability Best Practices for Production Systems
• Scaling Stateless Applications
• Reliability Best Practices for Production Systems