Cache in Software System Design — A Practical Guide

Caching is one of the most effective techniques for improving performance, scalability, and availability in software systems. Instead of repeatedly fetching data from a slower source such as a database, API, object storage, or another service, a cache stores frequently used data in a faster layer.

This article explains where caches fit in system design, common caching strategies, invalidation techniques, trade-offs, failure modes, and practical production recommendations.

Table of Contents

• Caching Fundamentals
  • What is Caching?
  • Why Caching Matters
  • When Caching Helps Most
• Cache Layers
  • Client-Side Cache
  • CDN Cache
  • Application-Level Cache
  • Database Cache
  • Operating System Cache
  • Cache Layer Comparison
• Cache Placement in Architecture
  • Multi-Layer Cache Architecture
  • Edge Caching
  • Mid-Tier Caching
  • Data Caching
• Caching Strategies
  • Cache-Aside
  • Read-Through
  • Write-Through
  • Write-Behind
  • Refresh-Ahead
  • Strategy Comparison
• Cache Correctness
  • Cache Invalidation
  • TTL-Based Invalidation
  • Event-Based Invalidation
  • Version-Based Invalidation
  • Manual Invalidation
  • Invalidation Strategy Comparison
• Cache Design
  • Cache Keys and Granularity
  • Good Cache Key Patterns
  • Granularity Trade-off
  • Eviction Policies
• Production Cache Design
  • Cache Stampede
  • Cache Penetration
  • Cache Avalanche
  • Cold Start
  • Cache Outage
• Observability and Metrics
  • Important Cache Metrics
  • Hit Ratio Warning
• Real-World Examples
  • E-Commerce Product Page
  • Social Media Feed
  • Analytics Dashboard
• Production Recommendations
• Conclusion

Caching Fundamentals

What is Caching?

Caching is the process of temporarily storing frequently accessed or expensive-to-compute data in a faster storage layer. Instead of recomputing data or repeatedly fetching it from a database, external API, object storage, or another service, the application can serve it quickly from memory, edge nodes, or nearby storage.

For example, an e-commerce product page may read product details, prices, images, reviews, and recommendations from multiple systems. Without caching, every page view may trigger many expensive reads. With caching, frequently requested product data can be served from Redis, CDN, or application memory, reducing latency and load on backend systems.

User Request
  |
Application
  |
Check Cache
  |
+-------------------+
|                   |
Cache Hit           Cache Miss
|                   |
Return data         Load from database
                    Save to cache
                    Return data

Why Caching Matters

Caching matters because many systems are read-heavy. Users often request the same pages, objects, configurations, profiles, feeds, reports, or images many times. Caching reduces repeated work and helps systems handle more traffic without constantly scaling the database or expensive backend services.

When Caching Helps Most

Caching is most useful when data is read frequently, expensive to compute, expensive to fetch, or acceptable to serve slightly stale. It is less useful for data that changes constantly, requires strict correctness, or is rarely reused.

Rule of thumb: cache data when repeated reads are more expensive than the complexity of keeping the cached value reasonably fresh.

Cache Layers

Client-Side Cache

Client-side caching stores data in the browser, mobile app, or desktop client. Examples include browser HTTP cache, localStorage, IndexedDB, service workers, and mobile app storage. This is useful for static assets, user preferences, offline support, and data that does not need immediate freshness.

For example, a web application may cache CSS, JavaScript, fonts, and images in the browser. A mobile app may cache recent feed items so users can still see some content with poor network connectivity.

CDN Cache

CDN caching stores content at edge locations close to users. CDNs are commonly used for images, JavaScript, CSS, videos, public pages, and sometimes API responses. CDN caching improves latency and can keep content available even when the origin is temporarily unavailable.

User
  |
Nearest CDN Edge
  |
Origin Server
  |
Database

Application-Level Cache

Application-level caching stores computed results, query results, sessions, permissions, or hot objects in memory or a distributed cache such as Redis or Memcached. This is one of the most common caching layers in backend systems because it protects databases from repeated reads.

For example, an API can cache user profile data, permission checks, feature flags, or expensive dashboard summaries. This reduces database load and improves response time for repeated requests.

Database Cache

Databases already use internal caches such as buffer pools, page caches, query plan caches, and indexes. These caches help reduce disk access and speed up repeated operations. However, database cache alone may not be enough when the same expensive queries are executed repeatedly by the application.

Operating System Cache

Operating systems cache file system pages and disk blocks in memory. This helps speed up repeated reads from disk. It is usually invisible to the application, but it can significantly affect performance benchmarks and production behavior.

Cache Layer Comparison

Cache Placement in Architecture

Multi-Layer Cache Architecture

Cache placement determines what problem the cache solves. Edge caching reduces global latency. Application caching reduces repeated backend work. Database caching reduces disk access. A well-designed system often uses multiple cache layers, but each layer should have a clear purpose.

User
  |
Browser Cache
  |
CDN Cache
  |
Load Balancer
  |
Application
  |
Redis / Memcached
  |
Database
  |
Disk / Storage Cache

Edge Caching

Edge caching stores content close to users through a CDN. This is ideal for static files, public pages, images, documentation, blog posts, and cacheable API responses. For example, a system design blog can serve article pages and images from a CDN, reducing load on the origin application.

Edge caching is especially effective when many users request the same public content. The main trade-off is freshness: once content is distributed across edge locations, it may take time or explicit invalidation to update everywhere.

Mid-Tier Caching

Mid-tier caching uses reverse proxies, API gateways, or service-level caches between clients and backend systems. This can reduce repeated API calls and protect backend services from spikes.

For example, an API gateway can cache public catalog responses for a short time. This prevents every request from reaching the application layer while still keeping data reasonably fresh.

Data Caching

Data caching stores objects, query results, serialized responses, counters, sessions, or computed aggregates close to the application. This is useful when the database is becoming a bottleneck or when expensive computations are repeated often.

Caching Strategies

Cache-Aside

Cache-aside, also called lazy loading, is one of the most common caching patterns. The application checks the cache first. If the data is missing, the application loads it from the database, stores it in the cache, and returns it.

data = cache.get(key)

if data is None:
    data = database.query(key)
    cache.set(key, data, ttl=300)

return data

Cache-aside is simple and flexible, but the first request after expiration is slower because it must load data from the source. It works well for product details, user profiles, public pages, and computed summaries.

Read-Through

Read-through caching makes the cache responsible for loading missing data from the source. The application only talks to the cache. On a cache miss, the cache loads the data, stores it, and returns it.

This pattern simplifies application code but requires cache infrastructure or libraries that understand how to load data from the source. It is useful when many services need the same loading behavior.

Write-Through

Write-through caching writes data to the cache and the database in the same operation. This keeps the cache fresh because updates pass through the cache layer before being persisted.

Write-through improves consistency between cache and database, but write latency can increase because each write must update both systems. It is useful when reads are frequent and the cache should be immediately updated after writes.

Write-Behind

Write-behind, also called write-back caching, writes data to the cache first and persists it to the database asynchronously later. This can improve write latency and absorb write spikes, but it increases the risk of data loss if the cache fails before persistence.

This pattern should be used carefully for critical data. It may be acceptable for analytics counters, logs, temporary state, or non-critical event buffers, but it is risky for payments, orders, or financial records unless durability is guaranteed.

Refresh-Ahead

Refresh-ahead refreshes cache entries before they expire. This avoids latency spikes when popular keys expire and many users request the same data at the same time.

Refresh-ahead works well for predictable hot data, such as homepage content, product catalogs, trending feeds, exchange rates, or dashboard summaries. The trade-off is extra background work to refresh data that may not always be requested.

Strategy Comparison

Cache Correctness

Cache Invalidation

Cache invalidation ensures cached data does not become dangerously outdated. It is one of the hardest parts of caching because the cache and source of truth must stay reasonably synchronized while still preserving performance benefits.

There is no universal invalidation strategy. A blog post, product image, feature flag, account permission, and payment status all have different correctness requirements. The stricter the correctness requirement, the more carefully invalidation must be designed.

TTL-Based Invalidation

TTL, or time to live, expires entries after a fixed time. This is the simplest invalidation method and works well when some staleness is acceptable.

cache.set("product:123", product_data, ttl=300)

After 300 seconds, the key expires automatically.

TTL-based invalidation is often enough for public content, catalog pages, configuration values, and dashboards. The main weakness is that data may remain stale until the TTL expires.

Event-Based Invalidation

Event-based invalidation removes or updates cache entries when the source data changes. For example, when a product price changes, the product service can publish an event that invalidates the related cache keys.

Product Updated
  |
Publish Event
  |
Cache Invalidator
  |
Delete product:123
Delete category:shoes:page:1

This keeps cache entries fresher than TTL alone, but it introduces more moving parts. The system must handle missed events, duplicate events, delayed events, and partial invalidation failures.

Version-Based Invalidation

Version-based invalidation includes a version number, timestamp, or content hash in the cache key. When the data changes, the version changes, and the application naturally reads from a new key.

Old key:
post:123:v1

New key:
post:123:v2

This is useful for CDNs, static assets, articles, configuration, and systems where deleting old cache entries everywhere is difficult. Old keys can remain until they expire or are evicted.

Manual Invalidation

Manual invalidation means explicitly clearing cache entries through admin tools, deployment scripts, or operational commands. This is useful for emergency fixes, content publishing, and rare correction workflows, but it should not be the primary strategy for high-frequency updates.

Invalidation Strategy Comparison

Cache Design

Cache Keys and Granularity

Cache keys uniquely identify cached data. Poor key design can cause collisions, stale data, incorrect user data exposure, and low hit ratios. Good keys are descriptive, stable, namespaced, and include all important dimensions of the cached result.

Good Cache Key Patterns

user:123:profile
user:123:permissions:v4
post:456:comments:page:2
product:789:details:currency:USD
tenant:42:dashboard:2026-06-04
prod:api:v1:homepage

• Namespace by environment: use prefixes such as prod:, staging:, or dev:.
• Include tenant or user scope: avoid leaking data between customers.
• Include query parameters: page, filters, locale, currency, and permissions may change the result.
• Avoid huge values: smaller composable objects are often easier to invalidate and reuse.
• Use versioning when needed: include schema, content, or permission versions in keys.

Granularity Trade-off

Cache granularity controls how much data each key stores. Larger cached objects can make responses very fast, but they are harder to invalidate. Smaller cached objects are more precise, but they may require more cache lookups and more application-side composition.

Eviction Policies

Eviction happens when the cache removes entries to free memory or because entries expired. Eviction policy determines which keys are removed first. The right policy depends on access patterns and memory constraints.

Production Cache Design

Cache Stampede

A cache stampede happens when many requests miss the same key at the same time and all try to recompute or reload the value from the database. This can overload the origin system.

Popular key expires
  |
Thousands of requests miss cache
  |
All requests query database
  |
Database overload

Common solutions include request coalescing, locks, stale-while-revalidate, refresh-ahead, jittered TTLs, and serving stale data while one process refreshes the key.

Cache Penetration

Cache penetration happens when requests repeatedly ask for data that does not exist. Since the cache has no value for these keys, every request hits the database. This can happen because of bugs, scraping, or malicious traffic.

Common solutions include negative caching, Bloom filters, validation, and rate limiting.

Cache Avalanche

A cache avalanche happens when many keys expire at the same time, causing a sudden spike in database traffic. This often happens when many keys are written with the same TTL.

Common solutions include TTL jitter, staggered expiration, refresh-ahead, and warming important keys before major traffic events.

Cold Start

A cold cache has little or no useful data. This can happen after deployment, cache restart, failover, or scaling up a new cache cluster. Cold starts increase latency and backend load until the cache warms up.

For example, after a Redis cluster restart, a product catalog API may suddenly send all reads to the database. If the database was sized assuming cache protection, it may become overloaded.

Cache Outage

If the application depends too heavily on the cache, a cache outage can become a system outage. Critical systems should define whether the cache is optional or required. For many read caches, the application should be able to bypass the cache and query the database directly, possibly with rate limits.

Observability and Metrics

Important Cache Metrics

Caching should be measured. Without metrics, it is hard to know whether the cache is helping, wasting memory, hiding database problems, or serving stale data.

Hit Ratio Warning

A high hit ratio is not always enough. A cache can have a high hit ratio while still failing critical requests. For example, caching many cheap objects may produce good metrics, while expensive checkout or permission requests still miss frequently. Measure cache effectiveness by business-critical paths, not only globally.

For example, a system may report a 95% cache hit ratio because images and public pages are cached well. But if the checkout API misses cache on every permission check and overloads the database, users still experience failures. Measure cache performance by critical workflow, not only by global average.

Real-World Examples

E-Commerce Product Page

An e-commerce product page may cache product details, images, category metadata, reviews, and recommendations. However, inventory and price may require shorter TTLs or event-based invalidation because stale data can create bad user experiences.

Social Media Feed

A social media feed may cache timelines, trending posts, user profiles, media, and engagement counters. Some values, such as likes and views, can be eventually consistent. Other values, such as privacy settings, need stricter invalidation.

User opens feed
  |
Read cached timeline
  |
Fetch missing posts
  |
Load media from CDN
  |
Update view counters asynchronously

This design keeps the feed fast while allowing counters and analytics to update later. Privacy and blocking rules should be handled more carefully because stale access-control data can expose content incorrectly.

Analytics Dashboard

Analytics dashboards often cache expensive aggregations. For example, daily revenue, active users, conversion rate, or campaign performance can be precomputed and cached. Users usually accept a dashboard that is delayed by a few minutes if it loads quickly and consistently.

Production Recommendations

• Start with simple TTL-based cache-aside. It is easy to understand, test, and operate.
• Cache read-heavy and expensive data first. Do not cache everything by default.
• Choose TTL based on business correctness. Product descriptions can live longer than inventory or permissions.
• Use external caches for distributed systems. Redis or Memcached is usually better than local memory when multiple app instances need shared cache.
• Protect the database from stampedes. Use locks, request coalescing, refresh-ahead, stale-while-revalidate, or TTL jitter.
• Design cache keys carefully. Include tenant, user, locale, currency, permissions, filters, and version when they affect the result.
• Do not make cache the only source of truth unless designed for it. For critical data, the database or durable storage should remain authoritative.
• Monitor hit ratio, evictions, latency, and memory usage. Caching without observability is risky.
• Document keys, TTLs, and invalidation rules. Future engineers need to understand why cached data behaves the way it does.

Conclusion

Caching is a powerful system-design tool, but it is not just a performance trick. It affects scalability, availability, consistency, cost, and operational complexity.

For most systems, the best approach is to start with simple cache-aside caching, short and safe TTLs, clear key naming, and strong observability. Add event-based invalidation, refresh-ahead, write-through, or write-behind only when the workload and correctness requirements justify the extra complexity.

Key takeaway: cache data that is expensive to fetch or compute, read frequently, and safe to serve slightly stale. Avoid caching critical data without a clear invalidation and recovery strategy.