Scalability for Dummies - Part 3: Cache

By Oleksandr Andrushchenko — Published on — Modified on

In Scalability for Dummies - Part 2: Database, we optimized the database using indexes, connection pools, replicas, and other techniques. But even a healthy database can become overloaded when users repeatedly request the same products, articles, profiles, and configuration data. This is where caching becomes one of the most powerful scalability tools.

Table of Contents

The Problem

Imagine a product page that receives one million visits per day. The product information changes only a few times per month, but every page load still executes the same database query.

SELECT *
FROM products
WHERE id = 42;

That means one million requests can become one million identical database queries. The database is not doing new work; it is repeating the same work again and again.

The Same Data Again and Again

Many systems have a small amount of data that receives most of the traffic. Popular products, article metadata, user profiles, application settings, feature flags, and homepage content are common examples. Reading the same information thousands or millions of times from the database is wasteful.

Why Databases Suffer

Even a powerful database must accept connections, parse queries, read indexes, read pages from memory or disk, and return results. Doing this repeatedly for identical data is expensive. A cache exists to eliminate that repeated work.

What Is a Cache?

A cache is a fast storage layer that sits closer to the application than the database. Instead of asking the database every time, the application first checks the cache. If the data is already cached, the database is never touched.

Application
     │
     ▼
   Cache
     │
     ▼
 Database

Real-World Analogy

Imagine a chef cooking from a recipe book. The recipe book is the database. A small note with today's most-used recipes is the cache. Instead of opening the book every few seconds, the chef uses the note. The work becomes faster because the information is closer.

Cache Hit vs Cache Miss

A cache hit occurs when data already exists in the cache. The application can return the cached value immediately, and the database is not used.

Application
     │
     ▼
   Cache
     │
     ▼
Product Found

Database not used

A cache miss occurs when the cache does not contain the data. In that case, the application retrieves the data from the database and usually stores it in the cache for future requests.

Application
     │
     ▼
   Cache
     │
     ▼
 Not Found
     │
     ▼
 Database

Where Caches Live

Application Cache

The simplest cache lives inside the application process. It is extremely fast because everything happens in local memory, but each clone has its own cache.

API 1 -> Cache 1
API 2 -> Cache 2
API 3 -> Cache 3

This is useful for small datasets, but it does not scale well across many servers because cached data is duplicated and not shared between clones.

Distributed Cache

A distributed cache is shared by all application instances. Redis is the most common example. Now every clone can reuse cached data created by another clone.

API Clones
      │
      ▼
    Redis
      │
      ▼
   Database

This is usually the first serious caching layer in a growing application because it works well with horizontally scaled app servers.

CDN Cache

Some data can be cached even closer to users. CDNs such as CloudFront store content at edge locations around the world. Images, CSS, JavaScript files, and even full HTML pages can often be served without reaching your servers at all.

This is one of the most powerful forms of caching because it removes traffic not only from the database, but also from the application servers.

Cache Strategies

Cache-Aside

Cache-aside is the most common pattern. The application checks the cache first. If the value is missing, it loads the data from the database, stores the result in the cache, and returns it to the user.

Request
   │
   ▼
Check Cache
   │
   ├── Hit  -> Return Data
   │
   └── Miss
          │
          ▼
       Database
          │
          ▼
      Store Cache
          │
          ▼
      Return Data

Most web applications use some variation of this approach because it is simple, practical, and easy to introduce gradually.

Write-Through

With write-through caching, the application updates both the cache and the database when data changes. This keeps cached data fresher, but it increases write complexity and can add latency to writes.

This pattern is useful when stale reads are risky and the application needs the cache to reflect writes quickly.

TTL Expiration

TTL means Time To Live. Many caches automatically expire entries after a period of time. This is simple and often good enough for data that can be slightly stale.

  • User profile -> 5 minutes
  • Product catalog -> 1 hour
  • Application settings -> 24 hours

The shorter the TTL, the fresher the data. The longer the TTL, the more load is removed from the database.

Architecture Example

A typical system evolves from direct database access to layered caching. Each layer removes work from the layer below it, reducing latency and database pressure.

Phase 1

API
 │
 ▼
Database


Phase 2

API Clones
    │
    ▼
Database


Phase 3

API Clones
    │
    ▼
Redis Cache
    │
    ▼
Database


Phase 4

Users
  │
  ▼
CDN Cache
  │
  ▼
API Clones
  │
  ▼
Redis Cache
  │
  ▼
Database

The goal is not to cache everything. The goal is to avoid repeated expensive work on hot paths.

Common Mistakes

Not Measuring Cache Hit Rate

A cache with a 10% hit rate may provide little benefit. Always track hit ratio and miss ratio. If most requests still go to the database, the cache is either too small, incorrectly keyed, expiring too quickly, or caching the wrong data.

Caching Everything

Not all data benefits from caching. Frequently changing data may generate more cache churn than performance gains. Cache data that is read often, expensive to compute, and acceptable to serve slightly stale when needed.

Ignoring Invalidation

The hardest cache problem is keeping data fresh. A fast answer is useless if it is wrong. Before introducing caching, decide how cached data will be refreshed, expired, or invalidated when the source data changes.

Using the Cache as the Database

Caches are usually temporary. Treat the database as the source of truth. The cache should improve performance, not become the only copy of important business data.

Conclusion

Caching is one of the highest-impact scalability techniques available. Instead of making the database faster, caching avoids using the database in the first place. This reduces latency, increases throughput, and allows systems to handle dramatically more traffic.

The fastest query is not an optimized query. The fastest query is the one that never runs.

In Scalability for Dummies - Part 4: Asynchronism, we will explore another powerful scalability technique: moving work out of the request path entirely.

Comments (0)