Caching Explained: Improving Performance Without Overloading Databases

Caching reduces latency and protects databases from repeatedly performing the same expensive work. A well-designed cache can serve high-volume reads in milliseconds while lowering database CPU, storage I/O, connection usage, and infrastructure cost.

A cache also creates another copy of data. That copy can become stale, disappear during a failure, overload under traffic spikes, or return inconsistent results. Production caching therefore requires explicit decisions about cache placement, keys, expiration, invalidation, failure recovery, and observability.

Table of Contents

• Why Caching Exists
• What Should Be Cached
  • Good Cache Candidates
  • Poor Cache Candidates
• Where Caches Live
  • Browser and CDN Caches
  • Application Memory Cache
  • Distributed Cache
• Cache Lifecycle
  • Cache Hit and Miss
  • Expiration and Freshness
  • Cache Keys
• Cache Performance and Database Protection
  • Hit Ratio
  • Negative Caching
  • Cache Failure Behavior
• Production Design Example
  • Architecture
  • Request Flow
  • Failure Scenarios
• Ready-to-Use Example
  • Redis Cache Service
  • FastAPI Endpoint
  • Cache Invalidation
  • Cache Monitoring
• Common Mistakes
• Production Checklist
• Conclusion
  • More Articles to Read

Why Caching Exists

Databases are designed for durability, transactions, indexing, concurrency control, recovery, and consistency. These guarantees make database operations more expensive than retrieving a value from memory.

A request reaching a relational database may require:

• acquiring a connection
• sending data over the network
• parsing and planning a query
• traversing one or more indexes
• reading database pages
• checking row visibility
• joining or aggregating records
• serializing the result

When thousands of requests repeatedly ask for the same product, configuration, permission set, or shipment summary, the database performs nearly identical work each time.

Without caching

Client
   |
   v
API service
   |
   v
Database connection pool
   |
   v
Database
   |
   v
Indexes and storage

Every request consumes database capacity.

With caching

Client
   |
   v
API service
   |
   +---- cache hit ----> Return response
   |
   +---- cache miss ---> Database
                            |
                            v
                       Store in cache
                            |
                            v
                       Return response

The cache does not make the database faster. It reduces how frequently the database must participate in the request.

Caching is most valuable when the saved work is expensive and the cached result is reused many times.

What Should Be Cached

A cache should contain data with a measurable reuse pattern and a defined freshness tolerance. Caching everything wastes memory, increases invalidation work, and can make the system less reliable.

Good Cache Candidates

Good candidates are read frequently, expensive to retrieve, and relatively stable.

• Reference data: countries, currencies, carrier codes, service types, tax categories.
• Configuration: feature flags, tenant settings, routing rules, application preferences.
• Frequently viewed entities: product details, public profiles, articles, shipment summaries.
• Computed results: dashboard totals, recommendations, pricing previews, route estimates.
• External API responses: exchange rates, geocoding results, address validation, provider capabilities.
• Short-lived authorization metadata: roles and permissions when brief staleness is acceptable.

A practical cache candidate can be evaluated using a simple reuse ratio:

reuse ratio =
    requests for the same value
    --------------------------------
    number of times the value changes

High reuse ratio  -> caching can be valuable
Low reuse ratio   -> caching may only waste memory

Poor Cache Candidates

Data should not be cached when stale values could violate business or security requirements.

• authoritative financial balances
• payment authorization results
• inventory quantities used to approve purchases
• security decisions requiring immediate revocation
• rapidly changing workflow state
• one-time query results with little reuse
• very large values that consume significant cache memory

A cache can still display an approximate balance or inventory count, but it must not become the source used to authorize the transaction.

Where Caches Live

Caches can exist at several layers. Each layer removes different work from the request path and introduces different invalidation and consistency concerns.

Browser and CDN Caches

Browser and CDN caches are closest to users. They can serve static assets and public responses without reaching the application infrastructure.

User
  |
  v
Browser cache
  |
  v
CDN edge cache
  |
  v
Reverse proxy
  |
  v
Application
  |
  v
Database

Typical candidates include:

• images
• JavaScript and CSS
• fonts
• public documentation
• public API responses
• generated files

These caches can remove network latency and application load, but personalized or authenticated responses require careful cache-key and privacy controls.

Application Memory Cache

An in-process cache stores values inside one application instance.

Advantages:

• no network request
• very low latency
• simple implementation
• useful for tiny frequently accessed objects

Disadvantages:

• each instance contains different cached state
• cache disappears during restarts
• memory competes with application memory
• invalidation must reach every instance

Local caches are suitable for small, short-lived values where temporary inconsistency between instances is acceptable.

Distributed Cache

A distributed cache such as Redis provides shared cached values for many application instances.

API instance A ----+
                   |
API instance B ----+---- Redis cluster
                   |
API instance C ----+
                   |
                   +---- Database

Advantages:

• all application instances use the same cached values
• memory is managed independently from application processes
• centralized expiration and invalidation
• supports atomic counters, locks, and conditional operations

Disadvantages:

• every operation requires a network call
• the cache becomes another production dependency
• large objects consume memory quickly
• hot keys can overload a small set of nodes
• failures can redirect sudden traffic to the database

A distributed cache should usually be treated as disposable. The application must remain correct when cached values are missing.

Cache Lifecycle

Every cache entry moves through a lifecycle: creation, reuse, expiration, invalidation, eviction, and recreation. Production behavior depends on how the application handles each stage.

Cache Hit and Miss

A cache hit returns a value without calling the database. A cache miss requires loading the authoritative value and usually storing it for later requests.

from __future__ import annotations

from dataclasses import dataclass
from typing import Protocol


class Cache(Protocol):
    async def get(self, key: str) -> str | None:
        ...

    async def set(
        self,
        key: str,
        value: str,
        ttl_seconds: int,
    ) -> None:
        ...


class ProductRepository(Protocol):
    async def get_by_id(
        self,
        product_id: int,
    ) -> dict[str, object] | None:
        ...


@dataclass(frozen=True)
class ProductCachePolicy:
    ttl_seconds: int = 300


class ProductReader:
    def __init__(
        self,
        cache: Cache,
        repository: ProductRepository,
        policy: ProductCachePolicy,
    ) -> None:
        self._cache = cache
        self._repository = repository
        self._policy = policy

    async def get(
        self,
        product_id: int,
    ) -> dict[str, object] | None:
        cache_key = f"product:v1:{product_id}"
        cached = await self._cache.get(cache_key)

        if cached is not None:
            return {"serialized": cached}

        product = await self._repository.get_by_id(product_id)

        if product is None:
            return None

        await self._cache.set(
            cache_key,
            str(product),
            ttl_seconds=self._policy.ttl_seconds,
        )
        return product

This is the basic cache-aside pattern. The next article in the series compares it with write-through and write-behind caching.

Expiration and Freshness

Time-to-live controls how long a cached value remains available before it must be refreshed.

Short expiration:

• improves freshness
• increases cache misses
• raises database traffic

Long expiration:

• reduces database load
• improves hit ratio
• increases stale-data exposure

from __future__ import annotations

from dataclasses import dataclass


@dataclass(frozen=True)
class CacheTTL:
    feature_flags: int = 30
    user_profile: int = 300
    product_details: int = 900
    exchange_rates: int = 3_600
    country_list: int = 86_400

Expiration should follow business tolerance. A feature flag may need refresh within seconds, while a country list can remain cached for a day.

Add random variation to large groups of cache entries so they do not expire simultaneously:

from __future__ import annotations

import random


def ttl_with_jitter(
    base_ttl_seconds: int,
    jitter_ratio: float = 0.15,
) -> int:
    if base_ttl_seconds <= 0:
        raise ValueError("base_ttl_seconds must be positive")

    maximum_jitter = int(base_ttl_seconds * jitter_ratio)
    return base_ttl_seconds + random.randint(
        -maximum_jitter,
        maximum_jitter,
    )

TTL jitter reduces synchronized expiration spikes that can overload the database.

Cache Keys

Cache keys define uniqueness, ownership, versioning, and invalidation scope. A weak key can return one tenant’s data to another tenant or mix incompatible representations.

A practical key format includes:

• entity type
• schema or response version
• tenant identifier
• entity identifier
• important query parameters

from __future__ import annotations

from dataclasses import dataclass


@dataclass(frozen=True)
class ShipmentCacheKey:
    account_id: int
    shipment_id: str
    representation_version: int = 1

    def build(self) -> str:
        return (
            f"shipment:v{self.representation_version}:"
            f"account:{self.account_id}:"
            f"id:{self.shipment_id}"
        )

Versioned keys allow schema changes without deserializing old incompatible values. Tenant identifiers prevent cross-tenant collisions.

Cache Performance and Database Protection

A cache should be measured by the work it removes from authoritative systems, not only by how quickly Redis responds.

Hit Ratio

The cache hit ratio measures the percentage of lookups served from the cache.

hit ratio =
    cache hits
    -------------------------
    cache hits + cache misses

A high hit ratio is useful only when cached values replace expensive work. A 99% hit ratio for a trivial database query may provide less value than a 60% hit ratio for a costly aggregation.

Monitor hit ratio by cache namespace rather than only as a global metric. A global value can hide one poorly performing cache behind several successful ones.

Negative Caching

Negative caching temporarily stores the fact that a record does not exist.

Without negative caching, repeated requests for an invalid product or unknown tracking number can continuously query the database.

from __future__ import annotations

import json
from typing import Any


NOT_FOUND_MARKER = {"found": False}


def encode_cache_value(
    value: dict[str, Any] | None,
) -> str:
    if value is None:
        return json.dumps(NOT_FOUND_MARKER)

    return json.dumps({
        "found": True,
        "value": value,
    })


def decode_cache_value(
    payload: str,
) -> tuple[bool, dict[str, Any] | None]:
    decoded = json.loads(payload)

    if not decoded["found"]:
        return False, None

    return True, decoded["value"]

Negative entries should use shorter TTLs because a missing record may be created soon after the lookup.

Cache Failure Behavior

A cache outage can be more dangerous than ordinary cache misses. When thousands of requests simultaneously fall back to the database, the database may become overloaded.

Normal traffic:

100,000 requests
    |
95,000 cache hits
    |
5,000 database reads


Cache outage:

100,000 requests
    |
100,000 database reads
    |
database saturation

Database protection during cache failure can include:

• request rate limits
• bounded database connection pools
• short database timeouts
• stale cached fallback values
• request coalescing
• circuit breakers
• load shedding for noncritical endpoints

The application must not retry cache operations indefinitely. A failing cache should not delay every request longer than reading directly from the database.

Production Design Example

Consider a logistics platform that displays shipment details to customers. Shipment records are stored in PostgreSQL and updated by carrier events. Customers frequently refresh active shipments, while delivered shipments change rarely.

The cache design should reduce repeated reads without using stale data to authorize business transitions.

Architecture

Client
   |
   v
FastAPI service
   |
   +---- local metadata cache
   |
   +---- Redis distributed cache
   |          |
   |          +---- cached shipment response
   |
   +---- PostgreSQL primary
   |          |
   |          +---- authoritative shipment state
   |
   +---- outbox events
              |
              v
        cache invalidation worker

PostgreSQL remains authoritative. Redis stores serialized shipment responses. An outbox-driven worker deletes or refreshes cached entries after shipment updates.

Request Flow

Read flow:

1. The API builds a tenant-scoped versioned cache key.
2. Redis is queried with a short timeout.
3. On a hit, the cached response is returned.
4. On a miss, PostgreSQL is queried.
5. The response is cached with TTL jitter.
6. Delivered shipments receive a longer TTL than active shipments.

Write flow:

1. A database transaction updates shipment state.
2. The same transaction inserts an outbox event.
3. A worker publishes the event after commit.
4. The invalidation consumer deletes the shipment cache key.
5. The next read reloads current state from PostgreSQL.

TTL policy:

Failure Scenarios

Redis becomes unavailable. The API uses a short cache timeout and falls back to PostgreSQL. Database concurrency limits prevent the fallback from exhausting all connections.

Invalidation event is delayed. The old value remains until its TTL expires. Short TTLs for active shipments limit stale-data duration.

Many popular entries expire together. TTL jitter spreads expiration times. Request coalescing prevents hundreds of requests from loading the same shipment simultaneously.

A hot shipment receives extreme traffic. Redis serves the value without reaching PostgreSQL. The key may still become hot at the cache-node level, requiring replication, local caching, or request distribution.

The database update commits but cache deletion fails. The cached value remains temporarily stale. The invalidation event is retried, and TTL provides a final recovery boundary.

Redis evicts values under memory pressure. Requests become misses and reload from PostgreSQL. Eviction-rate alerts reveal insufficient memory or overly broad caching.

Ready-to-Use Example

The following implementation provides a production-oriented Redis cache wrapper, FastAPI read path, event-driven invalidation, TTL jitter, negative caching, and metrics.

Redis Cache Service

from __future__ import annotations

import json
import logging
import random
from dataclasses import dataclass
from typing import Any

from redis.asyncio import Redis
from redis.exceptions import RedisError


logger = logging.getLogger(__name__)


@dataclass(frozen=True)
class CacheResult:
    hit: bool
    found: bool
    value: dict[str, Any] | None


class RedisJSONCache:
    def __init__(
        self,
        redis: Redis,
        namespace: str,
        default_ttl_seconds: int = 300,
        negative_ttl_seconds: int = 15,
    ) -> None:
        self._redis = redis
        self._namespace = namespace
        self._default_ttl_seconds = default_ttl_seconds
        self._negative_ttl_seconds = negative_ttl_seconds

    def build_key(
        self,
        account_id: int,
        entity_id: str,
        version: int = 1,
    ) -> str:
        return (
            f"{self._namespace}:v{version}:"
            f"account:{account_id}:"
            f"id:{entity_id}"
        )

    async def get(self, key: str) -> CacheResult:
        try:
            payload = await self._redis.get(key)
        except RedisError:
            logger.exception(
                "Cache read failed",
                extra={"cache_key": key},
            )
            return CacheResult(
                hit=False,
                found=False,
                value=None,
            )

        if payload is None:
            return CacheResult(
                hit=False,
                found=False,
                value=None,
            )

        decoded = json.loads(payload)

        if decoded["found"] is False:
            return CacheResult(
                hit=True,
                found=False,
                value=None,
            )

        return CacheResult(
            hit=True,
            found=True,
            value=decoded["value"],
        )

    async def set_value(
        self,
        key: str,
        value: dict[str, Any],
        ttl_seconds: int | None = None,
    ) -> None:
        ttl = self._with_jitter(
            ttl_seconds or self._default_ttl_seconds
        )

        try:
            await self._redis.set(
                key,
                json.dumps({
                    "found": True,
                    "value": value,
                }),
                ex=ttl,
            )
        except RedisError:
            logger.exception(
                "Cache write failed",
                extra={"cache_key": key},
            )

    async def set_not_found(self, key: str) -> None:
        try:
            await self._redis.set(
                key,
                json.dumps({"found": False}),
                ex=self._negative_ttl_seconds,
            )
        except RedisError:
            logger.exception(
                "Negative cache write failed",
                extra={"cache_key": key},
            )

    async def delete(self, key: str) -> None:
        try:
            await self._redis.delete(key)
        except RedisError:
            logger.exception(
                "Cache invalidation failed",
                extra={"cache_key": key},
            )
            raise

    @staticmethod
    def _with_jitter(
        ttl_seconds: int,
        jitter_ratio: float = 0.15,
    ) -> int:
        jitter = int(ttl_seconds * jitter_ratio)
        return max(
            1,
            ttl_seconds + random.randint(-jitter, jitter),
        )

Cache failures are logged but do not make the read path fail. Invalidation failures are raised because the event consumer should retry them.

FastAPI Endpoint

from __future__ import annotations

import asyncio
from dataclasses import asdict, dataclass
from typing import Protocol

from fastapi import FastAPI, HTTPException, status


@dataclass(frozen=True)
class Shipment:
    shipment_id: str
    account_id: int
    shipment_status: str
    carrier_code: str | None
    tracking_number: str | None
    version: int


class ShipmentRepository(Protocol):
    async def get(
        self,
        account_id: int,
        shipment_id: str,
    ) -> Shipment | None:
        ...


class ShipmentReader:
    def __init__(
        self,
        repository: ShipmentRepository,
        cache: RedisJSONCache,
    ) -> None:
        self._repository = repository
        self._cache = cache
        self._locks: dict[str, asyncio.Lock] = {}

    async def get(
        self,
        account_id: int,
        shipment_id: str,
    ) -> Shipment | None:
        cache_key = self._cache.build_key(
            account_id=account_id,
            entity_id=shipment_id,
        )

        cached = await self._cache.get(cache_key)

        if cached.hit:
            if not cached.found:
                return None

            return Shipment(**cached.value)

        # Coalesce simultaneous misses inside this application instance.
        lock = self._locks.setdefault(
            cache_key,
            asyncio.Lock(),
        )

        async with lock:
            cached = await self._cache.get(cache_key)

            if cached.hit:
                if not cached.found:
                    return None

                return Shipment(**cached.value)

            shipment = await self._repository.get(
                account_id=account_id,
                shipment_id=shipment_id,
            )

            if shipment is None:
                await self._cache.set_not_found(cache_key)
                return None

            ttl_seconds = (
                3_600
                if shipment.shipment_status == "delivered"
                else 60
            )

            await self._cache.set_value(
                key=cache_key,
                value=asdict(shipment),
                ttl_seconds=ttl_seconds,
            )

            return shipment


app = FastAPI()


@app.get("/accounts/{account_id}/shipments/{shipment_id}")
async def get_shipment(
    account_id: int,
    shipment_id: str,
) -> dict[str, object]:
    shipment = await shipment_reader.get(
        account_id=account_id,
        shipment_id=shipment_id,
    )

    if shipment is None:
        raise HTTPException(
            status_code=status.HTTP_404_NOT_FOUND,
            detail="Shipment not found",
        )

    return asdict(shipment)

The second cache lookup after acquiring the lock prevents another request from repeating the database query while waiting.

Cache Invalidation

Cache invalidation should occur after the authoritative transaction commits. A transactional outbox ensures the invalidation event is not lost.

BEGIN;

UPDATE shipments
SET status = 'delivered',
    version = version + 1,
    updated_at = CURRENT_TIMESTAMP
WHERE shipment_id = :shipment_id
  AND account_id = :account_id
  AND version = :expected_version
RETURNING shipment_id, account_id, version;

INSERT INTO outbox_events (
    event_id,
    aggregate_type,
    aggregate_id,
    aggregate_version,
    event_type,
    payload
)
VALUES (
    :event_id,
    'shipment',
    :shipment_id,
    :new_version,
    'shipment.updated',
    jsonb_build_object(
        'shipment_id', :shipment_id,
        'account_id', :account_id,
        'version', :new_version
    )
);

COMMIT;

from __future__ import annotations

from dataclasses import dataclass


@dataclass(frozen=True)
class ShipmentUpdatedEvent:
    event_id: str
    account_id: int
    shipment_id: str
    version: int


class ShipmentCacheInvalidator:
    def __init__(
        self,
        cache: RedisJSONCache,
    ) -> None:
        self._cache = cache

    async def handle(
        self,
        event: ShipmentUpdatedEvent,
    ) -> None:
        cache_key = self._cache.build_key(
            account_id=event.account_id,
            entity_id=event.shipment_id,
        )

        # Delete is naturally idempotent. Reprocessing the same
        # event produces the same result.
        await self._cache.delete(cache_key)

Deleting the value is often safer than immediately repopulating it because the next reader loads the latest committed representation.

Cache Monitoring

Monitor the cache together with database behavior. A falling hit ratio matters because it usually causes rising database traffic.

Important metrics include:

• cache hits and misses by namespace
• hit ratio
• Redis command latency
• cache connection failures
• evicted keys
• expired keys
• memory usage
• hot-key traffic
• database queries caused by cache misses
• cache fallback rate
• invalidation failures
• stale-data incidents

A minimal Python metrics wrapper can record cache outcomes:

from __future__ import annotations

from dataclasses import dataclass
from typing import Protocol


class Counter(Protocol):
    def increment(
        self,
        value: int = 1,
        **labels: str,
    ) -> None:
        ...


@dataclass(frozen=True)
class CacheMetrics:
    hits: Counter
    misses: Counter
    failures: Counter

    def record_hit(self, namespace: str) -> None:
        self.hits.increment(namespace=namespace)

    def record_miss(self, namespace: str) -> None:
        self.misses.increment(namespace=namespace)

    def record_failure(
        self,
        namespace: str,
        operation: str,
    ) -> None:
        self.failures.increment(
            namespace=namespace,
            operation=operation,
        )

Alerting should focus on changes from normal behavior. A sudden hit-ratio drop, eviction increase, or cache-latency spike can precede database saturation.

Common Mistakes

Production Checklist

• Cache only data with measurable reuse.
• Keep authoritative business state in the database.
• Use tenant-scoped, versioned cache keys.
• Define TTLs from freshness requirements.
• Add jitter to high-volume expiration times.
• Use short TTLs for negative caching.
• Set strict cache connection and command timeouts.
• Protect the database during cache failures.
• Coalesce concurrent misses for popular keys.
• Invalidate cached values only after database commit.
• Retry invalidation through durable events.
• Monitor hit ratio, evictions, latency, and failures.
• Track database traffic caused by cache misses.
• Load-test cache outages and expiration spikes.
• Ensure the application remains correct without cached data.

Conclusion

Caching improves scalability by serving repeated reads without using database capacity. It is most effective for frequently requested data with predictable freshness requirements.

A production cache must be treated as a disposable, potentially stale dependency. Correct keys, bounded TTLs, invalidation, negative caching, request coalescing, database protection, and observability are as important as raw cache speed.

Key Takeaway: Use caching to remove repeated work from databases, but keep authoritative decisions outside the cache and design every read path to survive misses, stale values, evictions, and complete cache failure.

More Articles to Read

• Cache in Software System Design — A Practical Guide
• Scalability for Dummies - Part 3: Cache
• Understanding Caching in Scalable Systems
• Cache-Aside vs Write-Through vs Write-Behind
• CDN vs Reverse Proxy vs Application Cache
• Designing Multi-Level Caching Architectures
• Cache Invalidation Strategies for Production Systems
• Preventing Cache Stampedes and Hot Keys
• Caching Best Practices for Distributed Applications