System Design Interview: How Would You Implement an API Rate Limiter in a Distributed Environment?

An API rate limiter sounds simple: count requests and reject clients that exceed a limit. The design becomes much harder when requests are handled by dozens or hundreds of application instances across multiple servers, availability zones, or regions.

A production rate limiter must make fast decisions under concurrency, enforce limits consistently enough for the business requirement, avoid becoming a bottleneck itself, tolerate partial failures, and distinguish between different users, API keys, endpoints, tenants, and traffic classes.

Table of Contents

• Define the Requirements
• Why Distributed Rate Limiting Is Hard
• Choose a Rate-Limiting Algorithm
  • Fixed Window Counter
  • Sliding Window Log
  • Sliding Window Counter
  • Token Bucket
• Where to Enforce the Limit
• Store Rate-Limit State in a Shared System
  • Why Local Memory Is Not Enough
  • Using Redis as the Counter Store
• Make Rate-Limit Updates Atomic
• Design the Rate-Limit Key
• Decide What Happens When the Limiter Fails
• Scale the Limiter Without Creating a Bottleneck
  • Partition Rate-Limit State
  • Use Local Budgets for Very High Traffic
• Multi-Region Rate Limiting
• Client Behavior and HTTP Responses
• What Does Not Work Well
• Production Design
• How to Answer This in a System Design Interview
• Conclusion
  • More Articles to Read

Define the Requirements

Before choosing Redis, Lua, token buckets, or any other implementation detail, define what the limiter actually needs to enforce.

A possible requirement might be:

Allow each API key to make at most 100 requests per second, permit short bursts up to 200 requests, and reject excess traffic before it reaches expensive backend services.

That immediately raises several design questions:

• Is the limit per user, tenant, API key, IP address, endpoint, or combination?
• Are short bursts allowed?
• Must the limit be globally exact, or is small temporary overage acceptable?
• Should different endpoints have different costs?
• What happens when the rate-limiter datastore is unavailable?
• Does the system operate in one region or many?

The answers determine whether a simple centralized counter is sufficient or whether the architecture needs partitioning, local budgets, and approximate coordination.

Why Distributed Rate Limiting Is Hard

Suppose an API runs on 20 application instances behind a load balancer and each instance independently allows 100 requests per second.

The real global limit is no longer 100 requests per second. A client that distributes traffic across instances could theoretically send close to 2,000 requests per second.

The instances therefore need some form of shared state or coordinated budget.

Concurrency creates another problem. If several servers read a counter value of 99 simultaneously, each may decide that the next request is allowed before any of them writes the new value.

The decision and state update must be atomic enough that concurrent requests cannot independently consume the same capacity.

Rate limiting is therefore a distributed coordination problem with a performance constraint: the coordination mechanism itself sits in the request path and must be extremely fast.

Choose a Rate-Limiting Algorithm

The algorithm controls how accurately traffic is limited and how bursts are handled. Different algorithms trade memory, precision, implementation complexity, and user experience.

Fixed Window Counter

A fixed window divides time into discrete intervals and maintains one counter for each interval.

For a limit of 100 requests per minute:

12:00:00–12:00:59 → max 100 requests

The implementation is cheap because only one counter is needed for each active key.

The main weakness is the window boundary. A client could send 100 requests at 12:00:59 and another 100 at 12:01:00, producing 200 requests in roughly one second while technically respecting both windows.

Fixed windows are useful when simplicity matters more than smooth traffic distribution.

Sliding Window Log

A sliding window log stores the timestamp of every accepted request within the active time range.

For each new request, timestamps older than the window are removed and the remaining entries are counted.

This provides precise enforcement but can consume significant memory for high-volume clients because every request requires a stored timestamp.

It also requires more work per request than a simple counter.

Sliding Window Counter

A sliding window counter approximates a moving window using counters from the current and previous windows.

Suppose a client made 80 requests in the previous minute and 20 requests in the current minute. If the current minute is 25% complete, the estimated count can weight only the relevant fraction of the previous window.

estimated_requests = (
    previous_window_count * previous_window_weight
    + current_window_count
)

This avoids storing every timestamp while reducing the boundary burst problem of a fixed window.

The result is approximate, but that is often acceptable for API protection.

Token Bucket

The token bucket is often the best interview choice when short traffic bursts should be allowed.

Each client owns a conceptual bucket containing tokens. Tokens are added at a fixed rate up to a maximum capacity. Every request consumes one or more tokens.

If enough tokens exist, the request proceeds. Otherwise it is rejected or delayed.

For example:

• bucket capacity: 200 tokens;
• refill rate: 100 tokens per second;
• normal request cost: 1 token.

A client that has been idle can temporarily burst up to 200 requests. Sustained traffic then converges toward 100 requests per second.

This behavior is usually more practical than enforcing an artificial hard boundary every second.

Where to Enforce the Limit

Rate limiting should normally happen as early as possible so rejected traffic does not consume expensive backend resources.

Possible enforcement points include:

• CDN or edge proxy;
• API gateway;
• reverse proxy;
• service mesh;
• application middleware;
• individual backend services.

An API gateway is often a good first layer because it can reject abusive traffic before requests reach application servers.

Application-level limiting may still be necessary when limits depend on business context unavailable at the gateway. For example, a reporting endpoint might cost 20 times more than a normal account lookup.

Large systems therefore frequently use multiple rate-limiting layers: coarse protection at the edge and fine-grained quotas inside the application.

Gateway placement and routing trade-offs are covered in more detail in API Gateway vs Backend-for-Frontend.

Store Rate-Limit State in a Shared System

Once requests can reach multiple application instances, the limiter needs a common view of consumed capacity unless the system deliberately uses approximate local budgets.

Why Local Memory Is Not Enough

A simple in-memory counter works only when all requests for a key are guaranteed to reach the same process.

That assumption usually breaks after horizontal scaling.

Sticky sessions can reduce the problem but do not solve it completely. Instances can restart, load balancing can change, clients can use multiple connections, and hot users can overload a single server.

Local memory is still useful as an optimization, but it should not be treated as the authoritative global counter when strict distributed limits are required.

Using Redis as the Counter Store

Redis is a common choice because rate-limit operations are small, frequent, latency-sensitive, and naturally map to counters, expiration, and atomic scripts.

A fixed-window key might look like:

rate_limit:api_key_481:/orders:20260905T1908

The value stores the number of requests observed in that window.

A simplified implementation can increment and expire the counter:

def allow_request(redis, key: str, limit: int, window_seconds: int) -> bool:
    count = redis.incr(key)

    if count == 1:
        redis.expire(key, window_seconds)

    return count <= limit

This illustrates the concept, but there is an important correctness issue: INCR and EXPIRE are separate commands. A failure between them can leave a counter without expiration.

The production implementation should combine the logic atomically.

Make Rate-Limit Updates Atomic

Rate-limiter state changes often involve several operations: reading capacity, refilling tokens, consuming tokens, and updating timestamps.

Executing these operations independently can introduce races under concurrent traffic.

A Redis Lua script can execute the complete decision atomically on the server.

A simplified token-bucket algorithm follows this logic:

1. Read the current token count and last refill timestamp.
2. Calculate how many tokens accumulated since the previous request.
3. Cap the result at the bucket capacity.
4. If at least one token exists, consume it and allow the request.
5. Otherwise reject the request.
6. Persist the new token count and timestamp.

from dataclasses import dataclass


@dataclass
class Bucket:
    tokens: float
    last_refill: float


def consume(
    bucket: Bucket,
    now: float,
    refill_rate: float,
    capacity: float,
    cost: float = 1.0,
) -> bool:
    elapsed = max(0.0, now - bucket.last_refill)

    bucket.tokens = min(
        capacity,
        bucket.tokens + elapsed * refill_rate,
    )
    bucket.last_refill = now

    if bucket.tokens < cost:
        return False

    bucket.tokens -= cost
    return True

In a distributed implementation, this computation should happen atomically near the authoritative state rather than separately in each application process.

Design the Rate-Limit Key

The algorithm can be correct while the product behavior is wrong if the rate-limit key is poorly designed.

A key should represent the entity whose usage is being constrained.

Common dimensions include:

• API key;
• authenticated user;
• tenant or organization;
• source IP;
• endpoint;
• HTTP method;
• subscription tier;
• region.

For example:

rate_limit:{tenant_id}:{api_key}:{route}

A tenant-level quota can prevent one customer from consuming disproportionate capacity even when that tenant owns thousands of API keys.

Endpoint-specific limits are useful because not all requests have the same cost. A cached profile lookup and a complex analytics export should not necessarily consume the same budget.

Some systems assign weighted costs:

• GET /users/{id} = 1 token;
• POST /search = 5 tokens;
• POST /reports/export = 25 tokens.

This turns the limiter into a rough form of resource admission control rather than only a request counter.

Decide What Happens When the Limiter Fails

A rate limiter is infrastructure in the critical request path. If its datastore becomes unavailable, every application request may be affected.

The design therefore needs an explicit choice between fail-open and fail-closed.

Fail-open means requests are allowed when the limiter cannot make a decision. Availability is preserved, but backend services temporarily lose protection.

Fail-closed means requests are rejected when the limiter is unavailable. Protection remains strict, but the limiter can become a single point of application unavailability.

A useful middle ground is a local fallback limit. If the shared limiter is unavailable, each application instance temporarily enforces a conservative local quota until connectivity recovers.

This sacrifices global precision but prevents both unlimited traffic and complete dependency on the shared store.

Scale the Limiter Without Creating a Bottleneck

A centralized limiter solves coordination but can itself become the hottest service in the architecture.

If an API handles one million requests per second and every request performs a synchronous Redis operation, the limiter must also handle roughly one million operations per second before application work even begins.

Partition Rate-Limit State

Rate-limit keys can be partitioned across multiple nodes using a hash of the logical key.

shard = hash(rate_limit_key) % number_of_shards

All requests for the same key must normally reach the same authoritative shard so the counter remains coherent.

High-cardinality workloads partition naturally because different users or API keys map to different nodes.

The harder case is a hot key. A single extremely active tenant may generate enough traffic to overload the shard responsible for its counter.

That case may require hierarchical limits, local token allocation, or deliberate approximation rather than forcing every request through one serialized counter.

Use Local Budgets for Very High Traffic

At very high scale, every request does not necessarily need a centralized coordination operation.

A central limiter can allocate small token budgets to application instances.

Suppose a tenant is allowed 100,000 requests per second. Instead of checking the global store 100,000 times, instances request token batches such as 500 tokens.

Each instance then consumes its local allocation without contacting the central store for every request.

Global quota → Allocate token batches → Local consumption

This dramatically reduces central traffic.

The trade-off is temporary over-allocation. If ten servers each hold unused tokens, the globally available capacity is not perfectly visible.

This is a common distributed-systems trade-off: stronger consistency requires more coordination, while lower coordination improves latency and throughput at the cost of precision.

Multi-Region Rate Limiting

Global rate limiting becomes much harder when the same client can send requests to multiple regions.

Assume a tenant has a global limit of 1,000 requests per second and traffic can reach both Virginia and Frankfurt.

If each region independently permits 1,000 requests per second, the effective global limit becomes 2,000.

Several strategies are possible.

• Global synchronous counter. Every region coordinates against one authoritative store. This offers stronger consistency but adds cross-region latency and creates a dependency on inter-region connectivity.
• Static regional quotas. Allocate 600 requests per second to one region and 400 to another. This is fast and simple but wastes unused capacity when traffic distribution changes.
• Dynamic regional budgets. A global control plane periodically reallocates quotas based on observed demand.
• Approximate independent limiting. Each region operates mostly independently and accepts small global overages.

The correct answer depends on the business requirement. For an ordinary public API, small temporary overage may be harmless. For expensive paid workloads, stricter global coordination may be worth the latency.

Multi-region architecture always introduces latency, consistency, and failure-recovery trade-offs. Related design considerations are covered in Multi-Region Architecture and Disaster Recovery.

Client Behavior and HTTP Responses

A rate limiter should give clients enough information to respond correctly.

When the limit is exceeded, the conventional HTTP response is:

HTTP/1.1 429 Too Many Requests

The response can include information such as:

• when the client may retry;
• the request quota;
• remaining capacity;
• when the quota resets.

Clients should not immediately retry a rejected request in a tight loop. Doing so creates unnecessary traffic precisely when the system is attempting to reduce load.

Retries should use delay and usually exponential backoff with jitter. The broader behavior is covered in Timeouts, Retries, and Exponential Backoff.

The API should also distinguish rate limiting from overload. A client exceeding its contractual quota is different from a service temporarily protecting itself because CPU, queue depth, or downstream capacity is saturated.

Rate limiting controls who is allowed to consume capacity. Load shedding controls whether the system currently has enough capacity to accept more work.

That distinction is explored further in Circuit Breaker vs Bulkhead vs Load Shedding.

What Does Not Work Well

Several implementations are reasonable at small scale but break once traffic becomes distributed.

Another mistake is pursuing perfect global accuracy without asking whether the requirement actually needs it. A few requests above a public API quota may be harmless, while synchronous global coordination could add measurable latency to every request.

Precision has a cost. The system should pay that cost only when exceeding the limit has meaningful financial, security, or reliability consequences.

Production Design

A practical distributed rate limiter can combine a token bucket with Redis-backed atomic state.

1. The request reaches an API gateway or application middleware.
2. The service authenticates enough of the request to identify the relevant rate-limit key.
3. The key combines dimensions such as tenant, API key, and endpoint class.
4. The limiter routes the key to the appropriate Redis shard.
5. An atomic operation refills and consumes tokens.
6. If capacity exists, the request continues.
7. If capacity is exhausted, the service returns 429 Too Many Requests.
8. When Redis is unavailable, a predefined fail-open, fail-closed, or local-fallback policy applies.
9. At extreme scale, instances consume small local token allocations to reduce shared-store operations.

Important metrics include:

• allowed requests per second by rate-limit policy;
• rejected requests per second and rejection percentage;
• rate-limiter p50, p95, and p99 latency;
• Redis command latency and error rate;
• hot-key frequency;
• shard CPU and memory utilization;
• fallback-mode activations;
• quota consumption by tenant;
• backend saturation after rate limiting.

A healthy limiter should consume only a small fraction of total request latency. If a 30-millisecond API call spends 15 milliseconds waiting for the rate limiter, the protection mechanism has become part of the performance problem.

The rate limiter should also be monitored together with downstream saturation. If rejection rates remain low while backend queue depth and latency continue increasing, the configured limits are not actually protecting the constrained resource.

How to Answer This in a System Design Interview

A strong interview answer can begin by clarifying that distributed instances cannot safely maintain independent counters.

Each application server sees only a fraction of total traffic, so a distributed rate limiter needs shared or coordinated state. The design should make the limit decision atomically while keeping that operation cheap enough to remain in the request path.

Then build the answer in layers.

1. Clarify the limit. Define whether the quota is per user, API key, tenant, endpoint, or region and whether bursts are allowed.
2. Select an algorithm. Use token bucket when sustained limits with controlled bursts are desirable.
3. Place enforcement early. Prefer the gateway or another layer before expensive backend work.
4. Use shared state. Store counters or token-bucket state in Redis or a similar low-latency distributed store.
5. Make the decision atomic. Update tokens and determine allow or reject within one atomic operation.
6. Partition by key. Distribute independent clients across shards to scale horizontally.
7. Plan failure behavior. Decide explicitly whether the application fails open, fails closed, or falls back to conservative local limits.
8. Reduce coordination at extreme scale. Allocate local token budgets when one central operation per request becomes too expensive.
9. Explain multi-region trade-offs. Strong global quotas require coordination; approximate regional budgets improve latency and availability.

The most important insight is that a distributed rate limiter is not primarily a counter problem; it is a coordination, consistency, latency, and failure-management problem.

Conclusion

A distributed API rate limiter needs more than an integer counter. Multiple application instances must coordinate consumption of a shared quota without turning the limiter into a serialization bottleneck.

Token buckets provide a practical balance between sustained limits and controlled bursts. Redis or another low-latency shared store can provide authoritative state, while atomic operations prevent races under concurrency. At higher scale, partitioning and local token allocation reduce central coordination.

The core design trade-off is between accuracy and coordination cost. Strong global enforcement requires more communication, while approximate local or regional limits improve latency, availability, and scalability. The right design enforces limits precisely enough to protect the constrained resource without making the limiter more expensive or fragile than the API it protects.

More Articles to Read

• Timeouts, Retries, and Exponential Backoff
• Circuit Breaker vs Bulkhead vs Load Shedding
• API Gateway vs Backend-for-Frontend
• Multi-Region Architecture and Disaster Recovery