API Rate Limiting and Throttling Explained

API rate limiting controls how many requests a client may send during a defined period. Throttling controls how the system responds when traffic exceeds its allowed rate or the infrastructure approaches capacity.

Production rate limiting is not only an abuse-prevention feature. It protects downstream services, creates fair resource allocation, enforces commercial quotas, controls infrastructure cost, and prevents one noisy client from degrading the API for everyone else.

Table of Contents

• Rate Limiting vs Throttling
• Why Rate Limits Exist
  • Protecting Capacity
  • Fairness and Quotas
  • Containing Failures
• Rate-Limit Dimensions
  • Identity Dimension
  • Resource Dimension
  • Time and Cost Dimension
• Fixed-Window Counter
  • Advantages
  • Disadvantages
  • When to Use / Real-World Use Cases
  • Example
• Sliding-Window Log
  • Advantages
  • Disadvantages
  • When to Use / Real-World Use Cases
  • Example
• Sliding-Window Counter
  • Advantages
  • Disadvantages
  • When to Use / Real-World Use Cases
  • Example
• Token Bucket
  • Advantages
  • Disadvantages
  • When to Use / Real-World Use Cases
  • Example
• Leaky Bucket
  • Advantages
  • Disadvantages
  • When to Use / Real-World Use Cases
  • Example
• Rate-Limiting Algorithm Comparison
• Distributed Rate Limiting
  • Centralized Counter Store
  • Atomic Updates
  • Regional and Global Limits
• Designing Rate-Limit Policies
  • Per-User and Per-Tenant Limits
  • Endpoint-Specific Limits
  • Weighted Request Cost
  • Burst and Sustained Limits
• Client Response Design
  • HTTP 429
  • Rate-Limit Headers
  • Client Retry Behavior
• Where to Enforce Rate Limits
  • Edge and API Gateway
  • Application Layer
  • Downstream Protection
• Operational Considerations
  • Fail Open vs Fail Closed
  • Hot Keys
  • Observability
• Ready-to-Use Example
  • Redis and Lua Example
  • FastAPI Integration
  • Python Client Example
  • API Gateway CloudFormation Example
• Common Mistakes
• Production Checklist
• Conclusion
• More Articles to Read

Rate Limiting vs Throttling

Rate limiting and throttling are closely related, but they are not exactly the same.

A rate limiter decides whether a request fits within a policy. A throttling mechanism determines what happens after capacity or policy is exceeded.

Incoming request
  -> Resolve client identity
  -> Load applicable limit
  -> Calculate current usage
  -> Allow request
     or
  -> Reject / delay / queue request

Key Point: rate limiting defines the boundary; throttling enforces behavior around that boundary.

Why Rate Limits Exist

Rate limiting protects more than the API process itself. A single request may consume database connections, cache capacity, third-party API quota, CPU time, memory, queue space, or expensive compute.

A strong policy considers the entire dependency chain rather than only web-server throughput.

Protecting Capacity

Every API has finite capacity. Uncontrolled request growth can exhaust worker pools, database connections, memory, or downstream quotas.

Client traffic
  -> API workers
  -> Database connection pool
  -> Cache
  -> Third-party payment API

Rate limit must protect the narrowest bottleneck.

Production Note: allowing more API requests than the database can sustain only moves the failure deeper into the system.

Fairness and Quotas

Multi-tenant APIs should prevent one client from consuming all shared capacity. Limits create predictable allocation between customers, users, applications, and pricing plans.

Containing Failures

Rate limits reduce the blast radius of client bugs, retry storms, credential leaks, and abusive automation.

They also protect internal services from cascading failures when callers retry faster than a dependency can recover.

Dependency slows down
  -> Requests time out
  -> Clients retry immediately
  -> Traffic multiplies
  -> Dependency receives even more load

Rate limiting + backoff
  -> reduces amplification

Rate-Limit Dimensions

A useful rate-limit key usually combines identity, resource, operation, and time. Limiting only by IP address is rarely enough for authenticated production APIs.

Identity Dimension

Limits can apply to different identities depending on the API.

• IP address for unauthenticated endpoints or basic abuse control.
• User ID for interactive applications.
• API key for third-party integrations.
• Tenant ID for shared SaaS capacity.
• Service identity for internal APIs.

Important: IP-based limits can punish many users behind the same NAT gateway and can be bypassed by distributed clients.

Resource Dimension

Different operations have different cost and abuse risk. Reading a cached profile is not equivalent to generating a report or starting a machine-learning job.

GET /v1/profile
  cost: 1 unit

POST /v1/reports
  cost: 20 units

POST /v1/video-transcodes
  cost: 100 units

Time and Cost Dimension

Production APIs often combine several limits.

Fixed-Window Counter

A fixed-window limiter divides time into discrete intervals and counts requests inside each interval.

Advantages

• Simple: one counter per identity and time window.
• Efficient: low storage and computational overhead.
• Easy expiration: counters can expire after the window.
• Easy reporting: usage maps directly to minute, hour, or day buckets.

Disadvantages

• Boundary bursts: clients can send one full limit before and another immediately after reset.
• Uneven traffic: a nominal per-minute limit can allow a much larger short spike.
• Clock coordination: distributed nodes must agree on window boundaries.
• Coarse control: it does not model smooth traffic well.

When to Use / Real-World Use Cases

• Daily or monthly product quotas.
• Simple internal APIs where boundary bursts are acceptable.
• Low-cost usage accounting.
• Coarse abuse limits layered with another burst limiter.

Example

Limit: 100 requests per minute

Counter key:
  rate:user_123:2026-07-12T18:42

Request:
  increment counter

If counter > 100:
  reject with HTTP 429

Sliding-Window Log

A sliding-window log stores the timestamp of each accepted request and counts timestamps that fall inside the active window.

Advantages

• Precise: enforces the exact number of requests in any rolling interval.
• No boundary burst: limits do not reset abruptly.
• Easy reasoning: behavior matches the policy directly.
• Useful audit data: individual request timestamps are available.

Disadvantages

• High storage cost: every request adds a timestamp.
• Cleanup overhead: old timestamps must be removed.
• Hot data structures: high-volume clients create large sorted sets.
• Higher latency: multiple operations may be required per request.

When to Use / Real-World Use Cases

• Security-sensitive endpoints such as password reset or login attempts.
• Low-volume APIs requiring precise rolling limits.
• Fraud controls where every event timestamp matters.
• Administrative actions with strict limits.

Example

Window: last 60 seconds
Limit: 10 requests

Sorted set:
  18:42:04
  18:42:09
  18:42:15
  ...

On request:
  remove entries older than now - 60 seconds
  count remaining entries
  allow if count < 10
  add current timestamp

Sliding-Window Counter

A sliding-window counter estimates rolling usage by combining the previous fixed window with the current window.

Advantages

• Lower storage: only a small number of counters are needed.
• Smoother behavior: reduces fixed-window boundary bursts.
• Good accuracy: close enough for most API limits.
• Efficient: suitable for high request volume.

Disadvantages

• Approximate: request distribution inside the previous window is estimated.
• More complex than fixed windows: weighted calculations are required.
• Clock dependence: nodes still need consistent time.
• Less intuitive debugging: usage is calculated rather than directly counted.

When to Use / Real-World Use Cases

• Public APIs needing smoother per-minute limits.
• High-throughput SaaS platforms.
• Tenant-level fairness controls.
• Systems where exact per-request logs are unnecessary.

Example

Previous minute count: 80
Current minute count: 20
Current minute elapsed: 25%

Estimated rolling usage:
  previous contribution = 80 * 75%
  current contribution = 20

Estimated total = 80

Token Bucket

A token bucket stores tokens that refill at a fixed rate. Each request consumes one or more tokens. Requests are allowed while enough tokens remain.

Advantages

• Supports bursts: unused capacity accumulates up to bucket size.
• Controls sustained rate: refill rate defines long-term throughput.
• Supports weighted requests: expensive operations can consume more tokens.
• Efficient: only token balance and refill timestamp are required.

Disadvantages

• More parameters: refill rate and bucket size must both be tuned.
• Atomicity required: concurrent requests must not overspend tokens.
• Burst load still reaches dependencies: bucket size must respect backend capacity.
• Time calculation complexity: distributed implementations need consistent refill behavior.

When to Use / Real-World Use Cases

• Public APIs allowing reasonable traffic bursts.
• API gateways with burst and sustained limits.
• Weighted API operations.
• Tenant-level traffic shaping.

Example

Bucket capacity: 100 tokens
Refill rate: 10 tokens per second
Request cost: 1 token

Idle client:
  bucket fills to 100

Burst:
  client may send 100 requests immediately

Sustained traffic:
  client receives about 10 new request tokens per second

Leaky Bucket

A leaky bucket places requests into a queue and processes them at a fixed rate. Excess traffic either waits or is rejected when the queue is full.

Advantages

• Smooth output: downstream systems receive traffic at a predictable rate.
• Strong traffic shaping: bursts are converted into steady flow.
• Protects fragile dependencies: concurrency and throughput remain controlled.
• Useful backpressure: queue capacity provides an explicit boundary.

Disadvantages

• Added latency: accepted requests may wait in a queue.
• Queue management: timeouts, cancellation, and overflow need handling.
• Poor fit for interactive APIs: users may prefer immediate rejection to hidden delay.
• Capacity planning required: a large queue can hide overload and create stale work.

When to Use / Real-World Use Cases

• Background job dispatch.
• Third-party APIs with strict requests-per-second limits.
• Email or notification delivery.
• Batch processing pipelines.

Example

Incoming requests
  -> bounded queue
  -> worker releases 20 requests per second
  -> downstream provider

If queue is full:
  reject or defer new requests

Rate-Limiting Algorithm Comparison

Algorithm choice depends on required precision, burst behavior, storage cost, and implementation complexity.

Distributed Rate Limiting

A rate limiter running inside one application instance sees only that instance’s requests. Production APIs usually run across multiple processes, containers, servers, or regions.

Distributed enforcement requires shared state or carefully partitioned limits.

Centralized Counter Store

Redis is a common rate-limit store because it provides fast counters, expiration, sorted sets, and atomic scripts.

API instance A
API instance B
API instance C
       │
       ▼
Shared Redis rate-limit state

The trade-off is dependency risk. If Redis becomes unavailable, the API needs an explicit failure strategy.

Atomic Updates

Rate-limit checks must be atomic. Separate read and write operations can allow concurrent requests to exceed the limit.

Unsafe:
  read token balance
  check balance
  subtract token

Two requests may read the same balance.

Safe:
  calculate refill
  check balance
  subtract token
  store state

All inside one atomic operation.

Redis Lua scripts or transactions are commonly used to keep the decision atomic.

Regional and Global Limits

Global limits are difficult because every region needs a consistent view of usage. Cross-region coordination adds latency and failure modes.

Rule of Thumb: enforce short-term traffic limits regionally and reconcile long-term commercial quotas globally.

Designing Rate-Limit Policies

A rate limiter is only as good as its policy. One global number rarely fits all clients and operations.

Per-User and Per-Tenant Limits

Per-user limits prevent individual abuse. Per-tenant limits protect shared capacity from large customer accounts.

Request must satisfy:

user limit
AND
tenant limit
AND
endpoint limit
AND
global service limit

Layered limits protect both fairness and infrastructure.

Endpoint-Specific Limits

Expensive endpoints should have lower limits than cached reads.

Weighted Request Cost

Weighted limits assign different token costs to different operations.

GET /orders/{id}
  cost: 1 token

GET /orders?include=items,payments,shipments
  cost: 5 tokens

POST /reports
  cost: 25 tokens

This model better represents backend work than simple request counts.

Burst and Sustained Limits

Clients often need short bursts without being allowed unlimited sustained traffic.

Client Response Design

Rate-limit responses should help well-behaved clients recover. A generic error without reset information encourages blind retries.

HTTP 429

APIs should return 429 Too Many Requests when a client exceeds a policy.

HTTP/1.1 429 Too Many Requests
Content-Type: application/json
Retry-After: 12

{
  "error": {
    "code": "rate_limit_exceeded",
    "message": "Request limit exceeded.",
    "request_id": "req_8d19a2",
    "retry_after_seconds": 12
  }
}

Rate-Limit Headers

Response headers can expose the active policy and current remaining capacity.

RateLimit-Limit: 100
RateLimit-Remaining: 37
RateLimit-Reset: 12
Retry-After: 12

Header semantics should remain consistent across endpoints and API versions.

Client Retry Behavior

Clients should respect Retry-After and use exponential backoff with jitter when retrying temporary failures.

Retry delay:
  server Retry-After value
  or
  exponential backoff + random jitter

Never:
  retry immediately in a tight loop

Important: automatic retries should be limited and safe for the operation’s idempotency behavior.

Where to Enforce Rate Limits

Rate limits can be enforced at the edge, gateway, application, or downstream dependency. Production systems often use multiple layers.

Edge and API Gateway

Edge and gateway limits reject excess traffic before it consumes application capacity.

• Good for IP and API-key limits.
• Good for broad per-route throttling.
• Reduces load on application services.
• May lack detailed business context.

Application Layer

Application-layer enforcement has access to user, tenant, subscription, resource, and request-cost information.

Gateway:
  basic IP and route protection

Application:
  user plan
  tenant quota
  endpoint cost
  resource-specific policy

Downstream Protection

Internal calls may need separate concurrency or throughput limits to protect databases and third-party providers.

A public request limit does not guarantee safe downstream behavior if each request fans out into many operations.

Operational Considerations

Rate limiting becomes a production dependency. Its availability, latency, key distribution, and metrics need deliberate design.

Fail Open vs Fail Closed

If the rate-limit store is unavailable, the API must decide whether to allow or reject requests.

Login, payment, and security endpoints may justify fail-closed behavior. Low-risk reads may fail open with conservative local limits.

Hot Keys

Large tenants or shared global limits can create hot Redis keys. Every request may contend on the same counter.

• Partition limits by tenant, region, or endpoint.
• Avoid unnecessary global counters.
• Use local admission control before shared enforcement.
• Monitor command latency and key concentration.

Observability

Rate limiting should expose metrics that distinguish client misuse from insufficient service capacity.

Ready-to-Use Example

The following implementation uses a Redis-backed token bucket. A Lua script performs refill, validation, token deduction, and state update atomically.

The FastAPI middleware applies the limiter per authenticated client and returns useful rate-limit headers.

Redis and Lua Example

-- KEYS[1]: Redis key for one client and policy.
-- ARGV[1]: bucket capacity.
-- ARGV[2]: refill rate in tokens per second.
-- ARGV[3]: current Unix timestamp in milliseconds.
-- ARGV[4]: token cost of this request.
-- ARGV[5]: key expiration in seconds.

local key = KEYS[1]
local capacity = tonumber(ARGV[1])
local refill_rate = tonumber(ARGV[2])
local now_ms = tonumber(ARGV[3])
local request_cost = tonumber(ARGV[4])
local ttl_seconds = tonumber(ARGV[5])

-- Store token balance and last refill time in one Redis hash.
local state = redis.call(
  "HMGET",
  key,
  "tokens",
  "last_refill_ms"
)

local tokens = tonumber(state[1])
local last_refill_ms = tonumber(state[2])

-- New clients start with a full bucket so a reasonable initial burst is allowed.
if tokens == nil then
  tokens = capacity
  last_refill_ms = now_ms
end

-- Refill based on elapsed time rather than running a background refill process.
local elapsed_seconds = math.max(0, now_ms - last_refill_ms) / 1000
local refilled_tokens = elapsed_seconds * refill_rate
tokens = math.min(capacity, tokens + refilled_tokens)

local allowed = 0
local retry_after_ms = 0

if tokens >= request_cost then
  -- Deduct only when enough capacity exists.
  tokens = tokens - request_cost
  allowed = 1
else
  -- Calculate how long until enough tokens are available.
  local missing_tokens = request_cost - tokens
  retry_after_ms = math.ceil((missing_tokens / refill_rate) * 1000)
end

-- Persist the recalculated state atomically with the decision.
redis.call(
  "HSET",
  key,
  "tokens",
  tokens,
  "last_refill_ms",
  now_ms
)

-- Expire inactive buckets so old clients do not consume memory forever.
redis.call("EXPIRE", key, ttl_seconds)

return {
  allowed,
  tostring(tokens),
  retry_after_ms
}

FastAPI Integration

from __future__ import annotations

import math
import time
from dataclasses import dataclass
from pathlib import Path

from fastapi import FastAPI, Request
from fastapi.responses import JSONResponse
from redis.asyncio import Redis

app = FastAPI(title="Rate-Limited API")

# Use one shared Redis connection pool for the application process.
redis = Redis.from_url(
    "redis://localhost:6379/0",
    encoding="utf-8",
    decode_responses=True,
)

# Load the script once from a version-controlled file in a real project.
LUA_SCRIPT = Path("token_bucket.lua").read_text(encoding="utf-8")


@dataclass(frozen=True)
class RateLimitPolicy:
    capacity: int
    refill_rate: float
    request_cost: int = 1

    @property
    def ttl_seconds(self) -> int:
        # Keep inactive bucket state for at least two full refill periods.
        refill_period = self.capacity / self.refill_rate
        return max(60, math.ceil(refill_period * 2))


DEFAULT_POLICY = RateLimitPolicy(
    # Allow a short burst of up to 100 requests.
    capacity=100,

    # Refill at a sustained rate of 10 requests per second.
    refill_rate=10.0,
)


def get_client_identity(request: Request) -> str:
    # Prefer an authenticated API key, user ID, or tenant ID in production.
    # IP addresses are only a fallback because many users may share one address.
    api_key = request.headers.get("x-api-key")

    if api_key:
        # Hash or map the key before using it in logs and Redis keys.
        return f"api-key:{api_key[:8]}"

    client_host = request.client.host if request.client else "unknown"
    return f"ip:{client_host}"


def get_request_cost(request: Request) -> int:
    # Expensive endpoints should consume more capacity than lightweight reads.
    if request.method == "POST" and request.url.path == "/v1/reports":
        return 20

    return 1


@app.middleware("http")
async def enforce_rate_limit(request: Request, call_next):
    identity = get_client_identity(request)
    request_cost = get_request_cost(request)

    # Include the route family in the key when endpoints need independent limits.
    key = f"rate-limit:{identity}:default"

    now_ms = int(time.time() * 1000)

    try:
        result = await redis.eval(
            LUA_SCRIPT,
            1,
            key,
            DEFAULT_POLICY.capacity,
            DEFAULT_POLICY.refill_rate,
            now_ms,
            request_cost,
            DEFAULT_POLICY.ttl_seconds,
        )
    except Exception:
        # This example fails open for general API traffic.
        # Security-sensitive endpoints may need fail-closed behavior instead.
        return await call_next(request)

    allowed = bool(int(result[0]))
    remaining_tokens = max(0, math.floor(float(result[1])))
    retry_after_ms = int(result[2])

    if not allowed:
        retry_after_seconds = max(1, math.ceil(retry_after_ms / 1000))

        return JSONResponse(
            status_code=429,
            headers={
                "Retry-After": str(retry_after_seconds),
                "RateLimit-Limit": str(DEFAULT_POLICY.capacity),
                "RateLimit-Remaining": str(remaining_tokens),
                "RateLimit-Reset": str(retry_after_seconds),
            },
            content={
                "error": {
                    "code": "rate_limit_exceeded",
                    "message": "Request limit exceeded.",
                    "retry_after_seconds": retry_after_seconds,
                }
            },
        )

    response = await call_next(request)

    # Return current policy state so clients can avoid unnecessary rejections.
    response.headers["RateLimit-Limit"] = str(DEFAULT_POLICY.capacity)
    response.headers["RateLimit-Remaining"] = str(remaining_tokens)

    return response


@app.get("/v1/orders")
async def list_orders() -> dict:
    return {"items": []}


@app.post("/v1/reports")
async def create_report() -> dict:
    # This route consumes 20 tokens because report generation is expensive.
    return {"status": "accepted"}

Production Note: do not place raw API keys or access tokens in Redis keys. Resolve credentials to stable internal identifiers or hash sensitive values first.

Python Client Example

import random
import time

import requests

API_URL = "https://api.example.com/v1/orders"
MAX_ATTEMPTS = 4

for attempt in range(1, MAX_ATTEMPTS + 1):
    response = requests.get(
        API_URL,
        headers={
            "Authorization": "Bearer ACCESS_TOKEN",
            "X-Request-ID": "req_client_example",
        },

        # Every production HTTP request needs an explicit timeout.
        timeout=3,
    )

    if response.status_code != 429:
        response.raise_for_status()
        print(response.json())
        break

    if attempt == MAX_ATTEMPTS:
        raise RuntimeError("API rate limit remained exceeded.")

    # Prefer explicit server guidance when Retry-After is available.
    retry_after = response.headers.get("Retry-After")

    if retry_after is not None:
        delay = float(retry_after)
    else:
        # Exponential backoff with jitter prevents synchronized retry storms.
        delay = min(30.0, (2 ** (attempt - 1)) + random.random())

    time.sleep(delay)

API Gateway CloudFormation Example

AWSTemplateFormatVersion: "2010-09-09"
Description: API Gateway stage-level throttling example

Parameters:
  ApiId:
    Type: String
    Description: Existing API Gateway REST API identifier.

Resources:
  ProductionDeployment:
    Type: AWS::ApiGateway::Deployment
    Properties:
      RestApiId: !Ref ApiId
      Description: Production deployment with stage throttling.

  ProductionStage:
    Type: AWS::ApiGateway::Stage
    Properties:
      RestApiId: !Ref ApiId
      DeploymentId: !Ref ProductionDeployment
      StageName: prod

      MethodSettings:
        - ResourcePath: "/*"
          HttpMethod: "*"

          # Allow a sustained average of 100 requests per second.
          ThrottlingRateLimit: 100

          # Allow short bursts without permitting unlimited sustained traffic.
          ThrottlingBurstLimit: 200

          # Enable metrics so throttled requests and latency are observable.
          MetricsEnabled: true

          # Avoid logging complete sensitive request and response payloads.
          DataTraceEnabled: false

          LoggingLevel: ERROR

Important: gateway throttling protects broad infrastructure capacity. Per-user, per-tenant, subscription-plan, and weighted-cost limits usually still require application-level enforcement.

Common Mistakes

Most rate-limiting mistakes come from choosing a limit without understanding client identity, request cost, distributed state, or downstream capacity.

• Limiting only by IP address for authenticated APIs.
• Using one policy for every endpoint regardless of operation cost.
• Applying only per-user limits and forgetting tenant or global capacity.
• Using non-atomic read-and-write operations in distributed limiters.
• Returning 500 instead of 429 when traffic exceeds policy.
• Omitting Retry-After information and encouraging blind retries.
• Allowing large bursts that exceed database or downstream capacity.
• Failing closed for every endpoint and turning Redis failure into a complete API outage.
• Failing open on security-sensitive operations such as login attempts.
• Counting rejected retries against the wrong quota without a deliberate policy.
• Using local memory only across multiple API instances.
• Not monitoring rejection rates by endpoint, tenant, and plan.

Production Checklist

A production rate limiter should protect capacity, enforce fairness, remain observable, and give clients enough information to recover correctly.

• Identify clients using trusted identities such as user, tenant, API key, or service ID.
• Define limits around real bottlenecks, not arbitrary request numbers.
• Use endpoint-specific or weighted costs for expensive operations.
• Combine burst and sustained limits where clients need temporary spikes.
• Use atomic updates for distributed counters and token balances.
• Choose regional or global enforcement intentionally.
• Return HTTP 429 for policy violations.
• Return Retry-After and rate-limit metadata.
• Use exponential backoff with jitter in clients.
• Choose fail-open or fail-closed behavior per endpoint risk.
• Protect login, password reset, and payment endpoints separately.
• Monitor allowed, rejected, and near-limit traffic.
• Monitor limiter latency and dependency errors.
• Avoid storing raw credentials in limiter keys.
• Load-test rate limits with realistic concurrency and burst patterns.
• Document limits for API consumers.

Conclusion

API rate limiting protects production systems from overload, abuse, retry storms, unfair resource consumption, and uncontrolled cost. Throttling defines how excess traffic is rejected, delayed, or smoothed after a limit is reached.

Fixed windows are simple, sliding windows provide smoother control, token buckets balance bursts with sustained throughput, and leaky buckets shape traffic into a predictable downstream rate. The correct choice depends on precision, burst behavior, scale, and dependency capacity.

Key Takeaway: rate limits should model real system capacity and client identity. Enforce them atomically, return actionable 429 responses, and combine gateway protection with application-level business policies.

More Articles to Read

• REST API Design for Production Systems
• API Authentication and Authorization Explained
• API Pagination, Filtering, and Sorting Explained
• API Versioning and Backward Compatibility
• API Performance and Reliability Best Practices