Scaling Stateless Applications

Horizontal scaling becomes significantly easier when application instances are interchangeable. A request can reach any healthy instance, failed instances can disappear without losing business state, and new capacity can join the fleet without synchronizing local files or sessions.

This is the core value of stateless application architecture. Stateless does not mean the application has no state. Production systems still depend on databases, caches, queues, object storage, and authentication state. The important distinction is that durable or shared state does not belong to an individual application instance.

Once compute becomes replaceable, load balancing, autoscaling, rolling deployments, multi-zone availability, and failure recovery become much simpler. The remaining challenge is ensuring that scaling the application tier does not merely move the bottleneck into databases, caches, queues, or external services.

Table of Contents

• What Stateless Really Means
• Externalizing Application State
  • Sessions and Authentication
  • Files and Generated Data
• Horizontal Scaling and Load Balancing
  • Autoscaling Signals
  • Safe Scale-In
• Protecting Downstream Dependencies
• Production Design Example
  • Autoscaling Configuration
  • Failure and Deployment Flow
• Common Mistakes
• Production Checklist
• Conclusion
  • More Articles to Read

What Stateless Really Means

A stateless application instance does not depend on local information from previous requests in order to process the next request. Any healthy replica can handle traffic because required shared state is available outside the compute instance.

Consider an application that stores authenticated sessions and uploaded files on its local filesystem:

                    Load Balancer
                    /           \
                   /             \
                  v               v
             Instance A       Instance B
             session #123     session #456
             file-a.pdf       file-b.pdf

If the first request creates session #123 on Instance A and the next request reaches Instance B, Instance B cannot find the session. Sticky sessions can route the client back to Instance A, but the architecture remains coupled to that machine.

If Instance A fails, its sessions and locally stored files disappear with it.

A stateless architecture moves that state into shared systems:

                      Load Balancer
                     /           \
                    v             v
               Instance A     Instance B
                    \             /
                     +-----+-----+
                           |
             +-------------+-------------+
             |             |             |
             v             v             v
          Database        Cache      Object Storage
             |
             v
            Queue

Instances now contain application code and temporary execution state, but durable state survives independently. Either instance can process the next request.

This property enables several important production capabilities:

• Horizontal scaling: replicas can be added without migrating local state.
• Automatic recovery: failed instances can simply be replaced.
• Rolling deployments: old and new replicas can coexist during releases.
• Multi-zone operation: requests can move between failure domains.
• Load distribution: traffic does not need permanent affinity to individual machines.

Stateless architecture does not remove state management. It moves state to systems designed to manage it. This can increase dependency traffic and make database, cache, and storage capacity more important.

Externalizing Application State

Application state should be classified by its durability and access requirements rather than moved indiscriminately into one shared database.

The distinction between request-local and application-instance-local state matters. Keeping a parsed request, database result, or temporary calculation in memory while processing one request is completely compatible with stateless architecture. Problems begin when future requests depend on that memory remaining on the same instance.

Sessions and Authentication

Authentication commonly introduces instance affinity accidentally. A server creates a session, stores it in memory, and returns a session identifier to the client. Horizontal scaling then requires every future request to find the instance that owns that identifier.

One solution is storing sessions in a shared low-latency store:

from dataclasses import dataclass
from datetime import timedelta
from uuid import UUID


@dataclass(frozen=True)
class Session:
    user_id: UUID
    account_id: UUID


class SessionRepository:
    def __init__(self, cache: "Cache") -> None:
        self.cache = cache

    async def create(
        self,
        session_id: str,
        session: Session,
    ) -> None:
        await self.cache.set_json(
            key=f"session:{session_id}",
            value={
                "user_id": str(session.user_id),
                "account_id": str(session.account_id),
            },
            ttl=timedelta(hours=12),
        )

    async def get(self, session_id: str) -> Session | None:
        value = await self.cache.get_json(
            key=f"session:{session_id}"
        )

        if value is None:
            return None

        return Session(
            user_id=UUID(value["user_id"]),
            account_id=UUID(value["account_id"]),
        )

Any API replica can now resolve the same session. The trade-off is that authentication requests depend on the shared session store, so its latency and availability become part of the request path.

Another approach uses cryptographically signed tokens containing sufficient authentication information for local validation. This removes a session-store lookup from many requests, but revocation, token size, expiration, permission changes, and key rotation require careful design.

Neither model is inherently stateless at the system level. The goal is that individual compute replicas do not own irreplaceable session state.

Files and Generated Data

Local files create similar coupling. Suppose an API generates an export and writes it to /tmp/export.csv. A later download request may reach another replica where the file does not exist.

Durable files should normally be stored outside application compute:

from uuid import UUID


async def complete_export(
    export_id: UUID,
    content: bytes,
) -> None:
    object_key = f"exports/{export_id}.csv"

    await object_storage.put(
        key=object_key,
        content=content,
        content_type="text/csv",
    )

    await export_repository.mark_ready(
        export_id=export_id,
        object_key=object_key,
    )

Temporary local disk remains useful for intermediate processing as long as the application can tolerate losing it when the instance disappears.

More about choosing storage models can be found here: Cloud Storage Patterns and Trade-Offs.

Horizontal Scaling and Load Balancing

Once instances are interchangeable, a load balancer can distribute requests across a changing fleet without tracking application state ownership.

                         Clients
                            |
                            v
                     Load Balancer
                 /       |        \
                /        |         \
               v         v          v
            API 1      API 2      API 3
              |          |          |
              +----------+----------+
                         |
                 Shared Dependencies

Horizontal scaling adds replicas instead of making one server continuously larger. This provides both capacity and failure isolation, but it does not produce unlimited scalability.

If one instance can safely process 300 requests per second, ten instances do not necessarily provide exactly 3,000 requests per second. Shared dependencies can saturate first.

A useful capacity model is:

application_capacity =
    healthy_instances
    x safe_requests_per_instance

effective_system_capacity =
    min(
        application_capacity,
        database_capacity,
        cache_capacity,
        downstream_capacity,
        network_capacity
    )

Scaling therefore requires monitoring saturation across the complete request path rather than only the application fleet.

Autoscaling Signals

CPU utilization is a common autoscaling metric, but it is not universally appropriate. A service spending most of its time waiting on databases or external APIs can saturate connection pools and request workers while CPU remains relatively low.

For HTTP services, request rate and concurrency can often provide better workload signals than CPU alone. For asynchronous workers, queue depth combined with message age and processing duration provides a clearer picture of backlog pressure.

Scaling must account for startup delay. If a new replica requires two minutes to download an image, initialize dependencies, and pass readiness checks, scaling only after saturation begins can produce several minutes of poor latency.

Minimum capacity provides another tool. Keeping several ready replicas costs more but provides immediate headroom for traffic bursts and instance failures.

Safe Scale-In

Removing capacity is more dangerous than adding it because active requests may still be executing on the instance selected for termination.

A safe scale-in sequence looks like this:

1. Mark the instance as unavailable for new traffic.
2. Remove it from load-balancer routing.
3. Allow in-flight requests to complete within a bounded drain period.
4. Stop accepting new asynchronous jobs.
5. Return unfinished retryable jobs to their queue if necessary.
6. Terminate the instance.

Applications should respond correctly to termination signals rather than disappearing immediately.

import asyncio
import signal

from fastapi import FastAPI

app = FastAPI()
shutdown_requested = asyncio.Event()


def request_shutdown() -> None:
    shutdown_requested.set()


@app.on_event("startup")
async def configure_shutdown() -> None:
    loop = asyncio.get_running_loop()

    for sig in (signal.SIGTERM, signal.SIGINT):
        loop.add_signal_handler(sig, request_shutdown)


@app.get("/health/ready")
async def readiness() -> tuple[dict[str, str], int]:
    if shutdown_requested.is_set():
        # Stop receiving new traffic while existing
        # requests are allowed to drain.
        return {"status": "draining"}, 503

    return {"status": "ready"}, 200

Production frameworks and orchestrators provide different lifecycle mechanisms, but the principle remains the same: traffic should stop before the process disappears.

Protecting Downstream Dependencies

Stateless compute can scale much faster than many stateful dependencies. This makes uncontrolled autoscaling a potential reliability problem.

Consider an API where every replica maintains a database connection pool of 30 connections:

5 replicas   x 30 =   150 possible connections
20 replicas  x 30 =   600 possible connections
100 replicas x 30 = 3,000 possible connections

Database maximum = 800

At 100 replicas, application autoscaling can exhaust the database even if each individual instance appears healthy.

The same problem affects external APIs, caches, queues, and other bounded dependencies.

                     Autoscaler
                         |
                         v
                +----------------+
                | 100 API replicas|
                +--------+-------+
                         |
                 thousands of calls
                         |
                         v
                +----------------+
                | Shared Database|
                | or External API|
                +----------------+
                         |
                         X
                      overload

Several controls help protect downstream capacity:

• Connection pooling: reuse connections while keeping pool size bounded.
• Connection proxies: multiplex many application clients over fewer backend connections where appropriate.
• Concurrency limits: restrict expensive operations per replica or across the workload.
• Rate limiting: keep request rates within dependency quotas.
• Queues: buffer work when immediate downstream execution is unnecessary.
• Load shedding: reject lower-priority requests when resources are saturated.
• Backpressure: slow producers when consumers cannot keep up.

Autoscaling limits should therefore be derived partly from downstream capacity.

database_connections_available = 600
connections_per_replica = 20

database_safe_replica_limit =
    600 / 20
    = 30 replicas

This does not mean the fleet must always stop at 30 replicas. A connection proxy, smaller pools, database scaling, caching, or architecture changes may increase the safe limit. The important point is that maximum replicas should not be selected independently from dependency limits.

Failure handling is equally important. Slow dependencies cause requests to remain active longer, which increases concurrency and can trigger additional scale-out. Without timeouts and bounded retries, autoscaling can amplify a dependency outage. More about protecting services from these failure patterns can be found here: Circuit Breaker vs Bulkhead vs Load Shedding.

Production Design Example

Consider a cloud-hosted shipment platform receiving customer API traffic while generating labels, importing bulk shipments, and synchronizing tracking information with external carriers.

The synchronous API and asynchronous workloads scale independently:

                           Clients
                              |
                              v
                       Load Balancer
                       /     |     \
                      v      v      v
                    API    API    API
                      \      |      /
                       +-----+-----+
                             |
                +------------+------------+
                |            |            |
                v            v            v
             Database      Cache      Object Storage
                |
                v
              Queue
            /   |   \
           v    v    v
        Worker Worker Worker
                |
                v
          Carrier APIs

API replicas are stateless. Authentication state is either validated from signed credentials or retrieved from shared storage. Uploaded shipping documents live in object storage. Business entities live in the database.

Label generation and carrier synchronization run through queues. This prevents slow carrier APIs from occupying synchronous application capacity and allows worker fleets to scale separately from customer traffic.

API scaling uses request concurrency and CPU. Worker scaling uses queue backlog and oldest-message age. The maximum size of both fleets is constrained by database and carrier capacity.

Autoscaling Configuration

The following CloudFormation example illustrates target tracking for an application fleet. The specific cloud implementation is less important than the architecture: capacity changes automatically while minimum and maximum bounds remain explicit.

Resources:
  ApiScalingTarget:
    Type: AWS::ApplicationAutoScaling::ScalableTarget
    Properties:
      MinCapacity: 4
      MaxCapacity: 30
      ResourceId: !Sub service/${ClusterName}/${ApiServiceName}
      RoleARN: !GetAtt AutoScalingRole.Arn
      ScalableDimension: ecs:service:DesiredCount
      ServiceNamespace: ecs

  ApiCpuScalingPolicy:
    Type: AWS::ApplicationAutoScaling::ScalingPolicy
    Properties:
      PolicyName: api-cpu-target
      PolicyType: TargetTrackingScaling
      ScalingTargetId: !Ref ApiScalingTarget
      TargetTrackingScalingPolicyConfiguration:
        TargetValue: 60
        ScaleOutCooldown: 30
        ScaleInCooldown: 300
        PredefinedMetricSpecification:
          PredefinedMetricType: ECSServiceAverageCPUUtilization

Scale-out is intentionally more responsive than scale-in. Extra capacity has a financial cost, but repeatedly removing and recreating capacity during fluctuating traffic increases latency and operational instability.

CPU would not necessarily be the only production signal. Request concurrency, load-balancer traffic, and application saturation metrics should be evaluated for the actual workload.

Failure and Deployment Flow

Stateless architecture changes the recovery model from repairing machines to replacing capacity.

If one API instance crashes:

1. health checks fail;
2. the load balancer removes the instance;
3. other replicas continue receiving requests;
4. the scheduler creates replacement capacity;
5. the replacement initializes and passes readiness checks;
6. traffic begins reaching the new replica.

No session or uploaded file needs to be copied from the failed instance.

The same property simplifies deployments:

Version 1
API API API API

        |
        | rolling deployment
        v

Version 1       Version 2
API API         API API

        |
        v

Version 2
API API API API

Old and new versions can coexist because requests are not permanently bound to individual instances. Database and API compatibility must still support the deployment window.

A failed release should stop before unhealthy replicas replace all known-good capacity. Readiness checks, deployment health thresholds, application error rates, and automated rollback provide stronger protection than merely verifying that new processes started.

Zone failure follows the same model at larger scale. Healthy replicas in another failure domain continue serving traffic while capacity is restored. More about failure-domain design can be found here: Designing Highly Available Cloud Systems.

Common Mistakes

Stateless application design is frequently undermined by small forms of hidden instance affinity or by autoscaling policies that ignore the rest of the system.

Production Checklist

A stateless application should remain correct when instances are created, terminated, rescheduled, or replaced without warning.

• Remove durable local state: verify that terminating any instance cannot lose business data, sessions, uploads, or queued work.
• Test random instance termination: remove healthy replicas under traffic and verify that requests continue successfully.
• Measure per-instance capacity: load-test safe request rate, concurrency, CPU, memory, and connection usage.
• Select workload-specific scaling signals: use CPU, concurrency, request rate, queue depth, or message age according to actual saturation behavior.
• Reserve scaling headroom: account for startup time, traffic bursts, deployments, and instance failures.
• Bound database pools: calculate total possible connections at maximum fleet size.
• Protect downstream APIs: enforce concurrency limits, rate limits, timeouts, and bounded retries.
• Implement graceful draining: stop new traffic before terminating application processes.
• Make workers idempotent: ensure interrupted and redelivered jobs can execute safely.
• Validate readiness: route traffic only after instances can safely serve production requests.
• Monitor system-wide saturation: correlate application scaling with database, cache, queue, and downstream capacity.
• Exercise rolling deployments: verify that old and new replicas can coexist and unhealthy releases stop automatically.

Conclusion

Stateless architecture makes application compute disposable. Requests can reach any healthy replica, instances can fail without owning irreplaceable state, and capacity can expand or contract without synchronizing individual machines.

The architecture does not eliminate state. It moves business data, sessions, files, jobs, and shared temporary state into databases, caches, object storage, and queues. Those dependencies then become important scalability and reliability boundaries of their own.

Horizontal scaling is therefore an end-to-end capacity problem. Adding replicas improves application capacity only until another dependency becomes the bottleneck. Effective autoscaling combines workload-specific signals, sufficient startup headroom, safe scale-in, dependency protection, and explicit maximum capacity.

Key Takeaway: Make compute replaceable before trying to scale it. Externalize durable state, allow any healthy instance to process any request, protect shared dependencies from uncontrolled concurrency, and design scaling around the capacity of the complete system rather than the application tier alone.

More Articles to Read

• Cloud Architecture Explained: Building Modern Applications
• Virtual Machines vs Containers vs Serverless
• Designing Highly Available Cloud Systems
• Multi-Region Architecture and Disaster Recovery
• Cloud Storage Patterns and Trade-Offs
• Cloud Architecture Best Practices