What Is Horizontal Scaling?

Horizontal scaling is the practice of increasing a system's capacity by adding more machines, instances, containers, or nodes instead of making a single machine more powerful. If one application server can handle 2,000 requests per second, horizontal scaling might run five servers behind a load balancer to distribute a much larger workload.

The idea is simple, but production horizontal scaling requires more than adding instances. Applications must avoid local state dependencies, traffic must be distributed correctly, databases and caches must scale with the application tier, and failures must be expected as the number of components grows.

Table of Contents

• How Horizontal Scaling Works
• Horizontal vs Vertical Scaling
• What Makes an Application Horizontally Scalable?
  • Keep Application Instances Stateless
  • Distribute Traffic Across Instances
  • Handle Instance Failures
• Horizontal Scaling Beyond Web Servers
  • Workers and Message Processing
  • Databases
  • Caches
• Where Horizontal Scaling Breaks
• Autoscaling in Production
• Designing for Horizontal Scaling
• Conclusion
  • More Articles to Read

How Horizontal Scaling Works

Consider an API running on one server. As traffic grows, CPU utilization reaches 90%, request queues increase, and p99 latency starts rising.

One option is to replace the server with a larger machine. Horizontal scaling takes a different approach: run additional copies of the application and distribute requests among them.

Clients → Load Balancer → Application Instances

Suppose one instance sustainably handles 1,500 requests per second. Four equivalent instances might theoretically provide approximately 6,000 requests per second of application capacity.

Real scaling is rarely perfectly linear. Shared dependencies such as databases, caches, external APIs, network bandwidth, locks, and message brokers eventually become bottlenecks.

Horizontal scaling increases capacity only while the rest of the architecture can support the additional concurrency.

Horizontal vs Vertical Scaling

Horizontal scaling is often compared with vertical scaling. Both increase capacity, but they do so differently.

Vertical scaling is often the simplest solution at smaller scale. Increasing a database from 8 GB to 32 GB of memory may be considerably easier than introducing partitioning, replication, and distributed coordination.

Horizontal scaling becomes valuable when workloads exceed practical single-machine limits, traffic changes dynamically, or availability requires the system to survive individual instance failures.

What Makes an Application Horizontally Scalable?

Running multiple copies of an application does not automatically make it horizontally scalable. Requests must be able to reach different instances without changing application correctness.

Keep Application Instances Stateless

Local process memory creates one of the most common scaling problems.

Suppose an application stores authenticated sessions in memory:

sessions: dict[str, dict] = {}


def create_session(session_id: str, user_id: int) -> None:
    sessions[session_id] = {
        "user_id": user_id,
    }

A user logs in through Instance A, so the session exists only in Instance A's memory. The next request reaches Instance B, which cannot find it.

Sticky sessions can temporarily hide this problem by routing the same client to the same instance, but they create uneven load distribution and complicate failure recovery.

A more scalable design stores shared state outside application instances:

Application Instances → Redis / Database / Object Storage

Instances can then be created, terminated, or replaced without losing application state.

Local memory is still useful for disposable caches and immutable configuration. The important distinction is that correctness must not depend on a particular application instance surviving.

Distribute Traffic Across Instances

A load balancer provides a stable endpoint and distributes incoming traffic among healthy instances.

Simple round-robin routing may work when requests have similar costs. More sophisticated systems can consider active connections, availability zones, health, or other routing information.

The load balancer also separates clients from the lifecycle of individual instances. An instance can disappear and another can replace it without clients needing to discover a new endpoint.

Load balancing is covered in more depth in Load Balancing Explained: Distributing Traffic at Scale.

Handle Instance Failures

Horizontal architectures contain more components, so individual failures should be considered normal.

An instance can fail because of:

• hardware or virtual-machine failure;
• out-of-memory termination;
• failed deployments;
• application crashes;
• network problems;
• availability-zone failures.

Health checks should stop traffic from reaching unhealthy instances, while the orchestration platform replaces failed capacity.

Graceful shutdown is equally important. During deployments or scale-in, an instance should stop receiving new traffic while allowing in-flight requests to complete before termination.

Horizontal scaling works best when application instances are disposable. Starting or terminating one should not require manual state recovery.

Horizontal Scaling Beyond Web Servers

The application tier is usually the easiest part of a system to scale horizontally. Once more application instances generate additional database queries, cache operations, messages, and network calls, bottlenecks often move downstream.

Workers and Message Processing

Background processing naturally supports horizontal scaling when work is distributed through a queue.

Producers → Queue → Worker Pool

If ten workers cannot process incoming jobs quickly enough, more workers can consume from the queue.

Suppose each worker processes approximately 50 jobs per second:

10 workers × 50 jobs/s = 500 jobs/s
20 workers × 50 jobs/s = 1,000 jobs/s

This approximation holds only until another resource saturates. If every job performs a database write, doubling workers may simply move the bottleneck to the database.

Queue depth and the age of the oldest pending message are usually better autoscaling signals than CPU for asynchronous workers because they directly represent unfinished work.

Databases

Databases are harder to scale horizontally because state must remain correct across nodes.

Read-heavy workloads can often add replicas:

Writes → Primary → Read Replicas

This increases read capacity but does not substantially increase write capacity. Replicas can also return stale data because replication is commonly asynchronous.

When one database can no longer handle the write workload or dataset, partitioning can distribute data across multiple database nodes. A shard key determines which node owns each record.

Unlike stateless application scaling, database partitioning introduces difficult questions around cross-shard queries, transactions, rebalancing, hot partitions, and failure recovery.

Database Sharding: Strategies and Trade-Offs explains these design decisions in detail.

Caches

Distributed caches can also scale horizontally by partitioning keys across nodes.

Instead of storing every cache entry on one server:

Cache Cluster → Node A | Node B | Node C

Adding nodes increases aggregate memory and throughput, but changing cluster membership can move keys between nodes. Systems therefore need predictable partitioning, replication, and failure handling.

More application instances can also increase cache concurrency dramatically. Connection counts, hot keys, network bandwidth, CPU, memory pressure, and eviction rate should be monitored rather than assuming the cache will scale automatically with the application tier.

Where Horizontal Scaling Breaks

A common misconception is that capacity can grow indefinitely by adding instances. Distributed systems eventually encounter shared bottlenecks or work that cannot be parallelized efficiently.

Consider an API tier:

100 instances → Shared Database

If every API request executes two database queries, scaling from 10 to 100 application instances can multiply the database concurrency even though database capacity has not changed.

Common scaling limits include:

• Database capacity. CPU, IOPS, locks, connections, and transaction contention become saturated.
• Hot partitions. One shard or key receives disproportionate traffic.
• External service quotas. A downstream API may impose fixed request limits.
• Shared locks. More workers increase contention rather than throughput.
• Network bandwidth. Additional compute cannot exceed network or storage throughput limits.
• Serial work. Some operations have dependencies that prevent useful parallel execution.

For example, suppose application capacity increases while database throughput remains fixed:

Once the database becomes saturated, adding API instances can make performance worse by increasing connection pressure, lock contention, retries, and queued work.

Horizontal scaling moves bottlenecks; it does not eliminate them.

Autoscaling in Production

Horizontal scaling becomes particularly useful when capacity can change automatically with demand. Autoscaling adds instances when load increases and removes them when demand falls.

CPU utilization is a common signal, but it is not always the best representation of system pressure.

Useful scaling signals depend on the workload:

Scaling also has delay. A new virtual machine or container may need seconds or minutes to start, register with the load balancer, initialize dependencies, and become healthy.

If traffic grows faster than new capacity becomes available, reactive autoscaling arrives too late. Systems with predictable traffic peaks can use scheduled or predictive scaling, while sufficient baseline capacity absorbs sudden bursts.

Scale-in requires caution as well. Removing capacity too aggressively can create oscillation where the platform repeatedly adds and removes instances. Cooldown periods, stabilization windows, and conservative scale-in policies reduce this behavior.

Designing for Horizontal Scaling

Horizontal scalability should be evaluated across the complete request path rather than only at the application layer.

A production design should answer several questions:

• Where does state live? Critical state should not depend on the lifecycle of an application instance.
• How is traffic distributed? Load balancing should route only to healthy capacity and avoid persistent hotspots.
• What happens during scale-out? New instances should become useful quickly without overwhelming shared dependencies during startup.
• What happens during scale-in? Instances should drain traffic and background work safely before termination.
• Which dependency saturates next? Database connections, cache throughput, queues, storage, and external APIs need their own capacity models.
• How is overload handled? Rate limits, queues, backpressure, load shedding, and bounded retries should prevent cascading failures.
• Which metrics trigger scaling? Signals should represent actual workload pressure rather than simply using CPU by default.

Capacity planning can start with a simple approximation. If one instance safely handles 800 requests per second at the desired p99 latency and expected peak traffic is 8,000 requests per second:

8,000 req/s ÷ 800 req/s = 10 instances

Running exactly ten instances leaves no room for failures or traffic variation. If one instance fails, remaining instances immediately exceed the tested operating point.

Production capacity therefore needs headroom for instance loss, deployments, traffic bursts, uneven load distribution, and downstream degradation.

Monitoring should include request throughput, p50/p95/p99 latency, error rate, CPU and memory saturation, active connections, queue depth, database connection utilization, cache latency, and downstream throttling.

The key metric is not simply instance count. The goal is maintaining the required throughput and latency while preserving enough spare capacity to survive expected failures.

Conclusion

Horizontal scaling increases system capacity by adding more nodes rather than continually increasing the resources of one machine. It is fundamental to high-traffic APIs, worker systems, distributed caches, databases, and cloud architectures because capacity can grow dynamically while individual nodes remain replaceable.

The application tier is usually straightforward to scale when instances are stateless and traffic is distributed through load balancing. The difficult problems appear in shared state and dependencies: databases, caches, queues, network limits, hot partitions, connection pools, and external services.

The most important principle is that horizontal scaling is an architectural property, not an instance-count setting. Adding nodes provides useful capacity only when state management, traffic distribution, failure handling, downstream dependencies, and operational limits are designed to scale with them.

More Articles to Read

• Scalability in Software
• Scalability in Software System Design — A Practical Guide