Scalability in Software
Scalability is the ability of a software system to handle increasing load without becoming unacceptably slow, unstable, or expensive.
The simplest example is a web application that works perfectly with 100 users but starts timing out when 100,000 users arrive. A scalable architecture provides ways to increase capacity as traffic, data, or computational work grows.
Scaling does not necessarily mean building a complicated distributed system. Sometimes the best scaling decision is simply adding more RAM, creating a database index, or caching an expensive query. At larger scale, it may mean adding application servers, database replicas, message queues, data partitions, or multiple geographic regions.
This article explains scalability from the ground up using simple examples and progressively larger architectures.
Table of Contents
- What Is Scalability?
- Vertical Scaling
- Horizontal Scaling
- Load Balancing
- Scaling with Caching
- Scaling the Database
- Asynchronous Processing
- Scaling Components Independently
- Autoscaling
- Example: Scaling an Online Store
- Common Scalability Mistakes
- Production Checklist
- Conclusion
What Is Scalability?
Imagine a simple application receiving 10 requests per second.
Users
│
▼
┌─────────────┐
│ Application │
└──────┬──────┘
│
▼
┌─────────────┐
│ Database │
└─────────────┘
The application works comfortably.
Traffic then increases:
10 requests/sec
↓
100 requests/sec
↓
1,000 requests/sec
↓
10,000 requests/sec
Eventually something reaches its limit. CPU may reach 100%, database connections may be exhausted, disk I/O may become saturated, or a downstream service may stop accepting requests fast enough.
Scaling means increasing the system's ability to handle that additional work.
The important word is system. Increasing application-server capacity does not help much if the database is already the bottleneck.
What Can Grow?
Scalability is not only about the number of users. Different systems experience different kinds of growth.
| Growth | Example |
|---|---|
| Traffic | 1,000 → 100,000 requests/sec |
| Users | 10,000 → 50 million accounts |
| Data | 50 GB → 500 TB |
| Connections | 1,000 → 5 million WebSocket connections |
| Background work | 100 → 1 million jobs/minute |
| Geography | One country → global traffic |
A photo application may primarily struggle with storage. A chat application may struggle with concurrent connections. An analytics platform may struggle with CPU-intensive queries. A payment system may struggle with write throughput.
There is therefore no single architecture called a "scalable architecture." The architecture must scale for the workload that actually grows.
Scalability vs Performance
Performance and scalability are related but different.
Performance asks:
How fast is one request?
Scalability asks:
What happens when the amount of work increases?
Consider two systems:
System A 100 requests/sec → 20 ms latency 10,000 requests/sec → 4,000 ms latency System B 100 requests/sec → 40 ms latency 10,000 requests/sec → 70 ms latency
System A is initially faster, but System B scales much better.
Vertical Scaling
The easiest way to increase capacity is often to make the existing machine more powerful. This is called vertical scaling or scaling up.
For example:
BEFORE
4 CPU
8 GB RAM
100 GB SSD
│
▼
AFTER
16 CPU
64 GB RAM
1 TB NVMe SSD
Nothing fundamental changes in the architecture. The machine simply gets more resources.
Suppose PostgreSQL is running on:
4 CPU 16 GB RAM
and CPU utilization regularly reaches 90%. Moving to:
16 CPU 64 GB RAM
may solve the immediate problem without introducing distributed-system complexity.
Vertical scaling is particularly attractive because it is simple. There are no new network calls, distributed locks, shard routing rules, or replication concerns.
But it has a physical and economic limit. Eventually there is no sufficiently large machine, or the next machine becomes disproportionately expensive.
Vertical scaling also leaves one machine responsible for a large amount of work.
Horizontal Scaling
Horizontal scaling, or scaling out, means adding more machines instead of making one machine larger.
Instead of:
┌─────────────────────┐ │ Very Big Server │ │ 64 CPU / 256 GB │ └─────────────────────┘
the system might use:
┌──────────┐ │ Server 1 │ └──────────┘ ┌──────────┐ │ Server 2 │ └──────────┘ ┌──────────┐ │ Server 3 │ └──────────┘ ┌──────────┐ │ Server 4 │ └──────────┘
If one application server handles approximately 1,000 requests per second, four similar servers may provide capacity for roughly 4,000 requests per second if the workload parallelizes well.
Real scaling is rarely perfectly linear because shared databases, network communication, locks, coordination, and other bottlenecks remain.
Still, horizontal scaling has an important property: capacity can continue growing by adding machines.
Why Stateless Services Scale Better
Horizontal scaling becomes much easier when application instances are stateless.
Consider a server storing login sessions in local memory:
User A ──────► Server 1
Session A stored here
If the next request goes to Server 2:
User A ──────► Server 2 Server 2: "Who is this user?"
The session exists only on Server 1.
A better architecture moves shared state outside application instances:
┌──────────────┐
┌────►│ App Server 1 │
│ └──────┬───────┘
Users ──► Load Balancer │
│ ┌──────▼───────┐
└────►│ App Server 2 │
└──────┬───────┘
│
▼
┌──────────────┐
│ Redis │
│ Sessions │
└──────────────┘
Now either application server can process the request.
More instances can be added or removed without moving user-specific state between them.
Load Balancing
Once multiple application servers exist, traffic needs to be distributed between them.
This is the job of a load balancer.
┌──────────┐
┌───►│ Server A │
│ └──────────┘
│
Users ──► Load Balancer ──► Server B
│
│ ┌──────────┐
└───►│ Server C │
└──────────┘
Suppose 3,000 requests arrive every second.
A simple distribution could be:
Server A → 1,000 req/sec Server B → 1,000 req/sec Server C → 1,000 req/sec
If another server is added:
Server A → 750 Server B → 750 Server C → 750 Server D → 750
The load balancer may also perform health checks. If Server B stops responding:
Server A ✓ Server B ✗ Server C ✓ Server D ✓
new requests can be sent only to the healthy instances.
This demonstrates an important property of many scaling techniques: scalability and availability often improve together. Multiple instances provide both additional capacity and redundancy.
Scaling with Caching
Adding servers is not always the best way to scale.
Sometimes the same expensive work is being performed repeatedly.
Imagine an endpoint:
GET /products/123
It receives 10,000 requests per minute.
Without caching:
10,000 requests
│
▼
10,000 database queries
But product 123 changes only once per hour.
A cache can store the result:
Request
│
▼
Cache
│ │
HIT MISS
│ │
▼ ▼
Return Database
│
▼
Update Cache
If the cache hit rate is 95%, only about 500 of those 10,000 requests need to reach the database.
Caching therefore increases scalability by removing work rather than merely adding resources to perform more work.
For a deeper discussion of layered caching, see Designing Multi-Level Caching Architectures.
Application Cache
Consider an expensive query:
SELECT category_id, COUNT(*)
FROM products
WHERE active = true
GROUP BY category_id;If this calculation takes 800 ms and the result changes infrequently, running it for every request wastes database resources.
The application can cache the result for 60 seconds:
First request
↓
Database → 800 ms
↓
Cache result
Next 5,000 requests
↓
Cache → 2 ms
The exact numbers depend on the workload, but the principle is simple: avoid repeating expensive work when a reusable result is acceptable.
CDN and Edge Caching
Static files can be cached even farther away from the application.
Without a CDN:
User in Europe
│
│ long network path
▼
Application in US
│
▼
image.jpg
With a CDN:
User in Europe
│
▼
European Edge Server
│
▼
image.jpg
Images, JavaScript, CSS, videos, downloads, and some API responses can be served from edge locations.
This simultaneously reduces latency and removes traffic from origin infrastructure.
Scaling the Database
Application servers are relatively easy to replicate because they can often be stateless. Databases are harder because they contain persistent state.
A common architecture eventually reaches this situation:
App 1 ──┐
│
App 2 ──┼──► Database
│ ▲
App 3 ──┤ │
│ BOTTLENECK
App 4 ──┘
Adding App 5 and App 6 may actually make the problem worse because they create even more database traffic.
Database scalability therefore requires its own techniques.
Indexes
Before distributing a database, optimize the work it already performs.
Consider:
SELECT *
FROM orders
WHERE customer_id = 98123;With 200 million orders and no useful index, the database may inspect a huge number of rows.
Adding:
CREATE INDEX idx_orders_customer_id
ON orders(customer_id);can turn an expensive scan into an efficient lookup.
This is an important scalability lesson:
The cheapest scaling technique is often doing less work.
A missing index should not be solved by creating a 20-node database cluster.
Read Replicas
Suppose an application performs:
10% writes 90% reads
One database currently handles everything:
Reads ───┐
├──► Primary Database
Writes ──┘
Read replicas can distribute the read workload:
Writes ─────────────► Primary
│
replication
┌─────┴─────┐
▼ ▼
Replica 1 Replica 2
▲ ▲
│ │
Reads Reads
The primary still processes writes, while multiple replicas process reads.
For example:
Before: Primary → 12,000 reads/sec After: Primary → writes Replica 1 → 4,000 reads/sec Replica 2 → 4,000 reads/sec Replica 3 → 4,000 reads/sec
The trade-off is that replicas may temporarily lag behind the primary. A write performed milliseconds ago may not immediately appear on a replica.
Partitioning
Large tables can sometimes be divided into smaller logical pieces.
For example, an events table may contain five years of data:
events ├── 2022 ├── 2023 ├── 2024 ├── 2025 └── 2026
A query requesting events from August 2026 may only need the 2026 partition instead of considering the entire dataset.
Another example is partitioning by tenant:
Customers A-F Customers G-M Customers N-S Customers T-Z
Partitioning can improve manageability and query efficiency, but the exact benefit depends on the database, partition key, indexes, and query patterns.
Sharding
Eventually one database machine may not be able to hold or process the entire workload.
Sharding distributes data across multiple database servers.
For example:
user_id
│
▼
Shard Function
┌───────┼───────┐
▼ ▼ ▼
Shard A Shard B Shard C
A simple rule might be:
shard = user_id % 4Then:
User 100 → Shard 0 User 101 → Shard 1 User 102 → Shard 2 User 103 → Shard 3 User 104 → Shard 0
Instead of one database handling 100 million users, four databases may each handle roughly 25 million.
But sharding introduces substantial complexity.
Consider:
SELECT COUNT(*)
FROM users;With one database, this is straightforward.
With four shards:
Shard A → count
Shard B → count
Shard C → count
Shard D → count
│
▼
Sum results
Cross-shard joins, transactions, unique constraints, rebalancing, backups, and hotspot prevention also become more difficult.
For this reason, sharding should solve an actual scaling problem rather than being added merely because the application might become large someday.
Asynchronous Processing
Not every operation needs to happen while the user waits.
Imagine creating an account requires:
Create user 100 ms Send email 600 ms Generate avatar 400 ms Analytics event 150 ms CRM synchronization 500 ms -------------------------------- Total 1,750 ms
The request is slow because unrelated work happens synchronously.
A queue changes the architecture:
User
│
▼
API
│
├──► Create User ──► Database
│
└──► Queue
│
├──► Email Worker
├──► Avatar Worker
├──► Analytics Worker
└──► CRM Worker
The API may now return after the essential 100 ms operation while background workers process the remaining tasks.
Queues also help absorb traffic spikes.
Suppose workers can process 5,000 image jobs per minute, but a temporary spike creates 20,000 jobs:
20,000 incoming jobs
│
▼
┌────────────────────┐
│ Queue │
│ ██████████████████ │
└─────────┬──────────┘
│
▼
Worker Pool
5,000/min
The queue acts as a buffer. Workers process the backlog instead of forcing the entire system to handle all 20,000 jobs simultaneously.
If the backlog grows continuously, additional workers can be added:
Queue │ ├──► Worker 1 ├──► Worker 2 ├──► Worker 3 ├──► Worker 4 └──► Worker 5
This makes asynchronous workloads particularly suitable for horizontal scaling.
Scaling Components Independently
Different parts of a system often grow at different rates.
Consider a video platform:
API requests: moderate Video uploads: moderate Video processing: extremely expensive Comments: high Search: high Static delivery: enormous
Scaling everything together would waste resources.
A better architecture separates workloads:
┌──► API Servers
│
Users ──► Gateway ──┼──► Search Service
│
└──► Upload Service
│
▼
Queue
│
┌─────────┼─────────┐
▼ ▼ ▼
Worker Worker Worker
If video processing becomes overloaded, only the worker pool needs additional capacity.
If search traffic grows, only the search infrastructure needs to scale.
This is one reason service boundaries can help scalability. However, splitting an application into dozens of services does not automatically make it scalable. It only provides independent scaling boundaries when those boundaries correspond to real workload differences.
Autoscaling
Traffic is rarely constant.
An online store might experience:
03:00 → 500 req/sec 09:00 → 2,000 req/sec 18:00 → 5,000 req/sec Black Friday → 25,000 req/sec
Keeping enough servers for Black Friday running all year would waste money.
Autoscaling adjusts capacity based on demand:
Low Traffic
Load Balancer
│
├── App 1
└── App 2
High Traffic
Load Balancer
│
├── App 1
├── App 2
├── App 3
├── App 4
├── App 5
└── App 6
Possible scaling signals include:
- CPU utilization
- memory utilization
- requests per second
- active connections
- queue depth
- request latency
Queue depth is particularly useful for worker systems.
For example:
Queue < 1,000 jobs → 5 workers Queue > 10,000 jobs → 20 workers Queue > 50,000 jobs → 100 workers
Autoscaling does not fix an inefficient architecture. If every application instance overloads the same database, automatically creating more instances can increase pressure on the bottleneck.
Example: Scaling an Online Store
Consider how a small online store might evolve as traffic grows.
Stage 1: Everything on one server.
Users │ ▼ ┌────────────────────┐ │ Application │ │ │ │ API │ │ Database │ │ Images │ │ Background Jobs │ └────────────────────┘
This architecture may be completely reasonable for a new product. It is simple, inexpensive, and easy to operate.
Traffic grows and CPU reaches 90%.
Stage 2: Scale vertically.
4 CPU → 16 CPU 8 GB → 64 GB RAM
This buys additional capacity with almost no architectural complexity.
Traffic continues growing. Application processing now consumes most CPU.
Stage 3: Separate application and database.
Users │ ▼ Application Server │ ▼ Database Server
Each component now has dedicated resources.
The application tier becomes overloaded again.
Stage 4: Add horizontal application scaling.
┌──► App 1 ──┐
Users ──► LB ─────┼──► App 2 ──┼──► Database
└──► App 3 ──┘
Product pages become extremely popular and generate millions of identical database reads.
Stage 5: Add caching.
┌──► App 1 ──┐
Users ──► LB ─────┼──► App 2 ──┼──► Redis
└──► App 3 ──┘ │
▼
Database
Most popular product reads are now served from Redis.
Product images consume large amounts of application bandwidth.
Stage 6: Add a CDN.
┌──► CDN ──► Images/CSS/JS
Users ───────────┤
└──► Load Balancer ──► Application
Static traffic no longer reaches the application servers.
Database reads eventually become the next bottleneck.
Stage 7: Add read replicas.
┌──► Replica 1
│
Apps ──► DB Router ──────┼──► Replica 2
│
└──► Primary
▲
│
Writes
Order confirmation currently sends email, creates invoices, updates analytics, and notifies external systems synchronously.
Stage 8: Introduce asynchronous processing.
Order API
│
├──► Database
│
└──► Queue
│
├──► Email Workers
├──► Invoice Workers
├──► Analytics Workers
└──► Integration Workers
The system expands internationally.
Stage 9: Add regional and edge infrastructure.
┌──► US Region
Global Traffic ─────┼──► EU Region
└──► Asia Region
At very large data volumes, some databases may eventually require partitioning or sharding.
The important lesson is that the architecture did not jump directly from one server to a globally distributed microservice platform.
Each change solved a specific bottleneck:
| Problem | Possible Scaling Technique |
|---|---|
| Server CPU exhausted | Vertical or horizontal scaling |
| Too many repeated reads | Caching |
| Application tier overloaded | Load balancer + more instances |
| Database reads overloaded | Indexes, cache, read replicas |
| Database too large | Partitioning or sharding |
| Background processing overloaded | Queue + more workers |
| Static traffic too high | CDN |
| Traffic changes dramatically | Autoscaling |
| Global users experience high latency | CDN and regional deployment |
Common Scalability Mistakes
Scaling Before Measuring
A slow system does not automatically need more servers.
Suppose an API takes three seconds:
API processing 20 ms Database query 2,950 ms Serialization 10 ms Network 20 ms
Adding ten API servers does almost nothing because the database query is the bottleneck.
Measuring CPU, memory, database latency, query plans, I/O, network traffic, queue depth, throughput, and latency percentiles should come before major architectural changes.
Sharding Too Early
Sharding is powerful, but it creates difficult operational problems.
Before sharding, simpler options may include:
Fix slow query
↓
Add index
↓
Add cache
↓
Increase DB resources
↓
Add read replicas
↓
Partition large tables
↓
Shard when necessary
The exact sequence varies by workload, but the principle remains: use the simplest technique that removes the real bottleneck.
Scaling Only the Application Tier
This architecture:
50 Application Servers
│
▼
One overloaded DB
is not meaningfully scalable.
Every shared dependency must be considered: databases, caches, queues, search clusters, object storage, external APIs, and network infrastructure.
Ignoring Hotspots
A system can have many machines and still fail because load is distributed unevenly.
Consider four shards:
Shard A → 10% Shard B → 10% Shard C → 10% Shard D → 70%
Adding shards does not automatically solve the problem if the partitioning strategy keeps sending most traffic to Shard D.
This is why shard keys, cache keys, tenant distribution, partitioning strategies, and traffic patterns matter.
Assuming Scaling Is Linear
Ten servers rarely provide exactly ten times the capacity of one server.
Distributed systems introduce overhead:
- network communication
- serialization
- coordination
- replication
- locking
- load balancing
- shared dependencies
A useful mental model is:
More machines
↓
More potential capacity
+
More coordination
+
More failure modes
+
More operational complexity
Scalability is therefore not about maximizing the number of servers. It is about increasing useful system capacity while keeping complexity, latency, reliability, and cost acceptable.
Production Checklist
- Define what is expected to grow: traffic, data, connections, jobs, or geography.
- Measure the current bottleneck before changing the architecture.
- Optimize expensive queries and algorithms before adding infrastructure.
- Use vertical scaling when it solves the problem simply.
- Keep horizontally scaled application services stateless where practical.
- Use load balancing across redundant application instances.
- Cache frequently requested or expensive-to-compute data.
- Use database replicas when read throughput becomes a bottleneck.
- Use queues for work that does not need to complete synchronously.
- Partition or shard data only when simpler database scaling techniques are insufficient.
- Monitor latency, throughput, saturation, errors, and queue depth.
- Load-test the architecture using realistic traffic patterns.
Conclusion
Scalability is fundamentally about handling more work.
A system can scale by making machines larger, adding more machines, avoiding repeated work through caching, distributing database reads, splitting data, moving background work to queues, or placing content closer to users.
The most important principle is that scalability should follow bottlenecks.
Measure ↓ Find the bottleneck ↓ Apply the simplest effective solution ↓ Measure again ↓ Repeat
A scalable system is therefore not necessarily a complicated system. It is a system whose architecture can evolve as its workload grows.
Key Takeaway: scalability is not about designing for billions of users from day one. It is about understanding where capacity limits exist and having practical ways to increase those limits when growth requires it.
Comments (0)