What Is a Load Balancer?
A load balancer distributes incoming network connections or application requests across multiple backend servers. Instead of directing every request to one application instance, clients connect to a stable endpoint while the load balancer selects an appropriate backend to handle the traffic.
Load balancers are essential components of scalable and highly available systems. They allow applications to run across multiple servers, remove unhealthy instances from traffic, and introduce additional capacity without requiring clients to discover individual server addresses.
However, adding a load balancer does not automatically make a system scalable or fault-tolerant. The balancing algorithm, health-check configuration, connection management, backend capacity, and failure behavior all determine how well the system performs under real workloads.
Table of Contents
- How a Load Balancer Works
- Load Balancing Algorithms
- Health Checks and Failover
- Load Balancer Architecture
- Implementing Load Balancing
- Production Design Example
- Ready-to-Use Example
- Common Mistakes
- Production Checklist
- Frequently Asked Questions
- Conclusion
How a Load Balancer Works
A load balancer provides a stable frontend address while managing a collection of backend targets. Clients connect to the frontend without needing to know how many application instances exist or where those instances are deployed.
The load balancer maintains routing configuration and, depending on its implementation, information about target health, active connections, or request distribution. It uses that information to select a backend for incoming traffic.
Request Routing Lifecycle
Consider an API deployed across three application instances. A client sends a request to https://api.example.com/orders, which resolves to a load-balanced endpoint.
- Connection: The client connects to the load balancer's frontend address.
- Routing: The load balancer identifies the appropriate target group or backend pool.
- Health filtering: Unhealthy targets are excluded according to configured health policies.
- Selection: A balancing algorithm chooses a target.
- Forwarding: The connection or HTTP request is directed to that target.
- Response: The response travels back to the client through the relevant network path.
The backend selection may occur once per TCP connection or separately for each HTTP request. This distinction matters because one long-lived connection can carry multiple HTTP requests, particularly with HTTP/2.
A load balancer also needs a policy for situations where no healthy targets remain. Depending on the product, it may reject traffic, return an error, or deliberately route to unhealthy targets under a fail-open policy.
Layer 4 vs. Layer 7 Load Balancing
Load balancers are commonly classified by the network information they use to make routing decisions. Layer 4 load balancers operate primarily at the transport layer, while Layer 7 load balancers understand application protocols such as HTTP.
| Capability | Layer 4 | Layer 7 |
|---|---|---|
| Routing information | IP address, port, protocol, connection | Hostname, path, headers, HTTP method |
| Typical protocols | TCP, UDP | HTTP, HTTPS, gRPC |
| Path-based routing | No | Yes |
| HTTP-aware policies | No | Yes |
| TLS termination | Implementation dependent | Common |
| Typical use case | Transport-level distribution | Web applications and APIs |
A Layer 4 load balancer can distribute TCP connections without understanding the HTTP requests inside them. This is useful for databases, message brokers, and services that do not use HTTP.
A Layer 7 load balancer can route /orders/ to one service and /products/ to another. It can also apply HTTP-specific rules such as redirects, header-based routing, and authentication integrations.
Layer 7 processing generally requires more protocol awareness, but that does not mean every Layer 4 product is faster in every deployment. Actual performance depends on implementation, workload, TLS configuration, and network architecture.
Load Balancing Algorithms
After identifying eligible targets, a load balancer needs a strategy for choosing one. Different algorithms optimize for simplicity, active connection distribution, server capacity, or request affinity.
No algorithm can perfectly balance arbitrary workloads. A request that performs an expensive database query may consume substantially more resources than a request returning a cached response.
Round Robin and Weighted Round Robin
Round robin distributes requests or connections among available targets in sequence. With three equally eligible servers, successive selections rotate through the pool.
This approach is simple and works reasonably well when servers have similar capacity and requests have comparable processing costs.
Weighted round robin assigns different selection frequencies to targets. For example, a server with twice the capacity of another may receive a larger share of traffic.
Weights are not a guarantee of proportional CPU utilization. They control selection behavior, while actual resource consumption depends on request characteristics.
Least Connections
Least-connections balancing selects a target with relatively fewer active connections. It can be useful when connection durations vary or when some requests remain open significantly longer than others.
Suppose three backend instances currently have the following connections:
| Server | Active Connections | Selection Preference |
|---|---|---|
| Server A | 120 | Lower |
| Server B | 45 | Higher |
| Server C | 90 | Lower |
Server B is the natural next candidate under a simple least-connections policy. Weighted variants adjust connection counts relative to configured target capacity.
However, connection count is not the same as CPU or memory usage. A server processing a few expensive analytical requests may be more saturated than one maintaining many inexpensive idle connections.
Hash-Based Routing and Session Affinity
Hash-based algorithms select targets using a stable input, such as a client address, session identifier, or application-defined key. Repeated requests with the same key are likely to reach the same target while the target set remains stable.
This behavior is called session affinity or sticky routing. It can be useful for workloads that maintain connection-local or instance-local state, but it also creates operational constraints.
If a target fails, its sessions must move elsewhere. If the application depends on local session data, users may lose state during failover.
Consistent hashing reduces unnecessary key remapping when the target pool changes. It is useful for certain affinity and partitioning workloads, although it does not remove the need to handle failed targets or uneven key popularity.
For most stateless HTTP APIs, shared session storage or signed client-side session data is preferable to making correctness depend on sticky routing.
Health Checks and Failover
A load balancer improves availability only if it can identify targets that should no longer receive traffic. A process may be running while its application is unable to process requests correctly.
Health checks provide information about target readiness, but they must be designed carefully. Incorrect health checks can remove healthy capacity or continue routing traffic to broken instances.
Active and Passive Health Checks
Active health checks send periodic probes to targets. For HTTP applications, the probe commonly requests a dedicated endpoint such as /ready and expects an appropriate success status.
Passive health checks infer target problems from real traffic, such as connection failures, timeouts, or selected upstream errors.
- Active checks: Detect failures even when a target receives little or no user traffic.
- Passive checks: Use actual request outcomes to identify unhealthy targets.
- Failure threshold: Require repeated failures before removing a target.
- Recovery threshold: Require successful checks before returning a target to service.
Aggressive checks detect failures quickly but can cause unnecessary target removal during brief network interruptions. Conservative checks reduce false positives but may continue sending traffic to a failed target for longer.
The exact behavior also depends on the load balancer product. Some managed services can route to all registered targets when every target is marked unhealthy, so failover behavior should never be assumed without testing.
Readiness vs. Liveness
Liveness answers whether an application process is functioning well enough to continue running. Readiness answers whether that instance should receive new traffic.
These signals should not always be identical. An application may be alive but temporarily unready because it is warming caches, initializing resources, or shutting down gracefully.
A readiness endpoint should verify conditions necessary to accept requests without turning every optional downstream dependency into a reason to remove the instance.
For example, if a recommendation service is unavailable but the core API can still process orders, removing every API instance because recommendations are failing may create a much larger outage.
Connection Draining and Graceful Shutdown
When a backend instance is removed during deployment or autoscaling, existing requests may still be executing. Immediately terminating the process can interrupt those requests.
Connection draining allows existing traffic to complete while preventing new requests or connections from being assigned to the retiring instance, according to the load balancer's protocol and configuration.
A typical shutdown sequence marks the instance unready, waits for routing updates to propagate, allows in-flight work to finish, and then terminates the process.
Long-lived WebSocket connections, streaming responses, and HTTP/2 connections require additional attention because they may remain active well beyond ordinary request durations.
Load Balancer Architecture
Load balancing rarely operates in isolation. Production traffic may pass through DNS, a CDN, a regional load balancer, a reverse proxy, and application services before reaching a database or another dependency.
Each layer makes a different routing decision and introduces its own failure modes. The goal is to make those decisions observable and avoid unnecessary duplication.
Load Balancer vs. Reverse Proxy
A load balancer primarily distributes traffic across targets. A reverse proxy accepts requests on behalf of backend servers and may provide HTTP routing, TLS termination, caching, and connection management.
The roles overlap significantly at Layer 7. NGINX, Envoy, and several managed cloud services can act as both reverse proxies and HTTP load balancers.
The distinction becomes clearer when examining the complete request path. DNS, Load Balancers, and Reverse Proxies explains how service discovery, traffic distribution, and application routing interact.
Regional and Global Load Balancing
Regional load balancing distributes traffic among targets within a deployment region, often across multiple availability zones. Global load balancing chooses among regional endpoints based on availability, geography, latency, or configured traffic policies.
DNS-based global routing can direct clients toward different regions, but DNS caching means traffic changes are not instantaneous. Existing client connections may continue using an older endpoint even after DNS responses change.
Global proxy networks and anycast-based architectures can make routing decisions differently, but they also introduce additional infrastructure dependencies and operational trade-offs.
Regional failover must account for stateful services. Sending API traffic to another region does not guarantee that the necessary database writes, sessions, or other state are available there.
Load Balancing and Autoscaling
Autoscaling changes the number of backend instances based on workload signals. The load balancer then distributes traffic among the available targets.
Scaling policies commonly consider CPU utilization, requests per target, queue depth, or application-specific saturation metrics. Choosing the wrong metric can cause scaling to lag behind demand.
New instances also need time to initialize and pass readiness checks. Adding capacity does not help immediately if instances require several minutes to start.
When instances are removed, the load balancer must stop assigning new work before termination. This makes autoscaling, readiness, and connection draining parts of the same operational design.
Implementing Load Balancing
Most production systems should use a mature load balancer rather than implementing network proxying from scratch. Custom code remains useful for understanding selection algorithms, readiness behavior, and application integration.
The following examples show a simple Python selection strategy, an NGINX configuration, and a FastAPI readiness endpoint.
Round Robin in Python
A round-robin selector maintains a list of eligible targets and advances an index for each selection. The implementation below uses a lock to protect the index when multiple threads access the same selector.
from itertools import cycle
from threading import Lock
class RoundRobinSelector:
def __init__(self, targets: list[str]):
if not targets:
raise ValueError("At least one target is required")
self._targets = cycle(targets)
self._lock = Lock()
def select(self) -> str:
with self._lock:
return next(self._targets)
selector = RoundRobinSelector([
"http://api-1:8000",
"http://api-2:8000",
"http://api-3:8000",
])
for _ in range(6):
print(selector.select())This selector demonstrates deterministic distribution, but it does not check health, proxy HTTP traffic, or coordinate state between separate processes.
A real load balancer also needs target discovery, connection management, failure detection, retries where appropriate, and graceful removal of unavailable targets.
NGINX Load Balancer Configuration
NGINX can distribute HTTP requests across multiple upstream servers. The following configuration uses least-connections selection and defines passive failure-detection parameters.
upstream api_backend {
least_conn;
server 10.0.1.11:8000 max_fails=3 fail_timeout=10s;
server 10.0.1.12:8000 max_fails=3 fail_timeout=10s;
server 10.0.2.13:8000 max_fails=3 fail_timeout=10s;
keepalive 32;
}
server {
listen 80;
server_name api.example.com;
location / {
proxy_pass http://api_backend;
proxy_http_version 1.1;
proxy_set_header Connection "";
proxy_set_header Host $host;
proxy_set_header X-Forwarded-For $remote_addr;
proxy_set_header X-Forwarded-Proto $scheme;
proxy_connect_timeout 2s;
proxy_read_timeout 15s;
proxy_send_timeout 15s;
}
}In this example, upstream keep-alive connections can be reused instead of opening a new connection for every request. The keepalive value controls cached idle upstream connections per worker, not a global limit on active connections.
The max_fails and fail_timeout settings support passive failure handling. Active HTTP health checks require additional capabilities or external infrastructure in standard NGINX Open Source deployments.
FastAPI Readiness Endpoint
A readiness endpoint should return HTTP 200 when an instance can accept new traffic and HTTP 503 when it is intentionally unavailable, such as during graceful shutdown.
The following FastAPI example exposes separate liveness and readiness endpoints. A shutdown controller can mark the instance as draining before the process exits.
from fastapi import FastAPI
from fastapi.responses import JSONResponse
app = FastAPI()
app.state.draining = False
@app.get("/live")
async def liveness():
return {"status": "alive"}
@app.get("/ready")
async def readiness():
if app.state.draining:
return JSONResponse(
status_code=503,
content={"status": "draining"},
)
return {"status": "ready"}
def begin_draining() -> None:
app.state.draining = TrueThe begin_draining() function must be called by an actual shutdown or deployment integration. Merely defining it does not enable graceful draining automatically.
In a multi-worker deployment, process-local readiness state also requires care because each worker has independent memory. The platform should coordinate shutdown behavior across workers and routing infrastructure.
Production Design Example
Consider an API platform serving an illustrative peak workload of 12,000 HTTP requests per second. The application runs in one AWS region across three availability zones, with separate instances for authentication, orders, and product catalog APIs.
The primary requirements are horizontal scaling, tolerance of individual instance failures, controlled deployments, and predictable latency under traffic spikes.
Multi-AZ API Architecture
The public API uses an Application Load Balancer with HTTPS listeners and separate target groups for different services. Application instances run on private subnets, and target registration is managed by the deployment platform.
Path-based routing directs /orders/* to the order service and /products/* to the product service. Each target group has its own health-check endpoint and traffic settings.
- DNS: Resolve the public API hostname to the regional load balancer.
- TLS: Terminate public HTTPS connections using a managed certificate.
- Routing: Match HTTP paths to service-specific target groups.
- Health: Select targets according to configured health-check results.
- Backend: Execute application logic on instances distributed across availability zones.
- Observability: Collect load balancer metrics, application traces, and target-level errors.
The load balancer is not the only component that needs redundancy. Databases, caches, service discovery, and network egress may also become shared failure points.
For a broader analysis of these dependencies, see Designing Highly Available Network Architectures.
Capacity Planning and Failure Scenarios
Suppose each application instance can sustainably process 1,500 requests per second under the expected workload while meeting latency objectives. At 12,000 requests per second, eight instances provide the theoretical aggregate throughput.
Running exactly eight instances would leave no headroom for failures, deployments, or traffic fluctuations. If one availability zone becomes unavailable, the remaining zones must still have sufficient capacity.
For illustration, twelve instances distributed evenly across three zones provide four instances per zone. Losing one zone leaves eight instances, with a theoretical capacity of 12,000 requests per second.
That calculation is only a starting point. It assumes identical instance performance, even traffic distribution, no dependency bottlenecks, and immediate redistribution. Real capacity planning needs load testing and additional safety margins.
Load balancer health checks should also be validated during dependency outages. A shared database incident must not unexpectedly cause every target to disappear from service if some API functionality remains available.
Ready-to-Use Example
The following example runs three FastAPI instances behind NGINX using Docker Compose. It demonstrates HTTP request distribution, instance identification, and basic failure handling without requiring cloud infrastructure.
The example is intended for local testing. NGINX is a single container, and the setup does not provide production-grade high availability or TLS termination.
Project Files
Create a directory containing four files: a FastAPI application, a Dockerfile, an NGINX configuration, and a Docker Compose definition.
load-balancer-demo/
app.py
Dockerfile
nginx.conf
compose.yamlEach backend runs the same application but receives a different instance identifier through an environment variable.
Save the following code as app.py.
import os
import asyncio
from fastapi import FastAPI
app = FastAPI()
INSTANCE_ID = os.getenv(
"INSTANCE_ID",
"unknown",
)
@app.get("/ready")
async def ready():
return {
"status": "ready",
"instance": INSTANCE_ID,
}
@app.get("/api/hello")
async def hello():
return {
"message": "Hello from backend",
"instance": INSTANCE_ID,
}
@app.get("/api/slow")
async def slow():
await asyncio.sleep(2)
return {
"message": "Completed slow request",
"instance": INSTANCE_ID,
}The slow endpoint makes it possible to observe how concurrent requests are distributed. It simulates asynchronous waiting, not CPU-intensive work.
Save this Dockerfile in the same directory.
FROM python:3.12-slim
WORKDIR /app
RUN pip install --no-cache-dir \
fastapi \
uvicorn
COPY app.py .
EXPOSE 8000
CMD ["uvicorn", "app:app", "--host", "0.0.0.0", "--port", "8000"]Docker Compose Setup
NGINX will receive requests on port 8080 and distribute them across three backend containers. The upstream uses round robin by default.
Save the following as nginx.conf.
events {
worker_connections 1024;
}
http {
upstream api_backend {
server app1:8000 max_fails=1 fail_timeout=5s;
server app2:8000 max_fails=1 fail_timeout=5s;
server app3:8000 max_fails=1 fail_timeout=5s;
}
server {
listen 80;
location / {
proxy_pass http://api_backend;
proxy_http_version 1.1;
proxy_set_header Host $host;
proxy_set_header X-Forwarded-For $remote_addr;
proxy_set_header X-Forwarded-Proto $scheme;
proxy_connect_timeout 1s;
proxy_read_timeout 10s;
proxy_send_timeout 10s;
}
}
}The failure settings allow NGINX to temporarily avoid targets after selected unsuccessful attempts. Because this is passive detection, an unavailable target may still receive a request before NGINX recognizes the failure.
Save the following as compose.yaml.
services:
app1:
build: .
environment:
INSTANCE_ID: app1
expose:
- "8000"
app2:
build: .
environment:
INSTANCE_ID: app2
expose:
- "8000"
app3:
build: .
environment:
INSTANCE_ID: app3
expose:
- "8000"
nginx:
image: nginx:stable-alpine
ports:
- "127.0.0.1:8080:80"
volumes:
- ./nginx.conf:/etc/nginx/nginx.conf:ro
depends_on:
- app1
- app2
- app3Only NGINX publishes a port to the host. The backend containers remain accessible within the Docker Compose network.
Run and Verify Load Balancing
Start the containers and validate the NGINX configuration before sending requests.
docker compose up -d --build
docker compose exec nginx nginx -t
for i in $(seq 1 9); do
curl -s http://127.0.0.1:8080/api/hello
echo
doneThe responses should show requests reaching app1, app2, and app3. With all instances healthy, the default round-robin strategy distributes successive HTTP requests among them.
Next, stop one backend and repeat the test.
docker compose stop app2
for i in $(seq 1 9); do
curl -s -i http://127.0.0.1:8080/api/hello
done
docker compose start app2Some initial requests may fail while NGINX detects that app2 is unavailable. The remaining instances can continue handling traffic according to the configured upstream failure policy.
For a concurrency experiment, send several slow requests in parallel.
seq 1 9 | xargs -P 9 -I {} \
curl -s http://127.0.0.1:8080/api/slowThis test illustrates how requests can execute concurrently across backend instances. It is not a reliable throughput benchmark because the endpoint performs only an asynchronous sleep.
For production benchmarking, use a workload generator such as k6 or Locust and measure throughput, p95 and p99 latency, error rates, and resource utilization at both the load balancer and application layers.
Common Mistakes
Load balancing problems often appear only during traffic spikes, deployments, or partial failures. A configuration that distributes requests correctly under normal conditions may still behave poorly when capacity changes.
Assuming Round Robin Balances Work
Round robin distributes requests, not computational cost. A backend receiving a few expensive requests can become overloaded while another handles many inexpensive requests.
Measure CPU, memory, queue depth, active requests, and latency before choosing a balancing strategy. Weighted or least-connections policies may help, but application-level bottlenecks still need to be addressed.
Using Incorrect Health Checks
A health endpoint that always returns HTTP 200 may keep broken instances in rotation. Conversely, a health endpoint that depends on every external service can remove all application capacity during a minor dependency outage.
Separate liveness from readiness and test health-check behavior during startup, shutdown, dependency failures, and partial degradation.
Retrying Unsafe Requests
Some load balancers and proxies can retry requests when an upstream connection fails. Retrying state-changing operations without understanding whether the first attempt completed can cause duplicate orders or payments.
Use bounded retries only when operation semantics permit them. For state-changing APIs, backend idempotency keys and deduplication are often required.
Ignoring Load Balancer Availability
Three healthy backend instances provide little value if every request depends on one failing load balancer process. Self-hosted load balancing therefore requires redundancy at the frontend as well as the backend.
Managed load balancers generally provide built-in redundancy, but their availability guarantees, regional dependencies, quotas, and failover behavior still need to be understood.
Production Checklist
Before deploying a load balancer, validate its target selection, health policies, capacity limits, and behavior during application changes. The following checks cover the most important production concerns.
- Health checks: Configure readiness probes with appropriate intervals, timeouts, and recovery thresholds.
- Distribution: Select algorithms based on actual request cost, connection behavior, and target capacity.
- Timeouts: Align frontend, upstream, and application deadlines.
- Draining: Allow in-flight requests to complete during deployments and instance termination.
- Capacity: Reserve sufficient headroom for traffic spikes and failure-domain outages.
- Security: Enforce TLS, trusted forwarding headers, and private backend connectivity.
- Observability: Monitor target health, request distribution, error rates, latency, and saturation.
- Failover: Test instance failures, zone outages, and scenarios with no healthy targets.
Frequently Asked Questions
The following questions address common decisions about load balancer placement, session handling, cloud services, and availability requirements.
Is a Load Balancer the Same as a Reverse Proxy?
Not exactly. A load balancer focuses on distributing traffic among targets, while a reverse proxy accepts requests on behalf of backend services and may perform routing, caching, TLS termination, and other HTTP functions.
At Layer 7, the responsibilities often overlap. Many reverse proxies can load balance HTTP requests, and many HTTP load balancers function as reverse proxies.
Does a Load Balancer Make Sense with One Server?
Sometimes. A managed load balancer can provide TLS termination, routing rules, and a stable endpoint even when only one backend exists.
However, one backend still represents a single application failure point. A load balancer cannot create backend redundancy when no alternative capacity exists.
Does Load Balancing Require Sticky Sessions?
No. Stateless applications can generally serve successive requests on different instances without requiring session affinity.
Sticky sessions may be useful for certain stateful workloads, but they complicate failover and can produce uneven traffic distribution. Shared session storage is often preferable for ordinary web applications.
What Is the Difference Between AWS ALB and NLB?
AWS Application Load Balancer operates at Layer 7 and supports HTTP-aware routing, including hostname and path rules. AWS Network Load Balancer operates primarily at Layer 4 and is designed for transport-level traffic such as TCP and UDP.
Selection depends on protocol requirements, routing features, connection behavior, and operational constraints rather than simply choosing whichever service appears faster.
Can a Load Balancer Become a Single Point of Failure?
Yes. A single self-hosted load balancer instance can become a single point of failure even when backend servers are redundant.
High-availability designs use redundant frontend capacity, managed load balancing, or other failover mechanisms. The complete request path must be examined because DNS, network routes, and shared dependencies can also create failure points.
Conclusion
A load balancer provides a stable entry point and distributes traffic among backend servers. It supports horizontal scaling, removes unhealthy targets from normal routing, and enables application deployments without requiring clients to track individual server instances.
Reliable load balancing requires more than selecting an algorithm. Health checks, connection draining, capacity planning, timeout budgets, and frontend redundancy determine how the system behaves during real failures.
Key Takeaway: A load balancer distributes traffic, but high availability comes from the entire architecture. Healthy backend selection, sufficient spare capacity, safe deployments, and tested failover behavior are what make load-balanced systems reliable.
More Articles to Read
These related articles explore networking fundamentals, service discovery, and deployment strategies that complement load balancing.
Comments (0)