Load Balancing Explained: Distributing Traffic at Scale

Load balancing distributes incoming requests across multiple servers, containers, or service instances so that no single backend becomes a bottleneck. It is one of the fundamental building blocks behind scalable and highly available production systems.

A load balancer does more than spread requests. In production, it also determines which backends are healthy, removes failed instances from rotation, manages connections, terminates TLS, and provides a controlled traffic boundary between clients and application infrastructure.

Table of Contents

• Why Load Balancing Exists
• How Load Balancing Works
  • Request Routing Flow
  • Health Checks
• Layer 4 vs Layer 7 Load Balancing
• Routing Algorithms and Backend Selection
• Production Design Example
• Common Mistakes
• Production Checklist
• Conclusion
• Key Takeaway
• More Articles to Read

Why Load Balancing Exists

A single application server has finite CPU, memory, network bandwidth, connection capacity, and operating-system resources. Eventually, increasing the size of that server becomes expensive or impossible.

Instead of relying on one increasingly powerful machine, applications can run multiple instances and distribute traffic between them.

Without load balancing

Clients
   |
   v
+----------------+
| Application    |
| Server         |
+----------------+
   |
   v
 Database


With load balancing

                 +--> Application 1
                 |
Clients --> Load +--> Application 2
           Balancer
                 +--> Application 3
                 |
                 +--> Application 4

This changes the scaling model from primarily vertical scaling to horizontal scaling. Capacity can be increased by adding instances instead of continuously replacing servers with larger machines.

Load balancing also provides a reliability boundary. If one application instance fails, requests can be routed to healthy instances instead of exposing that failure directly to clients.

The main production benefits are:

• Horizontal scalability: capacity can grow by adding backend instances.
• High availability: failed instances can be removed from traffic.
• Better resource utilization: traffic can be distributed across available capacity.
• Operational flexibility: instances can be deployed, replaced, or drained independently.
• Traffic control: routing decisions can incorporate paths, hosts, weights, regions, or backend health.

The load balancer therefore becomes part of the application's availability and scaling architecture, not merely a networking convenience.

How Load Balancing Works

A load balancer sits between clients and a pool of backend targets. Depending on the architecture, those targets might be virtual machines, containers, Kubernetes services, application processes, API gateways, or another layer of load balancers.

For each connection or request, the load balancer identifies eligible targets and selects one according to its routing algorithm and current backend state.

Request Routing Flow

A simplified HTTP request normally follows this path:

1. Client resolves api.example.com
             |
             v
2. Client connects to load balancer
             |
             v
3. Load balancer accepts connection
             |
             v
4. Healthy backend pool is evaluated
             |
             v
5. Routing algorithm selects target
             |
             v
6. Request forwarded to application
             |
             v
7. Application generates response
             |
             v
8. Response returned through load balancer

The client generally does not need to know which backend processed the request. Multiple backend instances appear behind a single stable endpoint.

For HTTP traffic, the load balancer can also add forwarding metadata such as the original client IP address or protocol. Applications and reverse proxies must be configured carefully to trust these headers only when they originate from known infrastructure.

Health Checks

Traffic distribution is useful only when the load balancer knows which targets can safely receive requests. This is usually determined through active health checks.

For example, an application might expose:

from fastapi import FastAPI

app = FastAPI()

@app.get("/health")
def health():
    # Keep load-balancer health checks lightweight.
    # Expensive dependency checks can amplify failures during incidents.
    return {"status": "ok"}

The load balancer periodically calls the endpoint and tracks the result. After a configured number of failures, the target can be marked unhealthy and removed from the routing pool.

                    Load Balancer
                   /      |      \
                  /       |       \
                 v        v        v
              App 1     App 2     App 3
              HEALTHY   FAILED    HEALTHY
                 ^                  ^
                 |                  |
                 +------ traffic ---+

                         X
                      no traffic
                         |
                       App 2

Health checks must represent the ability to serve traffic. A process being alive does not necessarily mean it is ready to process production requests.

At the same time, health checks should not blindly depend on every downstream service. If a database slowdown causes every application instance to fail its load-balancer health check simultaneously, the load balancer can remove the entire fleet and turn a partial dependency problem into a complete outage.

Health-check design is closely related to readiness and liveness behavior. For a deeper explanation, see: Health Checks, Readiness, and Liveness Probes.

Layer 4 vs Layer 7 Load Balancing

Load balancers are commonly divided into Layer 4 and Layer 7 systems. The distinction determines what information is available when routing traffic.

Layer 4 load balancing operates primarily on transport-level information such as IP addresses, ports, and TCP or UDP connections. It does not need to understand HTTP semantics.

Layer 7 load balancing understands application protocols such as HTTP and can route requests using hostnames, URL paths, headers, methods, or cookies.

Layer 7 routing enables architectures where one public endpoint fronts several independent services.

api.example.com/users/*   --> User Service
api.example.com/orders/*  --> Order Service
api.example.com/files/*   --> File Service

Host-based routing can similarly map:

api.example.com     --> API cluster
admin.example.com   --> Admin cluster
static.example.com  --> Static-content service

This flexibility makes Layer 7 load balancing common for HTTP applications, but it also means the load balancer performs more work and becomes more deeply involved in application behavior.

The correct layer depends on the required routing decisions. Using Layer 7 simply because it provides more features can introduce unnecessary processing and operational complexity when transport-level balancing is sufficient.

Routing Algorithms and Backend Selection

After determining which targets are healthy, the load balancer still needs to decide which target receives the next request or connection.

Different algorithms optimize for different assumptions.

• Round Robin distributes requests sequentially across available targets.
• Weighted Round Robin sends proportionally more traffic to targets with larger configured weights.
• Least Connections prefers targets currently handling fewer active connections.
• Least Response Time considers observed backend latency together with load.
• Consistent Hashing maps a stable key to a backend while minimizing remapping when the target pool changes.
• Random or Power of Two Choices uses randomized selection to achieve efficient distribution with relatively little coordination.

The correct algorithm depends heavily on workload characteristics. Round Robin works well when instances and requests are relatively similar, but it does not understand whether one backend is processing several expensive requests while another is mostly idle.

Least Connections can improve distribution for long-lived connections, but connection count is still only a proxy for actual resource consumption. One connection performing expensive computation may consume more resources than hundreds of mostly idle connections.

Consistent hashing is useful when requests benefit from reaching the same backend based on a stable key, such as a tenant, cache key, or shard identifier.

The algorithms themselves are covered in greater depth in Round Robin vs Least Connections vs Consistent Hashing.

Load balancing cannot create capacity that does not exist. If every backend is saturated, changing the routing algorithm only redistributes overload. Scaling, admission control, backpressure, caching, or load shedding may still be required.

Production Design Example

Consider a public API running across multiple application instances. The system must tolerate individual instance failures and support horizontal scaling without changing the public endpoint.

A practical architecture might use DNS to expose a managed Layer 7 load balancer, which terminates TLS and forwards requests to application instances distributed across multiple availability zones.

                         Internet
                            |
                            v
                           DNS
                            |
                            v
                  +-------------------+
                  |  Load Balancer    |
                  |  TLS + HTTP       |
                  +-------------------+
                    /               \
                   /                 \
          Availability Zone A   Availability Zone B
                 |                    |
          +------+------+      +------+------+
          |             |      |             |
          v             v      v             v
        App 1         App 2  App 3         App 4
          \             \      /             /
           \             \    /             /
            +-------------\--/-------------+
                           \/
                       Database

The load balancer performs health checks against each application instance. Failed instances are removed from service while healthy instances continue processing traffic.

Applications should preferably remain stateless from the load balancer's perspective. Session state, uploaded files, job state, and other shared information should live in systems accessible from every instance rather than in local process memory.

This allows requests from the same client to reach different instances:

Request 1 --> App 2
Request 2 --> App 4
Request 3 --> App 1
Request 4 --> App 3

No instance needs to "own" the client. This significantly simplifies scaling, deployments, instance replacement, and failure recovery.

Sticky sessions can provide affinity when local state cannot immediately be removed, but they create coupling between clients and backend instances. That trade-off is covered in Sticky Sessions and Stateless Applications.

During deployments, new instances should become routable only after initialization is complete. Old instances should stop receiving new traffic and receive enough time to complete in-flight requests before termination.

A typical lifecycle is:

1. Start the new application instance.
2. Initialize configuration and dependencies.
3. Pass readiness checks.
4. Register the instance for traffic.
5. Stop routing new traffic to the old instance.
6. Drain existing connections and requests.
7. Terminate the old instance.

This behavior is important for rolling, blue-green, and canary deployments because healthy traffic transitions matter as much as application startup. More about deployment routing can be found here: Traffic Routing Strategies for Zero-Downtime Deployments.

At larger scale, the architecture may contain several load-balancing layers: global routing selects a region, regional load balancers distribute traffic across availability zones, and service-level proxies distribute requests between individual containers.

Each layer solves a different routing problem. Adding layers without a clear responsibility increases latency, configuration surface, observability requirements, and failure modes.

Common Mistakes

Production Checklist

• Deploy multiple backend instances: ensure a single application failure does not remove the service.
• Separate failure domains: distribute instances across availability zones or equivalent infrastructure boundaries.
• Define meaningful health checks: verify that an instance is capable of receiving production traffic.
• Configure health thresholds: avoid removing targets because of isolated transient failures while still detecting real failures quickly.
• Use readiness before registration: prevent partially initialized instances from receiving requests.
• Enable graceful draining: allow active requests and connections to complete before target termination.
• Prefer stateless application instances: avoid unnecessary client-to-instance affinity.
• Choose routing algorithms intentionally: match request distribution to workload and backend characteristics.
• Set explicit connection and request timeouts: prevent abandoned connections from consuming capacity indefinitely.
• Protect forwarding headers: trust client-IP and protocol headers only from controlled proxies and load balancers.
• Monitor backend distribution: detect uneven traffic, hotspots, unhealthy targets, and overloaded zones.
• Track load-balancer saturation: measure connections, throughput, errors, TLS processing, and rejected traffic.
• Capacity-test failure scenarios: verify that remaining instances can handle traffic when part of the fleet disappears.
• Test deployment transitions: confirm that registration, deregistration, and draining work correctly during rolling releases.
• Document the traffic path: make DNS, global routing, proxies, load balancers, gateways, and backend ownership visible to operators.

Conclusion

Load balancing is a core mechanism for building horizontally scalable and highly available systems. It creates a stable traffic entry point while allowing backend instances to be added, removed, replaced, or isolated from traffic independently.

Production load balancing requires more than choosing a routing algorithm. Health checks, failure domains, readiness, connection draining, stateless application design, observability, and capacity planning determine whether the architecture actually survives failures and scaling events.

Key Takeaway

A load balancer does not simply distribute requests. It continuously decides which infrastructure is currently capable of receiving traffic. Reliable systems combine that routing decision with accurate health information, redundant capacity, controlled instance lifecycles, and application architectures designed to run across multiple interchangeable backends.

More Articles to Read

• Round Robin vs Least Connections vs Consistent Hashing
• Designing Highly Available Load Balancing Architectures
• Sticky Sessions and Stateless Applications
• Traffic Routing Strategies for Zero-Downtime Deployments
• Global Load Balancing and Multi-Region Traffic Routing
• Load Balancing Best Practices for Production Systems