Designing Highly Available Cloud Systems

High availability is not achieved by running more servers. A system becomes highly available when individual component failures do not become application-wide outages and recovery happens automatically within an acceptable time.

Cloud infrastructure makes redundant compute, networking, storage, and managed databases easier to provision, but redundancy alone is insufficient. Applications must distribute capacity across independent failure domains, remove unhealthy components from traffic, protect dependencies from cascading failures, and maintain enough spare capacity to survive degraded conditions.

The central design question is therefore not whether components can fail. They will. The question is how much of the system fails with them, how quickly failure is detected, and what continues operating during recovery.

Table of Contents

• Availability as an Architecture Property
• Designing Across Failure Domains
  • Active-Active Compute
  • Capacity During Failure
• Health Checks and Automatic Recovery
  • Liveness and Readiness
• Protecting Stateful Dependencies
  • Database Failover
  • Cache and Queue Failures
• Preventing Cascading Failures
• Production Design Example
  • Infrastructure Example
  • Failure Flow
• Common Mistakes
• Production Checklist
• Conclusion
  • More Articles to Read

Availability as an Architecture Property

Availability describes whether a system can successfully provide its required functionality when clients need it. A service may be running while still being unavailable because requests time out, dependencies are saturated, or responses are incorrect.

Availability targets are often expressed as percentages, but the operational meaning is easier to understand as an outage budget.

Each additional nine usually increases cost and complexity. A system targeting 99.99% availability cannot routinely depend on manual failover taking 30 minutes because one such incident can consume a large portion of the annual outage budget.

Availability must also be evaluated end to end. An API fleet with excellent availability still cannot serve requests if its only database is unavailable.

Client
  |
  v
Load Balancer
99.99%
  |
  v
Application
99.99%
  |
  v
Database
99.9%

The complete request path cannot be more available than a critical dependency that every successful request requires. Adding more application replicas does not solve a single point of failure deeper in the architecture.

Availability should therefore be designed from the required user operation backward: identify every component necessary for that operation and determine what happens when each component becomes unavailable.

Designing Across Failure Domains

A failure domain is a group of resources that can fail together. A physical host is one failure domain. A rack, power system, availability zone, data center, or entire cloud region can represent progressively larger failure domains.

Running three replicas does not provide meaningful redundancy if all three depend on the same underlying failure domain.

Poor Distribution

Availability Zone A
+--------------------------------+
| App 1   App 2   App 3          |
+--------------------------------+

Zone A fails
      |
      v
Entire application unavailable


Better Distribution

Availability Zone A     Availability Zone B
+------------------+    +------------------+
| App 1   App 2    |    | App 3   App 4    |
+------------------+    +------------------+
          \                    /
           +--------+---------+
                    |
              Load Balancer

For many cloud applications, distributing production capacity across multiple availability zones provides a practical balance between availability, latency, cost, and operational complexity. Regional failure requires a different architecture and is covered separately in Multi-Region Architecture and Disaster Recovery.

Active-Active Compute

Stateless application compute commonly uses an active-active architecture. Instances in multiple failure domains simultaneously receive production traffic.

This is preferable to keeping idle standby compute because active instances are continuously exercised. A hidden configuration or dependency problem in a standby environment may otherwise remain undetected until failover is required.

Active-active compute also simplifies failure recovery:

1. A node or zone becomes unhealthy.
2. Health checks detect the failure.
3. The load balancer stops sending new traffic to affected targets.
4. Healthy capacity continues processing requests.
5. The scheduler or autoscaler replaces lost capacity.

For this model to work, application instances should not own durable state. Session state, uploads, job progress, and business data should survive instance replacement. More about this design can be found here: Scaling Stateless Applications.

Capacity During Failure

Redundancy is useful only if remaining infrastructure can carry production traffic after capacity is lost.

Consider an application running across two zones:

Normal traffic: 800 requests/second

Zone A capacity: 500 requests/second
Zone B capacity: 500 requests/second

Normal utilization:
800 / 1000 = 80%

Zone A fails:

Remaining capacity = 500 requests/second
Required capacity  = 800 requests/second

Result: overload

The architecture is technically multi-zone but is not capable of surviving a zone failure at peak traffic.

A simple capacity requirement for two equal failure domains is:

capacity_per_zone >= expected_peak_load

With three zones, capacity can be distributed more efficiently while still maintaining enough capacity after one failure. The exact reserve depends on failure assumptions, autoscaling startup time, traffic variability, and cost tolerance.

High availability requires failure headroom. Infrastructure intentionally operating near 100% capacity cannot absorb meaningful failures.

Health Checks and Automatic Recovery

Redundant infrastructure does not improve availability unless failures can be detected and unhealthy resources removed automatically. Health checks therefore form part of the traffic-control system rather than merely providing monitoring information.

A load balancer should stop routing requests to an application instance that cannot safely process them. A scheduler should replace processes that have crashed. Autoscaling should restore lost capacity when failures reduce the healthy fleet.

Detection thresholds require trade-offs. Aggressive health checks reduce failure-detection time but increase the risk of removing healthy instances during temporary latency spikes. Conservative checks avoid false positives but continue routing traffic to broken instances longer.

Liveness and Readiness

Liveness answers whether the application process is functioning. Readiness answers whether it should currently receive new traffic.

These are different questions.

from fastapi import FastAPI, Response, status

app = FastAPI()


@app.get("/health/live")
async def liveness() -> dict[str, str]:
    # If the process can execute this endpoint,
    # it is generally alive.
    return {"status": "ok"}


@app.get("/health/ready")
async def readiness(response: Response) -> dict[str, str]:
    # Check only dependencies required to safely
    # process the application's critical requests.
    database_ready = await database.ping(timeout=0.2)

    if not database_ready:
        response.status_code = status.HTTP_503_SERVICE_UNAVAILABLE
        return {"status": "not_ready"}

    return {"status": "ready"}

A readiness check should not blindly test every external dependency. Suppose an optional recommendation service becomes unavailable. If every API instance reports itself unhealthy because recommendations cannot be loaded, the load balancer may remove the entire application fleet and convert a minor feature outage into a complete platform outage.

Health checks should represent whether the instance itself can provide the functionality expected through that traffic path.

Startup behavior matters as well. New instances should not receive traffic before database connections, configuration, application caches, or other required initialization has completed.

Protecting Stateful Dependencies

Stateless compute is relatively easy to replace. Stateful infrastructure is harder because recovery must preserve data and consistency while restoring service.

Databases, caches, queues, and storage systems therefore need availability strategies that match their role. Replicating everything identically is rarely appropriate because each component has different durability and consistency requirements.

Database Failover

A common relational database architecture uses a primary node for writes and one or more replicas that maintain copies of the data.

                  Application
                      |
                      v
               Database Endpoint
                      |
                      v
                Primary Node
                 /         \
                v           v
           Replica A     Replica B

If the primary fails, a healthy replica can be promoted:

Primary X
   |
   | failure detected
   v
Replica A ---- promotion ----> New Primary
                                  |
                                  v
                         Database Endpoint
                                  |
                                  v
                            Application

Failover is not instantaneous. Failure detection, promotion, DNS or endpoint changes, connection establishment, and application retries all contribute to recovery time.

Applications therefore need to tolerate database connection failures during the transition. Connection pools must discard broken connections rather than repeatedly returning them to callers.

Replication can also introduce lag. After a write commits on the primary, an asynchronous replica may temporarily return older data. Applications requiring read-after-write consistency should route those reads appropriately rather than assuming every replica is immediately current.

Cache and Queue Failures

A cache should normally improve performance without becoming the only durable copy of critical business data. If a cache cluster becomes unavailable, the application may fall back to the database, but this creates another failure mode: the fallback traffic can overload the database.

Recovery therefore needs rate limits, request coalescing, circuit breakers, or gradual cache warming rather than immediately redirecting unrestricted cache traffic to persistent storage.

Queues have different semantics. Durable queues protect asynchronous workflows by retaining messages while consumers fail or restart. Consumers should assume that messages may be delivered more than once.

async def process_payment_job(job: PaymentJob) -> None:
    existing = await payment_repository.find_by_idempotency_key(
        job.idempotency_key
    )

    if existing is not None:
        # Duplicate delivery is safe.
        return

    result = await payment_gateway.charge(
        customer_id=job.customer_id,
        amount=job.amount,
        idempotency_key=job.idempotency_key,
    )

    await payment_repository.record(
        job=job,
        external_payment_id=result.payment_id,
    )

If a worker dies after charging the customer but before acknowledging the queue message, another worker may receive the same job. Idempotency prevents recovery from creating duplicate side effects.

Preventing Cascading Failures

Many large outages begin with a small failure. One dependency slows down, callers accumulate waiting requests, connection pools fill, threads or workers become occupied, retries multiply traffic, and eventually unrelated functionality becomes unavailable.

This is a cascading failure.

Database slows
      |
      v
API requests wait longer
      |
      v
Worker pool saturates
      |
      v
Clients retry
      |
      v
Traffic increases
      |
      v
Database receives more work
      |
      +------------------+
      |                  |
      +---- feedback ----+

Highly available systems need mechanisms that bound the amount of damage one failing dependency can cause.

• Timeouts prevent requests from waiting indefinitely.
• Bounded retries recover from transient failures without creating unlimited amplification.
• Exponential backoff and jitter spread retries over time.
• Circuit breakers temporarily stop calls to dependencies that are repeatedly failing.
• Bulkheads isolate resource pools so one workload cannot consume everything.
• Load shedding rejects lower-priority work when capacity is exhausted.
• Queues absorb temporary differences between producer and consumer throughput.

These patterns are covered in greater depth in Timeouts, Retries, and Exponential Backoff and Circuit Breaker vs Bulkhead vs Load Shedding.

Graceful degradation is another important availability technique. If product recommendations fail, an e-commerce platform can still display products. If analytics ingestion fails, checkout should normally continue. Availability improves when optional functionality does not sit on critical request paths.

Production Design Example

Consider a production ordering platform serving customer APIs while processing payments, inventory reservations, notifications, and asynchronous fulfillment workflows.

The architecture needs to survive individual compute failures and the loss of one availability zone without losing committed orders.

                           Internet
                              |
                              v
                       CDN / Edge Layer
                              |
                              v
                        Load Balancer
                         /          \
                        /            \
                       v              v
                Availability     Availability
                   Zone A           Zone B
                +-----------+    +-----------+
                | API       |    | API       |
                | API       |    | API       |
                +-----------+    +-----------+
                       \              /
                        \            /
                         +----+-----+
                              |
                 +------------+------------+
                 |                         |
                 v                         v
          Database Primary             Distributed
                 |                        Cache
                 v
          Standby / Replica
                 |
                 v
              Queue
             /     \
            v       v
        Worker    Worker
        Zone A    Zone B

Customer traffic is distributed across active application instances in both zones. Losing one API instance requires no application-level failover because the load balancer already distributes traffic across other healthy instances.

The database uses replicated durable storage and automated failover. The application expects temporary database errors during promotion and reconnects through a stable database endpoint.

Order fulfillment is asynchronous. Once the durable order transaction commits and a corresponding job is safely published, temporary worker failures do not require the customer request to remain open.

Workers use idempotency keys because messages can be delivered again during recovery. Notification failures do not roll back a successfully created order.

Infrastructure Example

The following CloudFormation fragment demonstrates the multi-zone compute portion using AWS resources. Equivalent architecture can be implemented using other cloud platforms.

Resources:
  ApplicationLoadBalancer:
    Type: AWS::ElasticLoadBalancingV2::LoadBalancer
    Properties:
      Scheme: internet-facing
      Subnets:
        - !Ref PublicSubnetZoneA
        - !Ref PublicSubnetZoneB

  ApplicationTargetGroup:
    Type: AWS::ElasticLoadBalancingV2::TargetGroup
    Properties:
      VpcId: !Ref VPC
      Protocol: HTTP
      Port: 8080
      HealthCheckPath: /health/ready
      HealthCheckIntervalSeconds: 15
      HealthyThresholdCount: 2
      UnhealthyThresholdCount: 3

  ApplicationAutoScalingGroup:
    Type: AWS::AutoScaling::AutoScalingGroup
    Properties:
      MinSize: "6"
      DesiredCapacity: "6"
      MaxSize: "20"
      VPCZoneIdentifier:
        - !Ref PrivateSubnetZoneA
        - !Ref PrivateSubnetZoneB
      TargetGroupARNs:
        - !Ref ApplicationTargetGroup
      LaunchTemplate:
        LaunchTemplateId: !Ref ApplicationLaunchTemplate
        Version: !GetAtt ApplicationLaunchTemplate.LatestVersionNumber

The important property is not the specific cloud resource. Application capacity exists in independent zones, health checks control traffic eligibility, and unhealthy capacity can be replaced automatically.

Minimum capacity should be selected from failure requirements rather than normal average utilization. If losing half of the fleet causes immediate saturation, the deployment is redundant but not highly available.

Failure Flow

Suppose Zone A becomes unavailable during peak traffic.

1. Detection: load-balancer health checks begin failing for Zone A targets.
2. Traffic isolation: unhealthy targets stop receiving new requests.
3. Continuation: Zone B serves traffic using reserved failure capacity.
4. Scaling: the compute platform attempts to restore the desired healthy capacity where infrastructure remains available.
5. Monitoring: alerts fire for lost zone capacity, increased utilization, elevated latency, and dependency pressure.
6. Recovery: restored Zone A instances become ready and gradually rejoin traffic.

The system should not depend on an engineer manually changing DNS or launching replacement instances for this expected failure class.

Deployments should follow the same principle. New application versions should enter traffic gradually, and readiness or application metrics should stop the rollout when the new version fails. Availability engineering includes deployment failures because software releases are one of the most common ways healthy infrastructure becomes unhealthy.

Common Mistakes

Highly available architecture often fails because redundancy exists on diagrams but does not survive realistic capacity, dependency, or recovery conditions.

Production Checklist

High availability should be validated through measurable failure behavior rather than inferred from the number of deployed replicas.

• Define availability targets: translate the required percentage into an outage budget and recovery expectations.
• Map failure domains: identify which compute, network, database, cache, queue, and storage resources can fail together.
• Remove single points of failure: verify that every component on critical request paths has an appropriate recovery strategy.
• Reserve failure capacity: ensure remaining healthy infrastructure can serve peak traffic after the expected failure.
• Separate liveness and readiness: restart dead processes without unnecessarily removing functional capacity.
• Measure failover time: record actual detection, promotion, reconnection, and recovery duration.
• Protect retries: configure deadlines, bounded attempts, exponential backoff, and jitter for transient failures.
• Verify idempotency: ensure retried requests and redelivered messages cannot duplicate critical side effects.
• Monitor saturation: alert on connection pools, worker utilization, queue age, database capacity, and other bottlenecks before exhaustion.
• Test degraded dependencies: verify that optional service failures produce graceful degradation rather than complete outages.
• Exercise zone failure: remove an entire failure domain under realistic load and confirm that remaining capacity survives.
• Test failed deployments: verify that unhealthy releases stop or roll back before known-good capacity disappears.

Conclusion

Highly available cloud systems are built by containing failures rather than attempting to prevent every failure. Redundant compute, multiple failure domains, automated health checks, state replication, controlled retries, idempotent processing, and sufficient capacity headroom allow individual failures to occur without becoming complete outages.

The architecture must be evaluated end to end. Adding application replicas provides little value if all requests require a single database, remaining capacity cannot survive a zone failure, or a retry storm can overwhelm dependencies.

Higher availability also carries real cost. More replicas, additional failure domains, replicated data, automated recovery, spare capacity, testing, and operational tooling should be justified by concrete reliability requirements rather than added automatically.

Key Takeaway: High availability is the ability to continue useful service while components are failing. Design explicit failure boundaries, keep enough healthy capacity after failure, automate expected recovery paths, and test those paths before production incidents require them.

More Articles to Read

• Cloud Architecture Explained: Building Modern Applications
• Virtual Machines vs Containers vs Serverless
• Multi-Region Architecture and Disaster Recovery
• Cloud Storage Patterns and Trade-Offs
• Scaling Stateless Applications
• Cloud Architecture Best Practices