Designing Highly Available Kubernetes Applications

Kubernetes can restart containers, replace failed pods, and reschedule workloads after node failures, but those mechanisms do not automatically make an application highly available. A workload can have ten replicas and still fail completely if they share the same node, availability zone, overloaded dependency, or incorrectly configured traffic path.

High availability in Kubernetes requires designing around multiple failure domains: containers, pods, nodes, zones, networking components, dependencies, and sometimes entire clusters. The application must retain enough healthy capacity when one of those domains disappears.

The practical goal is not to prevent every failure. It is to make expected failures isolated, recoverable, observable, and small enough that the remaining system continues serving traffic.

Table of Contents

• High Availability in Kubernetes
  • Failure Domains
  • Availability Is End-to-End
• Replica and Workload Distribution
  • Pod Anti-Affinity
  • Topology Spread Constraints
• Health Checks and Traffic Management
  • Readiness and Liveness
  • Graceful Termination
• Capacity and Disruption Management
  • Spare Capacity
  • Pod Disruption Budgets
• Dependency and State Resilience
  • Downstream Failures
  • Stateful Workloads
• Production Design Example
  • Multi-Zone Architecture
  • Failure and Recovery Flow
• Common Mistakes
• Production Checklist
• Conclusion
• More Articles to Read

High Availability in Kubernetes

A highly available Kubernetes application continues providing acceptable service when expected infrastructure or software failures occur. That normally means enough healthy replicas remain reachable while Kubernetes replaces lost capacity.

The important metric is not simply replica count. It is surviving capacity after a failure. Six replicas concentrated on one node can be less resilient than three replicas distributed across three independent failure domains.

Failure Domains

Kubernetes applications operate across nested failure domains:

Cluster
|
+-- Availability Zone A
|   |
|   +-- Node A1
|   |   +-- Pod A
|   |   +-- Pod B
|   |
|   +-- Node A2
|
+-- Availability Zone B
|   |
|   +-- Node B1
|       +-- Pod C
|
+-- Availability Zone C
    |
    +-- Node C1
        +-- Pod D

A container crash might affect one process. A pod failure might affect one application replica. A node failure can remove many pods simultaneously. A zone failure can remove multiple nodes plus zone-specific infrastructure.

Architecture must therefore answer which failures the application is expected to survive. Surviving one pod failure requires a different design from surviving the loss of an availability zone.

Pod, node, and cluster boundaries are explained in greater depth here: Kubernetes Explained: Pods, Nodes, and Clusters.

Availability Is End-to-End

An application is available only when the entire request path works. Healthy pods do not help if ingress capacity is exhausted, DNS fails, a database becomes unavailable, or every application replica waits indefinitely on the same downstream service.

For an HTTP API, the request path might be:

Client
  |
  v
Load Balancer
  |
  v
Ingress
  |
  v
Service
  |
  v
API Pod
  |
  +---- Cache
  |
  +---- Database
  |
  +---- External API

Every synchronous dependency participates in application availability. Adding replicas to the API improves resilience to API-pod failures but does nothing when all replicas depend on the same unavailable database.

This is why high availability should be designed from the client request backward rather than measured only by Kubernetes object status.

Replica and Workload Distribution

Multiple replicas are the foundation of availability for stateless Kubernetes workloads, but replicas need to be distributed across infrastructure failure domains.

The scheduler optimizes placement according to declared requirements and available resources. High-availability requirements therefore need to be represented explicitly rather than assumed.

Pod Anti-Affinity

Pod anti-affinity can prevent or discourage replicas of the same workload from being placed together.

For example, a critical payments API might require replicas to run on different nodes:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: payments-api
spec:
  replicas: 3
  selector:
    matchLabels:
      app: payments-api
  template:
    metadata:
      labels:
        app: payments-api
    spec:
      affinity:
        podAntiAffinity:
          requiredDuringSchedulingIgnoredDuringExecution:
            - labelSelector:
                matchLabels:
                  app: payments-api
              topologyKey: kubernetes.io/hostname
      containers:
        - name: api
          image: registry.example.com/payments-api:5.4.2
          resources:
            requests:
              cpu: "500m"
              memory: "512Mi"
            limits:
              memory: "1Gi"

With three replicas, this prevents two matching pods from being scheduled onto the same node.

Advantage: one node failure cannot remove multiple replicas of this workload.

Disadvantage: strict anti-affinity can make pods unschedulable when too few eligible nodes exist. Availability constraints therefore interact directly with cluster capacity.

For less critical workloads, preferred rather than required anti-affinity can provide better distribution without making placement a hard requirement.

Topology Spread Constraints

Topology spread constraints provide more explicit control over how replicas are distributed across topology domains such as nodes or availability zones.

topologySpreadConstraints:
  - maxSkew: 1
    topologyKey: topology.kubernetes.io/zone
    whenUnsatisfiable: DoNotSchedule
    labelSelector:
      matchLabels:
        app: shipment-api

  - maxSkew: 1
    topologyKey: kubernetes.io/hostname
    whenUnsatisfiable: ScheduleAnyway
    labelSelector:
      matchLabels:
        app: shipment-api

The first constraint strongly distributes replicas across zones. The second attempts to distribute them across nodes while allowing scheduling when perfect node-level balance is impossible.

This distinction is useful in production because not every topology requirement needs the same strictness. Hard constraints improve deterministic failure isolation but increase the risk of Pending pods when infrastructure capacity is uneven.

Suppose nine replicas are distributed across three zones:

Zone A       Zone B       Zone C

Pod 1        Pod 4        Pod 7
Pod 2        Pod 5        Pod 8
Pod 3        Pod 6        Pod 9

3 replicas   3 replicas   3 replicas

Losing one zone leaves six replicas. The application can survive only if those six replicas and their dependencies can handle the resulting traffic.

Distribution therefore solves correlated failure, while capacity planning determines whether surviving replicas remain useful.

Health Checks and Traffic Management

High availability depends on sending traffic only to instances capable of serving it. Kubernetes can maintain ten running pods while the application effectively has zero usable capacity if those pods are deadlocked, initializing, overloaded, or disconnected from required resources.

Health checks and graceful lifecycle handling connect application state with Kubernetes traffic management.

Readiness and Liveness

Readiness answers whether a pod should receive new traffic. Liveness answers whether the container needs to be restarted because it cannot recover without intervention.

readinessProbe:
  httpGet:
    path: /health/ready
    port: 8000
  periodSeconds: 5
  timeoutSeconds: 2
  failureThreshold: 2

livenessProbe:
  httpGet:
    path: /health/live
    port: 8000
  periodSeconds: 10
  timeoutSeconds: 2
  failureThreshold: 3

These checks should not blindly perform deep synchronous tests against every dependency.

For example, if liveness fails whenever PostgreSQL experiences a temporary outage, every API pod can restart simultaneously. Kubernetes then converts a database incident into an application restart storm.

Readiness may reasonably consider whether the application can currently serve useful traffic, while liveness should usually focus on whether the local process can continue making progress.

For a deeper explanation, see: Health Checks, Readiness, and Liveness Probes.

Graceful Termination

High availability also depends on what happens when a healthy pod is intentionally removed during deployment, scaling, maintenance, or rescheduling.

A terminating API pod may still have:

• active HTTP requests
• long-running gRPC calls
• queue messages being processed
• open database transactions
• buffered telemetry

The application needs enough time to stop accepting new work and finish or safely abandon existing work.

spec:
  terminationGracePeriodSeconds: 30
  containers:
    - name: api
      image: registry.example.com/shipment-api:6.1.0
      lifecycle:
        preStop:
          exec:
            command:
              - /bin/sh
              - -c
              - "sleep 5"

A short delay can allow traffic-routing changes to propagate before the process exits. The exact implementation should match the application protocol and ingress architecture rather than relying on arbitrary sleep values as a universal solution.

Applications should also handle termination signals correctly, stop accepting new work, and finish requests within a bounded period.

Availability during deployments depends on both starting new replicas correctly and removing old replicas safely.

Capacity and Disruption Management

A cluster with no spare capacity may operate efficiently during normal conditions but fail to recover when nodes disappear. High availability therefore has an infrastructure cost: some capacity must exist for failures, deployments, and traffic redistribution.

Voluntary disruptions such as node maintenance create a separate problem. Kubernetes needs constraints describing how much application capacity may be intentionally removed at once.

Spare Capacity

Consider a service running 12 pods across four nodes:

Node A: 3 pods
Node B: 3 pods
Node C: 3 pods
Node D: 3 pods

If Node A fails, three replicas disappear immediately. Kubernetes can create replacements only if Nodes B-D have enough allocatable capacity or new nodes can become available quickly.

If each surviving node is already nearly full, replacement pods remain Pending:

Node A FAILED

Lost:
3 application pods

Replacement pods:
Pod X ---- Pending
Pod Y ---- Pending
Pod Z ---- Pending

Reason:
Insufficient CPU / memory on healthy nodes

The application may still operate with nine replicas, but recovery has stalled. Another failure can now reduce capacity further.

Production capacity planning should define an explicit target such as:

• survive loss of one worker node
• survive loss of one availability zone
• retain at least 60% serving capacity during failure
• restore desired capacity within a defined recovery target

Maximum utilization and maximum availability work against each other. Running every node at 90-95% requested capacity minimizes headroom but increases the probability that failures or rolling deployments produce unschedulable pods.

Pod Disruption Budgets

A PodDisruptionBudget limits how much workload availability Kubernetes should voluntarily disrupt at once.

apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
  name: shipment-api
spec:
  minAvailable: 5
  selector:
    matchLabels:
      app: shipment-api

If the application has six healthy replicas, the budget indicates that voluntary operations should preserve at least five.

This is useful during actions such as node drains, but a PodDisruptionBudget is not a shield against involuntary failures. It cannot prevent hardware failure, node crashes, network partitions, or zone outages.

A poorly designed budget can also block infrastructure maintenance. For example, requiring all three replicas of an application to remain available while draining a node hosting one of those replicas can prevent eviction from progressing.

The budget should reflect the application's real minimum serving capacity rather than simply setting the strictest possible value.

Dependency and State Resilience

Kubernetes can maintain application compute, but most production services depend on databases, caches, queues, object stores, third-party APIs, and other microservices. Those dependencies often dominate real application availability.

High availability therefore requires controlling how failures propagate through synchronous and asynchronous dependency chains.

Downstream Failures

Suppose 30 API pods synchronously call a pricing service. The pricing service becomes slow but does not fail completely.

Without bounded timeouts, API requests accumulate:

Clients
   |
   v
30 API Pods
   |
   | thousands of waiting requests
   v
Slow Pricing Service

Worker threads, async tasks, sockets, memory, and connection pools become occupied. Eventually the API becomes unavailable even though Kubernetes reports every pod as Running.

Resilience mechanisms can include:

• timeouts to bound waiting time
• limited retries for transient failures
• exponential backoff to reduce repeated pressure
• circuit breakers to stop repeatedly calling an unhealthy dependency
• bulkheads to isolate resource consumption
• load shedding when demand exceeds safe capacity
• graceful degradation when optional functionality is unavailable

These patterns are covered in greater depth here: Timeouts, Retries, and Exponential Backoff and Circuit Breaker vs Bulkhead vs Load Shedding.

Stateful Workloads

Stateful applications require additional availability design because replacement replicas may need storage attachment, replication recovery, quorum membership, or data synchronization before becoming useful.

A three-member database distributed across three zones may tolerate one member failure, but that depends on the database's own replication and consensus model.

Kubernetes StatefulSets can provide stable pod identities and persistent-volume relationships, but Kubernetes does not implement database replication or guarantee data consistency.

More about choosing stateful workload controllers can be found here: Deployments, ReplicaSets, and StatefulSets.

Production Design Example

Consider a logistics platform serving shipment creation, tracking queries, carrier integrations, and asynchronous tracking events. The availability target requires the public API to remain operational after losing any single worker node or one availability zone.

The architecture uses stateless API replicas across three zones, asynchronous processing for carrier events, and independently replicated external data infrastructure.

Multi-Zone Architecture

                       Internet
                          |
                          v
                 Multi-Zone Load Balancer
                          |
                          v
                   Ingress Replicas
                          |
                    Shipment Service
                          |
          +---------------+---------------+
          |               |               |
          v               v               v
       Zone A          Zone B          Zone C
      +------+         +------+         +------+
      | API  |         | API  |         | API  |
      | API  |         | API  |         | API  |
      | API  |         | API  |         | API  |
      +------+         +------+         +------+
          |               |               |
          +---------------+---------------+
                          |
                +---------+---------+
                |                   |
                v                   v
          Replicated DB        Message Broker
                                    |
                         +----------+----------+
                         |                     |
                         v                     v
                  Tracking Workers       Carrier Workers

Nine API replicas are spread approximately evenly across three zones. Losing one zone removes three replicas, leaving six.

The architecture is highly available only if those six replicas can handle expected traffic. If nine replicas normally run at 80% CPU, losing one-third of them will overload the remaining capacity.

A safer operating target might keep normal utilization low enough that surviving zones can absorb redistributed traffic for the expected failure period.

Failure and Recovery Flow

Suppose Zone B becomes unavailable.

1. Three API replicas and some workers disappear.
2. Load balancing stops sending traffic to unavailable paths.
3. Six surviving API replicas absorb incoming traffic.
4. Request latency and CPU utilization increase.
5. Kubernetes observes missing workload replicas.
6. Replacement pods are scheduled where topology and capacity allow.
7. New pods initialize and pass readiness checks.
8. They begin receiving traffic.
9. Worker consumers resume processing queued events.
10. Desired replica capacity is gradually restored.

Several failure modes can prevent successful recovery.

Insufficient compute capacity: replacement pods remain Pending because surviving zones are full.

Dependency bottleneck: surviving API pods create more database traffic per replica and exhaust database connections.

Queue backlog: losing worker capacity causes tracking events to accumulate faster than they are consumed.

Topology constraints: strict distribution rules may intentionally prevent all missing replicas from being recreated while one zone remains unavailable.

Ingress concentration: ingress replicas concentrated in the failed zone can make healthy application pods unreachable.

Slow startup: large images or expensive initialization can extend degraded operation for minutes.

Monitoring should therefore cover both Kubernetes state and service-level behavior:

• ready, desired, and unavailable replicas
• pods by node and zone
• Pending and unschedulable pods
• node and zone availability
• CPU and memory saturation
• pod startup and readiness duration
• request rate, latency, and error rate
• ingress capacity and failures
• database connection utilization
• cache latency and errors
• queue depth and consumer lag
• dependency timeout and retry rates

The architecture should be tested by deliberately removing capacity rather than assuming the scheduler will behave as expected. A useful failure exercise is draining a node or removing a failure domain in a non-production environment while measuring capacity loss, error rate, recovery time, and dependency saturation.

Multi-region disaster recovery is a separate architecture problem because a single Kubernetes cluster usually does not provide protection against every regional or cluster-wide failure. For a deeper explanation, see: Multi-Region Architecture and Disaster Recovery.

Common Mistakes

Highly available Kubernetes systems fail when redundancy exists only on paper. Replica counts are useful only when workloads are distributed, dependencies remain available, traffic can reach surviving instances, and sufficient capacity exists after failure.

Production Checklist

High availability should be expressed as concrete failure assumptions, scheduling rules, capacity requirements, and recovery tests.

• Define failure targets. Specify whether each critical workload must survive pod, node, zone, or cluster failure.
• Run sufficient replicas. Ensure critical stateless workloads retain serving capacity after expected failures.
• Spread replicas across nodes. Prevent a single machine from becoming the workload's failure domain.
• Distribute across zones. Use topology constraints where zone-level resilience is required.
• Reserve recovery capacity. Verify replacement pods can schedule after the planned infrastructure loss.
• Validate surviving throughput. Load-test the application with one node or zone worth of capacity removed.
• Configure readiness correctly. Route traffic only to replicas capable of serving useful requests.
• Keep liveness local. Avoid restarting healthy processes because shared dependencies are temporarily unavailable.
• Handle termination signals. Stop accepting new work and complete active requests within a bounded grace period.
• Protect voluntary disruptions. Use realistic PodDisruptionBudgets for critical workloads.
• Bound dependency latency. Configure timeouts, controlled retries, and isolation around remote calls.
• Capacity-plan downstream systems. Model connection counts and throughput after application traffic is redistributed.
• Monitor topology. Track replica distribution by node and zone rather than only total replica count.
• Measure recovery time. Track failure detection, scheduling, startup, readiness, and full capacity restoration.
• Exercise failures regularly. Validate node loss, pod loss, dependency outages, and degraded-zone operation before real incidents.

Conclusion

Kubernetes provides powerful recovery primitives, but high availability comes from architecture rather than automatic pod replacement. Replicas must be distributed across failure domains, surviving infrastructure must have enough capacity, traffic must reach only healthy instances, and dependencies must tolerate failures without causing cascading outages.

Availability also has a cost. Spare capacity, additional replicas, multi-zone infrastructure, resilient dependencies, and failure testing consume resources and increase operational complexity. The correct design is therefore based on explicit availability targets rather than maximizing redundancy everywhere.

Key Takeaway: design Kubernetes applications around the capacity that remains after failure, not the capacity available during normal operation. Replica distribution, health checks, graceful lifecycle management, disruption controls, dependency resilience, and tested recovery behavior together determine whether a Kubernetes application is actually highly available.

More Articles to Read

• Kubernetes Explained: Pods, Nodes, and Clusters
• Deployments, ReplicaSets, and StatefulSets
• Services, Ingress, and Networking
• Autoscaling Kubernetes Workloads
• Managing Configuration and Secrets
• Kubernetes Best Practices for Production