Health Checks, Readiness, and Liveness Probes

A running process is not necessarily a healthy service. An application can be alive while its connection pool is exhausted, still initializing, unable to accept traffic, stuck in a deadlock, or waiting for a dependency that will never recover without intervention.

Health checks, readiness probes, and liveness probes provide different signals about the operational state of an application. Health checks expose diagnostic information, readiness determines whether an instance should receive traffic, and liveness determines whether the process is unable to recover and should be restarted.

These signals are especially important in containerized and orchestrated environments. Incorrect probes can make incidents worse: aggressive liveness checks can create restart loops, while incorrect readiness checks can route production traffic to instances that cannot serve it.

Table of Contents

• Health Is Not a Single State
  • Healthy, Alive, and Ready
• Health Checks
  • What Health Checks Should Report
• Readiness Probes
  • When an Instance Is Not Ready
  • Dependency-Aware Readiness
• Liveness Probes
  • What Liveness Should Detect
  • Why Dependencies Should Not Control Liveness
• Startup and Graceful Shutdown
  • Safe Startup
  • Safe Shutdown
• Production Design Example
  • Designing Probes for an Order Service
• Common Mistakes
• Production Checklist
• Conclusion
  • Key Takeaway
• More Articles to Read

Health Is Not a Single State

Applications are often described as either healthy or unhealthy, but production systems have several operational states.

An instance can be:

• running but still initializing
• alive but unable to accept traffic
• ready while an optional dependency is degraded
• serving traffic but approaching resource saturation
• stuck and unable to recover without restart

Representing all these conditions with one endpoint creates ambiguity about what infrastructure should do when the check fails.

Healthy, Alive, and Ready

The three concepts answer different questions:

Consider an instance that is starting:

Process started
      |
      v
Liveness:  PASS
Readiness: FAIL
      |
      | initialization
      v
Configuration loaded
Connections established
Caches initialized
      |
      v
Liveness:  PASS
Readiness: PASS
      |
      v
Traffic begins

The process is alive throughout startup, but it should not receive traffic until initialization is complete.

Health Checks

A health check provides information about the current operational condition of a service. Unlike a liveness or readiness probe, a diagnostic health endpoint does not necessarily map directly to one infrastructure action.

It can expose whether the application itself is operational and whether important internal components are functioning.

What Health Checks Should Report

A useful health response can distinguish between the application and its dependencies:

from fastapi import FastAPI

app = FastAPI()


@app.get("/health")
async def health():
    return {
        "status": "degraded",
        "components": {
            "application": "healthy",
            "database": "healthy",
            "redis": "healthy",
            "recommendations": "unavailable",
        },
    }

This provides more operational information than a simple response such as:

200 OK
healthy

However, public health endpoints should not expose sensitive internal details such as credentials, infrastructure addresses, exception traces, database names, or topology information.

Health checks should also be cheap. A health endpoint that executes expensive database queries or calls several external APIs every few seconds creates its own production workload.

Health checks can report degraded operation without declaring the instance unable to serve traffic. If Recommendations is unavailable but the application can safely omit recommendations, the service can remain operational in degraded mode. More about this behavior can be found in Designing Graceful Degradation Strategies.

Readiness Probes

A readiness probe answers one specific question: should new production traffic be routed to this instance?

When readiness fails, the instance should normally remain running but stop receiving new traffic.

                    Load Balancer
                         |
            +------------+------------+
            |            |            |
            v            v            v
        Instance A   Instance B   Instance C

        READY ✓      READY ✓      NOT READY ✗
            |            |
            v            v
         traffic      traffic

This allows temporary problems to recover without restarting the process.

When an Instance Is Not Ready

Readiness can fail when the application cannot currently serve its expected traffic correctly.

Typical examples include:

• startup initialization has not completed
• required configuration is unavailable
• a required connection pool cannot be initialized
• the instance is intentionally draining during shutdown
• local resources required for requests are unavailable
• the service has entered a state where accepting additional requests would be unsafe

A simple implementation can maintain application readiness explicitly:

from fastapi import FastAPI, Response, status

app = FastAPI()

application_ready = False


@app.get("/ready")
async def readiness(response: Response):
    if not application_ready:
        response.status_code = status.HTTP_503_SERVICE_UNAVAILABLE
        return {"ready": False}

    return {"ready": True}

The application can set the state after initialization succeeds and clear it before graceful shutdown begins.

Dependency-Aware Readiness

One of the most important readiness decisions is whether dependency failures should make the instance unready.

Consider ten application instances using the same database:

              Database
                 X
             unavailable
                 |
      +----------+----------+
      |          |          |
      v          v          v
   App 1      App 2      App 3
   NOT READY  NOT READY  NOT READY
      ...
   App 10
   NOT READY

If every instance marks itself unready because the shared database is temporarily unavailable, the load balancer removes every instance from service. That may be appropriate if no useful operation can function without the database, but it can also unnecessarily remove endpoints that could still provide cached, static, or degraded functionality.

Readiness should therefore depend on whether the instance can serve the required contract, not simply whether every dependency reports healthy.

An optional Recommendation Service should usually not control readiness for a checkout service. A local initialization failure that makes all checkout requests impossible probably should.

Dependency checks also need protection. If 500 application instances perform readiness checks against the database every second, the health mechanism itself generates 500 additional requests per second during an incident.

Where dependency state is needed, prefer cheap checks, cached state, or signals derived from real application traffic rather than expensive synthetic queries.

Liveness Probes

A liveness probe determines whether the application process is still functioning well enough to recover without external intervention.

If liveness repeatedly fails, the orchestrator can restart the container or process.

Application
    |
    | liveness passes
    v
Keep running


Application
    |
    | liveness repeatedly fails
    v
Restart process
    |
    v
New application instance

Restarting is a much stronger action than removing an instance from traffic. For that reason, liveness checks should generally be simpler and more conservative than readiness checks.

What Liveness Should Detect

Liveness is useful for states where the process is unlikely to recover by itself, such as:

• deadlocked execution
• completely stuck event loop or worker system
• irrecoverable internal state
• critical background thread or runtime failure
• internal progress permanently stopping when progress is required

A basic liveness endpoint may intentionally do very little:

@app.get("/live")
async def liveness():
    return {"alive": True}

This appears trivial, but it verifies that the application runtime can still receive and process the probe request.

More advanced systems can verify internal progress. For example, a worker process expected to continuously consume messages can track the last successful processing loop:

import time


last_worker_heartbeat = time.monotonic()


def record_worker_progress() -> None:
    global last_worker_heartbeat
    last_worker_heartbeat = time.monotonic()


def worker_is_alive(max_stall_seconds: float = 30.0) -> bool:
    return (
        time.monotonic() - last_worker_heartbeat
        < max_stall_seconds
    )

The threshold must account for legitimate idle or long-running operations. Otherwise healthy workers can be restarted simply because they did not update the heartbeat quickly enough.

Why Dependencies Should Not Control Liveness

A dangerous design is making liveness depend directly on an external database:

Liveness Probe
      |
      v
Application
      |
      v
Database
      X

Probe FAILS
      |
      v
Restart application

If the database remains unavailable, the restarted application checks again:

Database outage
      |
      v
App fails liveness
      |
      v
Restart
      |
      v
App fails liveness
      |
      v
Restart
      |
      v
App fails liveness
      |
      v
Restart storm

Now a database outage has become both a database incident and an application restart incident.

Restarting the caller does not repair the database. It can actually increase load because every restarted instance reconnects, reloads configuration, rebuilds caches, performs migrations or initialization checks, and generates new dependency traffic.

This is an important reliability principle: liveness should primarily answer whether restarting this process is likely to improve the situation.

Startup and Graceful Shutdown

Probe design must account for application lifecycle. Instances need time to initialize before serving traffic and time to stop receiving new traffic before termination.

Without lifecycle-aware probes, deployments can produce temporary errors even when the application itself is correct.

Safe Startup

Some applications require significant startup work:

• loading configuration
• initializing database pools
• loading machine-learning models
• warming local caches
• discovering services
• loading large data files

Traffic should not reach the instance until required initialization has completed.

Container starts
      |
      v
Application process starts
      |
      v
Load configuration
      |
      v
Initialize resources
      |
      v
Warm required state
      |
      v
READINESS = TRUE
      |
      v
Receive traffic

Slow startup must also not be mistaken for process failure. In orchestration environments that support a separate startup probe, it can protect slow-starting applications from liveness checks until initialization completes.

Safe Shutdown

Shutdown should happen in the opposite order. An instance should stop accepting new requests before existing work is terminated.

Termination requested
        |
        v
READINESS = FALSE
        |
        v
Stop receiving new traffic
        |
        v
Finish in-flight requests
        |
        v
Stop background consumers
        |
        v
Flush important buffers
        |
        v
Close connections
        |
        v
Exit process

This is especially important during rolling deployments. If the process exits immediately while the load balancer still considers it ready, active requests can be terminated and new requests may continue arriving during shutdown.

Message consumers should similarly stop receiving new messages before terminating and complete or safely release work already in progress.

Production Design Example

Consider an Order Service deployed as multiple container replicas. It uses PostgreSQL for orders, Redis for non-critical caching, a Payment Service for checkout, and a message broker for publishing order events.

These dependencies should not all affect probes in the same way.

Designing Probes for an Order Service

                       Load Balancer
                            |
                +-----------+-----------+
                |                       |
                v                       v
          Order Service A         Order Service B
          LIVE  ✓                 LIVE  ✓
          READY ✓                 READY ✓
                |                       |
                +-----------+-----------+
                            |
          +-----------------+-----------------+
          |                 |                 |
          v                 v                 v
      PostgreSQL          Redis         Payment Service
       REQUIRED          OPTIONAL          REQUIRED
          |
          v
     Message Broker
     REQUIRED for
     event publishing

The service can expose separate endpoints:

from fastapi import FastAPI, Response, status

app = FastAPI()

startup_complete = False
shutting_down = False


@app.get("/health")
async def health():
    return {
        "status": "healthy",
        "startup_complete": startup_complete,
        "shutting_down": shutting_down,
    }


@app.get("/live")
async def live():
    # No remote dependency checks.
    # Successful execution demonstrates that the
    # application runtime can still handle requests.
    return {"alive": True}


@app.get("/ready")
async def ready(response: Response):
    if not startup_complete or shutting_down:
        response.status_code = status.HTTP_503_SERVICE_UNAVAILABLE
        return {"ready": False}

    return {"ready": True}

This intentionally does not query every dependency on every probe request.

Instead, the application can maintain internal dependency state based on connection management and real traffic:

from dataclasses import dataclass


@dataclass
class DependencyState:
    database_available: bool = False
    broker_available: bool = False
    redis_available: bool = False


dependency_state = DependencyState()


def can_accept_orders() -> bool:
    return (
        dependency_state.database_available
        and dependency_state.broker_available
    )

If the service contract requires PostgreSQL and durable event publishing before an order can be safely accepted, their state can contribute to readiness:

@app.get("/ready")
async def ready(response: Response):
    ready_to_serve = (
        startup_complete
        and not shutting_down
        and can_accept_orders()
    )

    if not ready_to_serve:
        response.status_code = status.HTTP_503_SERVICE_UNAVAILABLE
        return {"ready": False}

    return {"ready": True}

Redis does not control readiness because it is used only as an optimization. If Redis fails, Order Service can continue using PostgreSQL with appropriate capacity protection.

Payment Service also does not necessarily control instance readiness. Every Order Service replica uses the same Payment Service, so removing all replicas from traffic does not repair the dependency.

Instead, payment calls can use bounded timeouts, retries, and a circuit breaker:

Order Service
      |
      v
Payment Circuit Breaker
      |
   +--+--+
   |     |
CLOSED  OPEN
   |     |
   v     v
Payment  Fail checkout quickly
Service  without restarting Order Service

Timeout and retry behavior is covered in Timeouts, Retries, and Exponential Backoff, while circuit breakers and resource isolation are covered in Circuit Breaker vs Bulkhead vs Load Shedding.

Now consider a rolling deployment with four replicas:

Before deployment

A READY
B READY
C READY
D READY


Deploy A'

A  -> readiness false -> drain -> terminate
A' -> start -> initialize -> readiness true


Then B, C, D follow.

At every stage:
healthy ready replicas continue receiving traffic.

Correct readiness behavior allows new instances to warm before entering service and old instances to drain before termination.

Monitoring should track probe state and business behavior separately:

order.instances.ready
order.instances.not_ready

order.readiness_failures
order.liveness_failures
order.container_restarts

order.database.available
order.broker.available
order.redis.available

order.requests
order.error_rate
order.p95_latency
order.checkout_success_rate

A high restart rate with repeated liveness failures indicates a different problem from temporary readiness failures during deployment. Similarly, all replicas becoming unready simultaneously often suggests a shared dependency or configuration problem rather than independent instance failures.

Common Mistakes

Probe configuration can directly affect production availability. A health mechanism that takes aggressive corrective action based on the wrong signal can amplify the incident it was intended to detect.

Production Checklist

Probe design should define both what each signal means and what infrastructure action follows when it fails.

• Separate probe responsibilities: use health for diagnostics, readiness for traffic eligibility, and liveness for process recovery.
• Keep liveness simple: avoid remote dependencies unless their failure truly means the process itself must restart.
• Define readiness from the service contract: remove an instance from traffic only when it cannot serve required operations safely.
• Classify dependencies: distinguish required dependencies from optional services that support degraded operation.
• Keep checks inexpensive: avoid expensive database queries and external API calls from frequently executed probes.
• Protect slow startup: allow required initialization to complete before liveness enforcement begins.
• Start unready: do not accept production traffic before required initialization completes.
• Drain before shutdown: become unready before terminating active workers and connections.
• Use realistic thresholds: tolerate short scheduling delays and temporary latency spikes.
• Avoid restart storms: do not restart healthy callers because a shared downstream dependency is unavailable.
• Protect diagnostic information: keep detailed internal health data away from unrestricted public endpoints.
• Monitor readiness transitions: unexpected flapping can indicate resource pressure or dependency instability.
• Monitor restart frequency: repeated liveness-triggered restarts should be treated as a production signal.
• Test rolling deployments: verify new instances become ready before receiving traffic and old instances drain correctly.
• Test dependency outages: confirm optional failures degrade safely while critical failures produce the intended readiness behavior.

Conclusion

Health checks, readiness probes, and liveness probes represent different aspects of application health and should trigger different operational responses. Health checks provide diagnostic visibility, readiness controls whether an instance receives traffic, and liveness determines whether restarting the process is likely to restore normal operation.

The most important distinction is that a dependency failure does not automatically mean an application process is dead. Shared dependency outages should usually be handled through timeouts, circuit breakers, degradation, and recovery rather than restarting every caller.

Key Takeaway

Readiness asks whether an instance should receive traffic; liveness asks whether the process should be restarted. Keep those decisions separate, make probes cheap and conservative, treat optional dependency failures as degraded operation where possible, and design startup and shutdown so traffic reaches only instances capable of serving it safely.

More Articles to Read

• Building Reliable Systems: Core Reliability Patterns Explained
• Timeouts, Retries, and Exponential Backoff
• Circuit Breaker vs Bulkhead vs Load Shedding
• Designing Graceful Degradation Strategies
• Failure Recovery in Distributed Systems
• Reliability Best Practices for Production Systems