Handling Partial Failures in Production Systems

A distributed system rarely fails as one complete unit. More often, one dependency becomes slow, one availability zone loses connectivity, one database replica falls behind, or one request times out while the operation still completes in the background.

These partial failures are difficult because the caller cannot always determine whether an operation failed, succeeded, or is still running. Production resilience depends on combining timeouts, retries, idempotency, circuit breakers, isolation, load shedding, and graceful degradation without creating retry storms or duplicated side effects.

Table of Contents

• Why Partial Failures Are Different
• Common Types of Partial Failures
  • Network Failures
  • Slow Services
  • Service Crashes
  • Dependency Failures
  • Resource Exhaustion
  • Stale and Inconsistent State
• Failure Handling Patterns
  • Timeouts
  • Retries
  • Exponential Backoff
  • Jitter
  • Circuit Breakers
  • Bulkheads
  • Load Shedding
  • Graceful Degradation
  • Failure Pattern Comparison
• Failure Detection
  • Health Checks
  • Heartbeats
  • Passive Monitoring
  • Synthetic Monitoring
• Designing for Failures
  • Idempotent Operations
  • Stateless Services
  • Fallback Strategies
  • Redundancy
  • Bounded Concurrency
  • Asynchronous Recovery
• Production Implementation Example
  • Resilient Request Flow
  • FastAPI Client with Timeouts and Retries
  • Idempotent Order Creation
  • Circuit Breaker Implementation
  • Kubernetes Health Configuration
• Failure Scenarios in Production
  • Payment Provider Timeout
  • Database Connection Pool Exhaustion
  • Message Consumer Failure
  • Regional Dependency Outage
• Observability for Partial Failures
  • Failure Metrics
  • Distributed Tracing
  • Structured Logging
  • Alerting on Symptoms
• Common Mistakes
• Production Checklist
• Conclusion
• Key Takeaway
• More Articles to Read

Why Partial Failures Are Different

In a single-process application, a function either returns, raises an error, or the process terminates. Distributed calls have an additional state: the outcome may be unknown.

A caller can time out while the remote service continues processing. Retrying may be necessary, but the retry can also duplicate the original operation.

Client              API              Payment Provider
  │                  │                       │
  │── create order ─►│                       │
  │                  │── charge card ───────►│
  │                  │                       │
  │                  │       charge succeeds│
  │                  │◄──── response lost ──X│
  │                  │                       │
  │◄──── timeout ────│                       │
  │                  │                       │
  │── retry order ──►│── charge again ─────►│

The timeout tells the client that no response arrived before the deadline. It does not prove that the remote operation failed.

Partial failures create several forms of uncertainty:

• Transport uncertainty: the request or response may have been lost.
• Execution uncertainty: the remote operation may still be running.
• Commit uncertainty: a database transaction may have committed before the connection failed.
• Replication uncertainty: one replica may contain newer data than another.
• Ownership uncertainty: a failed worker may recover after another worker takes over.

Production resilience is therefore not based on eliminating failures. It is based on limiting failure propagation and making recovery safe.

Common Types of Partial Failures

Partial failures appear at every layer of a distributed architecture. Different failure types require different handling because a retry that helps one failure may amplify another.

Network Failures

Networks can drop, delay, duplicate, or reorder packets. A connection can be established successfully and fail while the response is being transmitted.

Common symptoms include connection resets, DNS failures, TLS handshake errors, packet loss, and timeouts.

Production impact: callers may retry an operation that already completed, or hold threads and connections while waiting for a response that will never arrive.

Recommended response: use bounded timeouts, retry only safe operations, attach idempotency keys to state-changing requests, and record the remote operation identifier when available.

Slow Services

A service can remain technically available while responding too slowly to be useful. Slow database queries, overloaded worker pools, garbage collection, and downstream saturation can all increase latency.

Slow dependencies are often more dangerous than immediate failures because requests continue accumulating while resources remain occupied.

Normal dependency:

100 requests/sec
20 ms average latency
≈ 2 concurrent requests


Slow dependency:

100 requests/sec
5 sec latency
≈ 500 concurrent requests

Concurrency grows roughly with arrival rate multiplied by response time. A latency increase can therefore exhaust connection pools, memory, threads, and file descriptors even when request volume remains unchanged.

Service Crashes

A service may terminate before processing, during processing, or after committing state but before acknowledging success.

The recovery strategy depends on where durable progress was recorded. Stateless requests can be retried elsewhere, while stateful workflows require checkpoints or idempotency records.

Production note: process restart is not sufficient recovery when the crashed process owned in-memory queues, locks, or uncommitted workflow state.

Dependency Failures

A healthy service can fail because one of its dependencies is unavailable. A product API may depend on pricing, inventory, recommendations, authentication, and database services.

Product API
├── Product Database
├── Inventory Service
├── Pricing Service
└── Recommendation Service

Not every dependency is equally critical. Product data may be mandatory, while recommendations may be optional.

A resilient design classifies dependencies as:

• Required: the request cannot produce a valid result without the dependency.
• Degradable: the request can return reduced functionality.
• Asynchronous: work can be completed later without blocking the response.

Resource Exhaustion

Services fail partially when a limited resource is exhausted. The process may remain alive while being unable to serve new requests.

Typical exhausted resources include:

• database connections;
• HTTP connection pools;
• worker threads;
• CPU time;
• memory;
• disk space;
• message-consumer capacity;
• external API quotas.

Retries often make resource exhaustion worse. A service already unable to process 1,000 requests should not receive 3,000 retry attempts.

Stale and Inconsistent State

Replicated systems may return different results from different nodes. A write can succeed on the leader while a replica remains behind.

A client that writes data and immediately reads from a stale replica may incorrectly conclude that the write failed.

Recommended response: define consistency requirements explicitly. Use leader reads, session consistency, version checks, or read-after-write routing when stale data would trigger incorrect recovery.

Failure Handling Patterns

No single resilience pattern handles every failure. Timeouts limit waiting, retries recover transient errors, circuit breakers reduce pressure, bulkheads isolate resources, and graceful degradation preserves useful functionality.

Timeouts

A timeout limits how long a caller waits for an operation. Every remote call should have a bounded deadline because the default operating-system timeout may be far longer than the user or upstream service can tolerate.

Different phases may require different limits:

• Connection timeout: maximum time to establish a connection.
• Read timeout: maximum wait for response data.
• Write timeout: maximum time to transmit the request.
• Pool timeout: maximum wait for an available connection.
• Overall deadline: total time allowed for the operation.

Incoming request deadline: 2,000 ms

Authentication:       150 ms
Database query:       500 ms
Payment provider:    900 ms
Response serialization:
                      100 ms
Safety margin:        350 ms

Downstream timeouts must fit inside the upstream request deadline. A service with two seconds remaining should not start a downstream call configured to wait five seconds.

Advantages

• Prevents requests from waiting indefinitely.
• Releases connections and worker capacity.
• Limits latency propagation through service chains.

Disadvantages

• A timeout may occur while the remote operation succeeds.
• Aggressive values create false failures.
• Static values may not fit every operation.

When to Use / Real-World Use Cases

Use timeouts for every database, cache, message broker, HTTP, DNS, and external API operation. Choose values from service-level objectives and measured tail latency rather than arbitrary defaults.

Example

import httpx

timeout = httpx.Timeout(
    connect=0.5,  # Fail quickly when a host cannot be reached.
    read=1.0,     # Bound the wait for response data.
    write=0.5,    # Avoid blocking while sending a request.
    pool=0.2,     # Do not wait indefinitely for a pooled connection.
)

Retries

Retries repeat operations that may have failed because of a temporary condition. They work well for short network interruptions, leader changes, throttling, and transient dependency errors.

Retries are dangerous when the operation is not idempotent or the dependency is overloaded.

Advantages

• Recover from brief transient failures.
• Hide short failovers from callers.
• Improve success rates without manual intervention.

Disadvantages

• Increase dependency load during incidents.
• Extend request latency.
• Can duplicate non-idempotent operations.
• Can hide persistent failures until retry budgets are exhausted.

When to Use / Real-World Use Cases

Retry connection resets, selected timeouts, HTTP 429 responses, and temporary 5xx responses when the operation is safe to repeat. Do not retry validation failures, authentication errors, or deterministic business-rule failures.

Example

Exponential Backoff

Exponential backoff increases the delay after each failed attempt. This gives the dependency time to recover and reduces repeated pressure.

def backoff_delay(
    attempt: int,
    base_seconds: float = 0.2,
    maximum_seconds: float = 5.0,
) -> float:
    """
    Increase retry delay exponentially while enforcing an upper bound.
    """
    return min(base_seconds * (2 ** attempt), maximum_seconds)

A retry sequence with a 200-millisecond base may wait approximately 200 milliseconds, 400 milliseconds, 800 milliseconds, and 1.6 seconds.

Advantages

• Reduces pressure during prolonged failures.
• Allows dependencies time to recover.
• Works well with rate limiting and temporary unavailability.

Disadvantages

• Increases total operation latency.
• Identical clients may still retry simultaneously.
• Large retry counts can exceed the request deadline.

When to Use / Real-World Use Cases

Use exponential backoff for temporary network failures, service failovers, rate limits, and message reprocessing. Stop when the operation deadline or retry budget is exhausted.

Example

Attempt 1 ── fails
             wait 200 ms

Attempt 2 ── fails
             wait 400 ms

Attempt 3 ── fails
             wait 800 ms

Attempt 4 ── final failure

Jitter

Jitter adds randomness to retry delays. Without it, thousands of clients that fail at the same time may retry at the same intervals.

Without jitter:

Client A: retry at 1s, 2s, 4s
Client B: retry at 1s, 2s, 4s
Client C: retry at 1s, 2s, 4s

With jitter:

Client A: retry at 0.7s, 2.4s, 3.6s
Client B: retry at 1.2s, 1.8s, 4.5s
Client C: retry at 0.9s, 2.7s, 3.3s

Advantages

• Spreads retries over time.
• Reduces synchronized traffic spikes.
• Improves recovery after shared outages.

Disadvantages

• Makes exact retry timing less predictable.
• Requires a bounded strategy to avoid excessive delay.

When to Use / Real-World Use Cases

Use jitter whenever many clients can experience the same failure, including service outages, deployment restarts, cache expiration, and rate-limit recovery.

Example

import random


def full_jitter_delay(
    attempt: int,
    base_seconds: float = 0.2,
    maximum_seconds: float = 5.0,
) -> float:
    """
    Select a random delay between zero and the exponential cap.
    """
    cap = min(base_seconds * (2 ** attempt), maximum_seconds)
    return random.uniform(0, cap)

Circuit Breakers

A circuit breaker stops sending requests to a dependency after failures exceed a threshold. It protects the caller from repeatedly consuming resources on an unhealthy dependency.

                 failure threshold reached
CLOSED ─────────────────────────────────────► OPEN
  ▲                                             │
  │                                             │ wait recovery interval
  │                                             ▼
  └──────── successful test request ───── HALF-OPEN
                               failure ────────► OPEN

Closed allows normal requests. Open rejects requests immediately. Half-open allows a small number of test requests to determine whether recovery has occurred.

Advantages

• Fails quickly while a dependency is unhealthy.
• Reduces connection and thread consumption.
• Gives the dependency time to recover.
• Can trigger fallbacks immediately.

Disadvantages

• Poor thresholds can open during normal latency variation.
• Local breakers may disagree across application instances.
• An open breaker intentionally rejects requests that might succeed.

When to Use / Real-World Use Cases

Use circuit breakers around remote dependencies where repeated failures consume meaningful resources. Typical examples include payment gateways, recommendation services, external APIs, and overloaded databases.

Example

A product API opens the recommendation-service circuit after ten failures within thirty seconds. Product requests continue without recommendations until periodic test requests succeed.

Bulkheads

Bulkheads isolate resources so one failing dependency or workload cannot exhaust the entire application.

Application
├── Payment pool:        20 connections
├── Inventory pool:     30 connections
├── Recommendation pool: 5 connections
└── Internal database:  40 connections

If the recommendation service becomes slow, only its five connections remain occupied. Payment and inventory operations continue using separate capacity.

Advantages

• Limits blast radius.
• Protects critical workloads from optional workloads.
• Makes capacity allocation explicit.

Disadvantages

• Reserved capacity may remain unused.
• Requires separate pools, queues, or worker groups.
• Incorrect limits can underutilize or starve workloads.

When to Use / Real-World Use Cases

Use bulkheads when one process handles workloads with different criticality or dependencies with different failure characteristics.

Example

Background exports run in a separate worker queue from checkout operations. A large export backlog cannot consume the workers needed to place orders.

Load Shedding

Load shedding rejects low-priority or excessive work before the system becomes fully saturated. Controlled rejection preserves capacity for requests that can still complete successfully.

Advantages

• Prevents queue growth and cascading latency.
• Protects critical traffic.
• Allows faster recovery after overload.

Disadvantages

• Some valid requests are intentionally rejected.
• Prioritization rules can be difficult to define.
• Clients must handle rejection correctly.

When to Use / Real-World Use Cases

Use load shedding when queues, concurrency, memory, or downstream capacity approach safe limits. Reject optional analytics, expensive searches, or low-priority batch work before checkout or authentication traffic.

Example

from fastapi import HTTPException


def reject_when_saturated(
    active_requests: int,
    maximum_requests: int,
) -> None:
    """
    Reject new work before resource exhaustion causes system-wide failure.
    """
    if active_requests >= maximum_requests:
        raise HTTPException(
            status_code=503,
            detail="Service is temporarily at capacity",
            headers={"Retry-After": "2"},
        )

Graceful Degradation

Graceful degradation returns reduced but valid functionality when an optional dependency fails.

Advantages

• Preserves core functionality.
• Reduces user-visible outages.
• Allows optional dependencies to fail independently.

Disadvantages

• Fallback data may be stale or incomplete.
• Degraded behavior can hide dependency failures.
• Fallback logic increases test complexity.

When to Use / Real-World Use Cases

Use graceful degradation when the response remains correct without optional enrichment. Examples include omitting recommendations, returning cached pricing with an expiration notice, or delaying analytics events.

Example

{
  "product_id": "p-4821",
  "name": "Mechanical Keyboard",
  "price": 129.99,
  "recommendations": [],
  "degraded_components": [
    "recommendation-service"
  ]
}

Failure Pattern Comparison

Resilience patterns address different stages of failure propagation and are commonly combined.

Failure Detection

Failure handling depends on detecting unhealthy behavior without incorrectly removing healthy instances. Detection should combine active checks, passive request data, and end-to-end monitoring.

Health Checks

Health endpoints expose whether a process is alive and whether it is ready to receive traffic.

Liveness answers whether the process should be restarted. Readiness answers whether the process should receive new requests.

Liveness checks should not fail because an optional dependency is unavailable. Restarting every application instance during a shared dependency outage can convert a partial failure into a complete outage.

Heartbeats

Heartbeats indicate that workers, consumers, replicas, or leaders remain active. Missing heartbeats trigger suspicion and eventual reassignment.

Heartbeat systems require an expiration policy and a recovery policy. A worker that resumes after expiration must not continue processing work already reassigned to another worker.

Production note: combine heartbeat expiration with ownership generations or fencing tokens when duplicate ownership would be unsafe.

Passive Monitoring

Passive monitoring derives health from real request outcomes. Error rate, latency, connection failures, queue depth, and timeout rates often reveal degradation earlier than a simple health endpoint.

A service can return HTTP 200 from its health endpoint while every business request times out on the database.

Useful passive indicators include:

• success rate by operation;
• p95, p99, and p99.9 latency;
• retry rate;
• circuit-breaker state;
• connection-pool wait time;
• queue age;
• dependency error rate.

Synthetic Monitoring

Synthetic checks execute controlled end-to-end requests from outside the service. They validate routing, authentication, application logic, dependencies, and response correctness.

Use synthetic transactions for critical flows such as login, checkout, order lookup, and message publication.

Warning: synthetic requests should use isolated test accounts and idempotent cleanup. Monitoring must not create real orders, charges, or customer notifications.

Designing for Failures

Resilience is easier when failure handling is built into data models, APIs, and deployment architecture rather than added after incidents.

Idempotent Operations

An idempotent operation can be repeated without producing additional side effects. Idempotency is essential when callers retry after uncertain outcomes.

A state-changing API can require an idempotency key:

POST /payments
Idempotency-Key: order-93482-payment-v1

The service stores the key with the completed result. Later requests with the same key return the original result instead of executing the operation again.

Stateless Services

Stateless service instances do not depend on local memory for durable workflow progress. Any healthy instance can handle the next request.

Session data, idempotency records, job state, and workflow checkpoints should be stored in durable systems when they must survive process replacement.

Trade-off: moving state to shared infrastructure improves failover but adds network calls and dependency complexity.

Fallback Strategies

A fallback provides an alternative result when the preferred dependency is unavailable.

Common strategies include:

• returning cached data;
• using a secondary provider;
• omitting optional enrichment;
• queueing work for later;
• returning a stable default;
• requiring manual review for uncertain outcomes.

Fallbacks must preserve correctness. Returning a cached product description may be acceptable, while returning stale inventory during checkout may oversell stock.

Redundancy

Redundancy runs multiple instances, replicas, zones, or providers so one failure does not remove the entire capability.

Redundancy must cover independent failure domains. Three replicas on one host do not protect against host failure.

Region
├── Availability Zone A
│   ├── API instance
│   └── Database replica
├── Availability Zone B
│   ├── API instance
│   └── Database replica
└── Availability Zone C
    ├── API instance
    └── Database replica

Redundancy also introduces coordination problems: replica lag, failover selection, split brain, duplicate consumers, and state synchronization.

Bounded Concurrency

Bounded concurrency limits the number of operations running against a dependency. It prevents one slow dependency from creating unlimited in-flight work.

import asyncio


payment_semaphore = asyncio.Semaphore(20)


async def call_payment_provider(
    request: dict,
) -> dict:
    """
    Limit concurrent payment requests so provider latency cannot consume
    every application connection and task.
    """
    async with payment_semaphore:
        return await send_payment_request(request)

Concurrency limits should be paired with queue limits or acquisition timeouts. Otherwise requests may wait indefinitely before entering the bounded section.

Asynchronous Recovery

Some work does not need to complete inside the original request. Durable queues allow APIs to acknowledge accepted work and complete it asynchronously.

Client
  │
  │── submit export ──► API
  │                     │
  │                     ├── store export request
  │                     └── publish durable job
  │
  │◄── 202 Accepted ────│

Worker
  │── process export
  │── retry transient failures
  └── store final result

Asynchronous recovery is useful for exports, notifications, media processing, synchronization, and other long-running operations.

Queue consumers still need idempotency, dead-letter handling, retry limits, and visibility into message age.

Production Implementation Example

The following example combines request deadlines, idempotency, bounded retries, jitter, a circuit breaker, and Kubernetes health checks for an order API calling an external payment provider.

Resilient Request Flow

The order flow stores durable intent before calling the provider. A stable idempotency key links every retry to the same payment operation.

Client
  │
  │── POST /orders + idempotency key
  ▼
Order API
  │
  ├── reserve idempotency record
  ├── create pending order
  ├── call payment provider
  │      ├── timeout
  │      ├── bounded retries
  │      ├── exponential backoff
  │      ├── jitter
  │      └── circuit breaker
  │
  ├── payment confirmed ──► mark order paid
  │
  └── payment uncertain ──► mark verification pending
                             publish reconciliation job

Unknown payment outcomes should not be converted directly into failure. A reconciliation process can query the provider using the stable external payment identifier.

FastAPI Client with Timeouts and Retries

The client retries only selected transient failures and stops when the overall deadline is exhausted.

from __future__ import annotations

import asyncio
import random
import time
from dataclasses import dataclass

import httpx


class PaymentUnavailableError(RuntimeError):
    pass


class PaymentRejectedError(RuntimeError):
    pass


@dataclass(frozen=True)
class PaymentResult:
    provider_payment_id: str
    status: str


class PaymentClient:
    def __init__(
        self,
        base_url: str,
        maximum_attempts: int = 3,
    ) -> None:
        self.maximum_attempts = maximum_attempts

        self.client = httpx.AsyncClient(
            base_url=base_url,
            timeout=httpx.Timeout(
                connect=0.5,
                read=1.5,
                write=0.5,
                pool=0.2,
            ),
            limits=httpx.Limits(
                # Bound resource usage during provider degradation.
                max_connections=30,
                max_keepalive_connections=15,
            ),
        )

    async def create_payment(
        self,
        order_id: str,
        amount_cents: int,
        idempotency_key: str,
        deadline_monotonic: float,
    ) -> PaymentResult:
        """
        Create one logical payment operation.

        The same idempotency key is sent on every attempt so a timeout cannot
        cause multiple provider charges.
        """
        last_error: Exception | None = None

        for attempt in range(self.maximum_attempts):
            remaining = deadline_monotonic - time.monotonic()

            if remaining <= 0:
                raise PaymentUnavailableError(
                    "Payment deadline was exhausted"
                ) from last_error

            try:
                response = await self.client.post(
                    "/payments",
                    json={
                        "order_id": order_id,
                        "amount_cents": amount_cents,
                    },
                    headers={
                        "Idempotency-Key": idempotency_key,
                    },
                    # Never allow one attempt to exceed the remaining deadline.
                    timeout=min(remaining, 1.5),
                )
            except (
                httpx.ConnectError,
                httpx.ConnectTimeout,
                httpx.ReadTimeout,
                httpx.RemoteProtocolError,
            ) as error:
                last_error = error

                if attempt == self.maximum_attempts - 1:
                    break

                await asyncio.sleep(
                    self._retry_delay(attempt)
                )
                continue

            if response.status_code in {200, 201}:
                payload = response.json()

                return PaymentResult(
                    provider_payment_id=payload["payment_id"],
                    status=payload["status"],
                )

            if response.status_code == 429:
                last_error = PaymentUnavailableError(
                    "Payment provider rate limited the request"
                )

                if attempt == self.maximum_attempts - 1:
                    break

                retry_after = self._parse_retry_after(
                    response.headers.get("Retry-After")
                )

                await asyncio.sleep(
                    min(retry_after, max(0, remaining))
                )
                continue

            if response.status_code in {502, 503, 504}:
                last_error = PaymentUnavailableError(
                    f"Transient provider response: {response.status_code}"
                )

                if attempt == self.maximum_attempts - 1:
                    break

                await asyncio.sleep(
                    self._retry_delay(attempt)
                )
                continue

            if 400 <= response.status_code < 500:
                # Validation and business rejections should not be retried.
                raise PaymentRejectedError(response.text)

            last_error = PaymentUnavailableError(
                f"Unexpected provider response: {response.status_code}"
            )
            break

        raise PaymentUnavailableError(
            "Payment provider did not return a confirmed result"
        ) from last_error

    @staticmethod
    def _retry_delay(attempt: int) -> float:
        """
        Full jitter prevents application replicas from retrying together.
        """
        exponential_cap = min(0.2 * (2 ** attempt), 2.0)
        return random.uniform(0, exponential_cap)

    @staticmethod
    def _parse_retry_after(value: str | None) -> float:
        if value is None:
            return 1.0

        try:
            return max(float(value), 0.0)
        except ValueError:
            return 1.0

The client does not retry all errors. Business rejections and invalid requests return immediately because repeating them would not change the outcome.

Idempotent Order Creation

The API reserves the idempotency key before performing external side effects.

CREATE TABLE idempotency_records (
    idempotency_key VARCHAR(200) PRIMARY KEY,
    request_hash VARCHAR(64) NOT NULL,
    status VARCHAR(30) NOT NULL,
    response_status INTEGER,
    response_body JSONB,
    created_at TIMESTAMP NOT NULL,
    updated_at TIMESTAMP NOT NULL
);

CREATE TABLE orders (
    id UUID PRIMARY KEY,
    idempotency_key VARCHAR(200) NOT NULL UNIQUE,
    status VARCHAR(30) NOT NULL,
    amount_cents INTEGER NOT NULL,
    provider_payment_id VARCHAR(200),
    created_at TIMESTAMP NOT NULL,
    updated_at TIMESTAMP NOT NULL
);

The request hash prevents reuse of the same idempotency key with different request data.

from __future__ import annotations

import hashlib
import json
import time
from uuid import uuid4

from fastapi import FastAPI, Header, HTTPException
from pydantic import BaseModel, Field


app = FastAPI()


class CreateOrderRequest(BaseModel):
    amount_cents: int = Field(gt=0)


class OrderResponse(BaseModel):
    order_id: str
    status: str


@app.post("/orders", response_model=OrderResponse)
async def create_order(
    request: CreateOrderRequest,
    idempotency_key: str = Header(alias="Idempotency-Key"),
) -> OrderResponse:
    """
    Create an order with durable idempotency protection.
    """
    request_hash = hashlib.sha256(
        json.dumps(
            request.model_dump(),
            sort_keys=True,
        ).encode()
    ).hexdigest()

    existing = await repository.get_idempotency_record(
        idempotency_key
    )

    if existing is not None:
        if existing.request_hash != request_hash:
            raise HTTPException(
                status_code=409,
                detail="Idempotency key was used with different request data",
            )

        if existing.status == "completed":
            # Return the exact previously stored response.
            return OrderResponse.model_validate(
                existing.response_body
            )

        if existing.status == "processing":
            raise HTTPException(
                status_code=409,
                detail="The operation is already being processed",
                headers={"Retry-After": "1"},
            )

    order_id = str(uuid4())

    acquired = await repository.reserve_idempotency_key(
        idempotency_key=idempotency_key,
        request_hash=request_hash,
        order_id=order_id,
        amount_cents=request.amount_cents,
    )

    if not acquired:
        # Another request acquired the same key concurrently.
        raise HTTPException(
            status_code=409,
            detail="The operation is already being processed",
            headers={"Retry-After": "1"},
        )

    try:
        payment = await payment_client.create_payment(
            order_id=order_id,
            amount_cents=request.amount_cents,
            idempotency_key=idempotency_key,
            deadline_monotonic=time.monotonic() + 2.0,
        )
    except PaymentRejectedError:
        response = OrderResponse(
            order_id=order_id,
            status="payment_rejected",
        )

        await repository.complete_order(
            order_id=order_id,
            order_status="payment_rejected",
            idempotency_key=idempotency_key,
            response_body=response.model_dump(),
            response_status=200,
        )

        return response
    except PaymentUnavailableError:
        # The payment outcome may be unknown rather than failed.
        await repository.mark_payment_verification_pending(
            order_id=order_id,
            idempotency_key=idempotency_key,
        )

        await reconciliation_queue.publish(
            {
                "order_id": order_id,
                "idempotency_key": idempotency_key,
            }
        )

        return OrderResponse(
            order_id=order_id,
            status="payment_verification_pending",
        )

    response = OrderResponse(
        order_id=order_id,
        status="paid",
    )

    await repository.complete_order(
        order_id=order_id,
        order_status="paid",
        provider_payment_id=payment.provider_payment_id,
        idempotency_key=idempotency_key,
        response_body=response.model_dump(),
        response_status=200,
    )

    return response

The uncertain outcome is represented explicitly as payment_verification_pending. This prevents the API from incorrectly telling the client to submit a new payment.

Circuit Breaker Implementation

The circuit breaker tracks recent failures and temporarily rejects calls after the failure threshold is reached.

from __future__ import annotations

import asyncio
import time
from enum import Enum
from typing import Awaitable, Callable, TypeVar


T = TypeVar("T")


class CircuitState(str, Enum):
    CLOSED = "closed"
    OPEN = "open"
    HALF_OPEN = "half_open"


class CircuitOpenError(RuntimeError):
    pass


class AsyncCircuitBreaker:
    def __init__(
        self,
        failure_threshold: int = 5,
        recovery_timeout_seconds: float = 10.0,
    ) -> None:
        self.failure_threshold = failure_threshold
        self.recovery_timeout_seconds = recovery_timeout_seconds
        self.state = CircuitState.CLOSED
        self.failure_count = 0
        self.opened_at: float | None = None
        self._half_open_request_in_progress = False
        self._lock = asyncio.Lock()

    async def call(
        self,
        operation: Callable[[], Awaitable[T]],
    ) -> T:
        """
        Execute the operation only when the circuit permits it.

        Only one test request enters while half-open, preventing a recovery
        probe from becoming another traffic spike.
        """
        async with self._lock:
            self._update_state()

            if self.state == CircuitState.OPEN:
                raise CircuitOpenError(
                    "Dependency circuit is open"
                )

            if self.state == CircuitState.HALF_OPEN:
                if self._half_open_request_in_progress:
                    raise CircuitOpenError(
                        "Dependency recovery probe is already running"
                    )

                self._half_open_request_in_progress = True

        try:
            result = await operation()
        except Exception:
            await self._record_failure()
            raise
        else:
            await self._record_success()
            return result

    def _update_state(self) -> None:
        if (
            self.state == CircuitState.OPEN
            and self.opened_at is not None
            and time.monotonic() - self.opened_at
            >= self.recovery_timeout_seconds
        ):
            self.state = CircuitState.HALF_OPEN
            self._half_open_request_in_progress = False

    async def _record_failure(self) -> None:
        async with self._lock:
            self.failure_count += 1
            self._half_open_request_in_progress = False

            if (
                self.state == CircuitState.HALF_OPEN
                or self.failure_count >= self.failure_threshold
            ):
                self.state = CircuitState.OPEN
                self.opened_at = time.monotonic()

    async def _record_success(self) -> None:
        async with self._lock:
            self.state = CircuitState.CLOSED
            self.failure_count = 0
            self.opened_at = None
            self._half_open_request_in_progress = False

A production breaker should commonly use a rolling failure window rather than one lifetime counter. It may also distinguish slow-call thresholds from explicit failures.

Kubernetes Health Configuration

Kubernetes probes should distinguish startup, liveness, and readiness.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: order-api
spec:
  replicas: 3
  selector:
    matchLabels:
      app: order-api
  template:
    metadata:
      labels:
        app: order-api
    spec:
      containers:
        - name: order-api
          image: example/order-api:1.8.0
          ports:
            - containerPort: 8000

          # Startup probe prevents slow initialization from triggering
          # premature liveness failures.
          startupProbe:
            httpGet:
              path: /health/startup
              port: 8000
            periodSeconds: 2
            failureThreshold: 30

          # Liveness checks only whether the process is irrecoverably stuck.
          # It should not fail because an optional remote service is down.
          livenessProbe:
            httpGet:
              path: /health/live
              port: 8000
            periodSeconds: 10
            timeoutSeconds: 2
            failureThreshold: 3

          # Readiness removes the instance from service when it cannot handle
          # normal requests, without restarting it.
          readinessProbe:
            httpGet:
              path: /health/ready
              port: 8000
            periodSeconds: 5
            timeoutSeconds: 2
            failureThreshold: 2

          resources:
            requests:
              cpu: "250m"
              memory: "256Mi"
            limits:
              cpu: "1000m"
              memory: "512Mi"

The readiness endpoint may check critical local state and connection-pool availability. It should avoid expensive dependency calls on every probe.

Failure Scenarios in Production

Resilience patterns become easier to evaluate when applied to complete failure scenarios.

Payment Provider Timeout

A payment request times out after the provider receives it. The provider may have charged the card even though the API did not receive a response.

Unsafe response: mark the payment failed and ask the client to retry with a new operation identifier.

Safer response:

1. Use one stable idempotency key for every attempt.
2. Record a pending payment before the provider call.
3. Retry only within a bounded deadline.
4. Mark the result as uncertain if no confirmation arrives.
5. Query the provider asynchronously using the same payment reference.
6. Complete or reverse the order after reconciliation.

This approach treats uncertainty as a valid workflow state rather than incorrectly converting it into failure.

Database Connection Pool Exhaustion

Slow queries hold connections longer, causing new requests to wait for the pool. Application latency rises, clients retry, and additional requests consume more memory.

Slow queries
    ↓
Connections remain busy
    ↓
Pool wait time increases
    ↓
Request latency increases
    ↓
Clients retry
    ↓
More concurrent requests
    ↓
Application saturation

Recovery strategy:

• set a short pool acquisition timeout;
• limit request concurrency;
• cancel queries that exceed execution budgets;
• reject low-priority traffic;
• identify and optimize the slow query;
• avoid increasing pool size beyond database capacity.

A larger connection pool can worsen the incident by moving saturation from the application to the database.

Message Consumer Failure

A consumer processes a message, writes to a database, and crashes before acknowledging the message. The broker later delivers it again.

Required protections:

• unique message or operation identifiers;
• idempotent state transitions;
• transactional processing where possible;
• bounded retries;
• dead-letter handling for persistent failures;
• monitoring of oldest-message age.

CREATE TABLE processed_messages (
    consumer_name VARCHAR(100) NOT NULL,
    message_id VARCHAR(200) NOT NULL,
    processed_at TIMESTAMP NOT NULL,
    PRIMARY KEY (consumer_name, message_id)
);

BEGIN;

-- Duplicate delivery produces no new record.
INSERT INTO processed_messages (
    consumer_name,
    message_id,
    processed_at
)
VALUES (
    :consumer_name,
    :message_id,
    CURRENT_TIMESTAMP
)
ON CONFLICT DO NOTHING;

-- Continue only when the message was inserted for the first time.
-- Application code should verify the affected row count before applying
-- the business update in this same transaction.

COMMIT;

Regional Dependency Outage

A service in one region becomes unavailable while the application remains healthy. Immediate failover to another region may be possible, but cross-region consistency and capacity must be considered.

Secondary regions must be tested under real traffic assumptions. A standby with only 20 percent of required capacity does not provide reliable failover.

Observability for Partial Failures

Partial failures can remain hidden because aggregate availability may look healthy while one dependency, tenant, region, or operation is failing.

Failure Metrics

Metrics should separate original requests from retries and expose both dependency and application behavior.

Retry success should be measured. A retry policy that adds 30 percent more traffic but recovers only 0.1 percent of requests is likely harmful.

Distributed Tracing

Distributed tracing connects operations across service boundaries. A trace should record every dependency call, retry attempt, timeout, circuit decision, and fallback.

POST /orders                         1,840 ms
├── INSERT pending order                18 ms
├── POST payment attempt 1             520 ms timeout
├── retry backoff                      173 ms
├── POST payment attempt 2             604 ms timeout
├── publish reconciliation job          21 ms
└── return verification pending         12 ms

Without retry spans, one slow logical request may appear as a single unexplained latency spike.

Structured Logging

Logs should contain identifiers that connect retries and recovery operations.

{
  "event": "payment_attempt_failed",
  "order_id": "8f7b1c1c-73be-4aa5-908f-c2031d7c7b99",
  "idempotency_key": "order-93482-payment-v1",
  "attempt": 2,
  "failure_type": "read_timeout",
  "elapsed_ms": 1504,
  "remaining_deadline_ms": 287,
  "circuit_state": "closed",
  "provider": "primary-payment-provider"
}

Do not log sensitive payment data, access tokens, passwords, or complete personal information.

Alerting on Symptoms

Alerts should focus on user impact and exhaustion signals rather than every individual exception.

Useful alerts include:

• Error-budget burn: success rate is consuming the allowed reliability budget too quickly.
• Tail-latency increase: p99 latency exceeds the service objective.
• Retry amplification: total dependency calls significantly exceed original requests.
• Pool saturation: connection acquisition time or utilization approaches limits.
• Queue delay: oldest-message age exceeds the processing objective.
• Circuit open duration: a dependency remains isolated beyond the normal recovery window.
• Degraded response rate: fallback behavior exceeds an expected threshold.

One dependency error may be harmless. A rising timeout rate combined with pool saturation and retry growth indicates a cascading failure.

Common Mistakes

Production Checklist

• Define request deadlines: assign a maximum total duration to every synchronous operation and propagate the remaining budget downstream.
• Configure phase-specific timeouts: set explicit connection, read, write, and pool acquisition limits for remote clients.
• Classify retryable failures: document which status codes, exceptions, and business outcomes may be retried.
• Limit retry attempts: enforce a small maximum attempt count that fits inside the operation deadline.
• Add exponential backoff: increase delays between attempts so unhealthy dependencies receive less pressure.
• Add jitter: randomize retry timing to prevent synchronized clients from generating recovery spikes.
• Require idempotency keys: protect non-idempotent APIs, message handlers, and external side effects from duplicate execution.
• Store idempotency results durably: preserve the original status and response so retries return the same outcome.
• Represent uncertain outcomes: use states such as pending verification instead of assuming a timeout means failure.
• Use circuit breakers: stop repeated calls to dependencies that are consistently failing or timing out.
• Isolate dependency resources: use separate connection pools, concurrency limits, or worker queues for unrelated workloads.
• Bound every queue: define maximum queue size, maximum message age, and overload behavior.
• Implement load shedding: reject low-priority work before critical resources become fully exhausted.
• Define fallback safety: document which data may be stale, omitted, delayed, or replaced by defaults.
• Separate liveness and readiness: restart only irrecoverably unhealthy processes and remove temporarily impaired instances from traffic.
• Make consumers idempotent: assume every message can be delivered more than once.
• Configure dead-letter handling: isolate persistent message failures after a bounded retry count.
• Measure retry amplification: compare original request volume with total downstream attempts.
• Monitor saturation: track connection-pool utilization, worker concurrency, queue age, memory, CPU, and file descriptors.
• Test realistic failures: simulate timeouts, packet loss, slow dependencies, process pauses, database failover, and regional outages.

Conclusion

Partial failures are unavoidable in distributed systems because components communicate over unreliable networks and fail independently. A service can remain healthy while one dependency is slow, one replica is stale, or one request has an unknown outcome.

Reliable systems combine timeouts, bounded retries, exponential backoff, jitter, circuit breakers, bulkheads, load shedding, graceful degradation, and idempotency. Each pattern solves a different part of the failure path.

The strongest designs do not merely retry until success. They control resource usage, preserve correctness during uncertainty, isolate failures, expose degraded behavior, and provide an asynchronous recovery path when a synchronous result cannot be confirmed.

Key Takeaway

A partial failure is not only an error; it is often an unknown outcome combined with limited resources. Production resilience requires bounding the time and capacity spent on failures while ensuring that retries, failover, and recovery cannot duplicate or corrupt business operations.

More Articles to Read

• Consistency Models in Distributed Systems
• CAP Theorem: Practical Trade-Offs and Real-World Examples
• Designing Distributed Transactions with Sagas and Two-Phase Commit
• Idempotency and Deduplication in Distributed Systems
• Leader Election and Distributed Coordination