What Is a Circuit Breaker?

By Oleksandr Andrushchenko — Published on Sep 18

Tags: circuit-breaker fault-tolerance distributed-systems

0 Likes

0 Dislikes

A circuit breaker is a reliability pattern that temporarily stops calls to a failing dependency instead of repeatedly sending requests that are likely to fail.

It protects both sides of a distributed system. The caller avoids wasting threads, connections, CPU, and request time on an unhealthy dependency, while the failing service gets time to recover without being continuously flooded with traffic.

Why Circuit Breakers Exist
How a Circuit Breaker Works
What Should Trip a Circuit Breaker?
Failure Thresholds and Sliding Windows
Circuit Breakers and Timeouts
Circuit Breakers and Retries
What Happens When the Circuit Is Open?
Circuit Breaker Scope
Production Design Example
Monitoring Circuit Breakers
Common Circuit Breaker Mistakes
When a Circuit Breaker Is Useful
Conclusion
- More Articles to Read

Why Circuit Breakers Exist

Consider an API that depends on a payment service:

Client → Order API → Payment Service

Normally, the payment service responds in 100 ms. During an outage, requests begin timing out after five seconds.

If the Order API continues sending every request to the unhealthy service, each request may hold resources for the entire timeout period:

Request 1 → wait 5s → timeout
Request 2 → wait 5s → timeout
Request 3 → wait 5s → timeout
Request 4 → wait 5s → timeout

As traffic increases, the Order API accumulates waiting requests. Worker threads, connection pools, memory, and downstream connections can become exhausted.

A failure that originally affected only the payment service can now spread upstream:

Payment Service fails
        ↓
Order API requests wait
        ↓
Connection pool fills
        ↓
Order API becomes slow
        ↓
Clients retry
        ↓
Traffic increases
        ↓
More services begin failing

This is a cascading failure.

A circuit breaker interrupts the chain. Once the dependency is considered unhealthy, calls fail immediately instead of waiting for the remote operation to fail again.

Order API → Circuit Breaker → Payment Service
                 |
                 +→ OPEN: fail fast

The dependency receives less traffic, and the caller preserves resources for operations that can still succeed.

How a Circuit Breaker Works

A circuit breaker typically behaves as a small state machine with three states: closed, open, and half-open.

CLOSED → OPEN → HALF-OPEN → CLOSED

The state changes according to observed failures and recovery attempts.

Closed State

Closed is the normal operating state. Requests are allowed to reach the dependency.

Request → Circuit Breaker → Dependency

The circuit breaker records outcomes such as successful responses, timeouts, connection failures, and selected server errors.

If failures remain below the configured threshold, the circuit stays closed.

If failures exceed that threshold, the circuit opens.

Open State

When the circuit is open, normal requests do not reach the unhealthy dependency.

Request → Circuit Breaker → Fast Failure
                            X Dependency

Instead of spending several seconds waiting for another timeout, the caller might receive an error immediately.

Suppose a dependency normally has a five-second timeout. During a complete outage, 1,000 incoming requests could otherwise create 1,000 slow remote calls.

With an open circuit, those calls can fail locally in milliseconds.

The circuit normally remains open for a configured recovery period such as 10, 30, or 60 seconds.

Half-Open State

A dependency should not remain blocked forever. After the open interval expires, the circuit breaker needs to determine whether the service has recovered.

It enters the half-open state and permits a limited number of probe requests.

HALF-OPEN
    |
    +→ Probe request → Success → CLOSED
    |
    +→ Probe request → Failure → OPEN

If the probes succeed, normal traffic gradually resumes.

If they fail, the circuit opens again and waits before another recovery attempt.

Limiting probe traffic is important. Sending the entire production workload immediately after the timeout can overwhelm a service that has only partially recovered.

What Should Trip a Circuit Breaker?

Not every unsuccessful response means the dependency is unhealthy.

Consider these HTTP responses:

400 Bad Request
401 Unauthorized
404 Not Found
429 Too Many Requests
500 Internal Server Error
503 Service Unavailable

A 400 response usually indicates an invalid caller request. Opening the circuit because clients send invalid payloads would not protect anything.

A 404 may also be a valid business response.

By contrast, repeated connection failures, timeouts, 500 responses, and 503 responses can indicate dependency failure.

Failure	Usually Count Toward Circuit?
Connection refused	Yes
Connection timeout	Yes
Request timeout	Yes
HTTP 500	Usually
HTTP 503	Usually
HTTP 429	Depends on the design
HTTP 404	Usually no
Validation error	No

The exact policy depends on the dependency's contract. A circuit breaker should measure service health failures, not ordinary business outcomes.

Failure Thresholds and Sliding Windows

Opening a circuit after one failure is usually too aggressive. Distributed systems experience occasional timeouts and connection errors even when they are healthy.

A circuit breaker therefore evaluates failures over a window.

One policy might be:

Window size:       100 requests
Minimum requests:   20
Failure threshold:  50%

Suppose the last 100 requests contain:

72 successes
28 failures

Failure rate = 28%

The circuit remains closed.

Later:

42 successes
58 failures

Failure rate = 58%

The failure threshold has been crossed, so the circuit opens.

The minimum-request requirement prevents small samples from triggering the breaker. Without it, two failures immediately after startup could produce a 100% failure rate and open the circuit unnecessarily.

Another design uses consecutive failures:

Open after 5 consecutive failures

This is simpler but reacts differently to intermittent problems. A percentage-based sliding window usually provides more control for high-volume services.

Some implementations also track slow calls separately. A dependency returning successful responses in 15 seconds may be operationally unhealthy even if its HTTP status codes are all 200.

Circuit Breakers and Timeouts

A circuit breaker does not replace timeouts.

The timeout determines how long one remote call may wait. The circuit breaker determines whether the call should be attempted at all.

Without a timeout:

Request → Dependency → waits indefinitely

The circuit breaker may not receive a failure signal quickly enough to react.

With a timeout:

Request → Dependency → 2s timeout → recorded failure

After enough failures:

Future request → OPEN circuit → immediate failure

The patterns therefore work together.

A timeout should normally be configured according to the caller's latency budget rather than relying on a library's large default timeout.

If an API must respond within one second, a five-second downstream timeout is already incompatible with that requirement.

Timeouts, Retries, and Exponential Backoff covers timeout selection and retry timing in more detail.

Circuit Breakers and Retries

Retries can improve reliability when failures are transient, but they can also amplify an outage.

Suppose 10,000 requests arrive and each request allows three retries:

10,000 original requests
+ 30,000 possible retries
= 40,000 attempts

If the dependency is already overloaded, retrying aggressively can make recovery harder.

A circuit breaker can stop retries from continuously reaching a dependency once failure becomes widespread.

Request
   ↓
Retry Policy
   ↓
Circuit Breaker
   ↓
Dependency

The exact composition depends on the client library and desired semantics, but the retry policy should respect circuit state rather than repeatedly bypassing it.

Retries should also use backoff and usually jitter. Immediate retry loops can generate synchronized traffic spikes.

The general rule is:

timeouts bound individual attempts, retries handle occasional transient failures, and circuit breakers stop repeated attempts when the dependency appears broadly unhealthy.

What Happens When the Circuit Is Open?

Failing fast is only the first part of the design. The application must decide what to do with the request after the circuit rejects the dependency call.

Possible strategies include:

return an explicit service-unavailable response;
serve cached or stale data;
use a degraded response;
queue work for asynchronous processing;
route to an alternative provider;
disable the affected feature temporarily.

Consider a product page that calls a recommendation service. If recommendations fail, the entire product page probably should not return HTTP 500.

Product data       → available
Inventory          → available
Recommendations    → unavailable

The response can omit recommendations while preserving the core product experience.

Payment processing is different. Silently pretending that a payment succeeded would be incorrect. The system might instead reject checkout temporarily or queue the operation only if the business workflow safely supports delayed payment.

The fallback therefore depends on the importance and semantics of the dependency.

Designing Graceful Degradation Strategies explains how systems preserve partial functionality when dependencies fail.

Circuit Breaker Scope

A circuit breaker needs an appropriate scope. One global circuit for every remote operation is often too broad.

Suppose an application calls three services:

Application
   ├→ Payment Service
   ├→ Inventory Service
   └→ Recommendation Service

If the recommendation service fails, opening one application-wide circuit should not block payment and inventory calls.

Each independent dependency generally needs separate health state.

Sometimes even one dependency requires multiple breakers. If a service exposes unrelated operations with different performance characteristics, failures in one endpoint may not mean every endpoint is unhealthy.

Deployment topology matters too.

An in-process circuit breaker normally has local state:

Instance A → breaker state A
Instance B → breaker state B
Instance C → breaker state C

Each application instance independently observes failures and opens its own circuit.

This is usually acceptable and avoids distributed coordination. Breaker state does not normally need strong global consistency.

However, traffic distribution can cause instances to observe different failure rates. That behavior should be understood when interpreting breaker metrics.

Production Design Example

Consider an Order API that synchronously calls an Inventory Service before confirming an order.

Client → Order API → Inventory Service

The normal inventory latency is around 80 ms. The Order API has a 500 ms downstream latency budget.

A reasonable starting policy might be:

Request timeout:      400 ms
Sliding window:        50 calls
Minimum calls:         20
Failure threshold:     50%
Open duration:         30 seconds
Half-open probes:       5

The application records timeouts, connection failures, and selected 5xx responses as circuit-breaker failures.

A simplified Python implementation illustrates the behavior:

from enum import Enum
import time


class CircuitState(Enum):
    CLOSED = "closed"
    OPEN = "open"
    HALF_OPEN = "half_open"


class CircuitOpenError(Exception):
    pass


class CircuitBreaker:
    def __init__(self, failure_limit=5, recovery_timeout=30):
        self.failure_limit = failure_limit
        self.recovery_timeout = recovery_timeout
        self.failures = 0
        self.state = CircuitState.CLOSED
        self.opened_at = None

    def call(self, operation):
        if self.state == CircuitState.OPEN:
            if time.monotonic() - self.opened_at < self.recovery_timeout:
                raise CircuitOpenError()

            self.state = CircuitState.HALF_OPEN

        try:
            result = operation()
        except Exception:
            self.record_failure()
            raise

        self.record_success()
        return result

    def record_failure(self):
        self.failures += 1

        if (
            self.state == CircuitState.HALF_OPEN
            or self.failures >= self.failure_limit
        ):
            self.state = CircuitState.OPEN
            self.opened_at = time.monotonic()

    def record_success(self):
        self.failures = 0
        self.state = CircuitState.CLOSED

This implementation is intentionally small. A production library would normally support sliding windows, concurrency safety, failure classification, metrics, slow-call thresholds, and controlled half-open probes.

The application can then define behavior for an open circuit:

try:
    inventory = inventory_breaker.call(
        lambda: inventory_client.reserve(order)
    )
except CircuitOpenError:
    return {
        "status": "temporarily_unavailable",
        "reason": "inventory_service_unavailable",
    }

Suppose the Inventory Service begins timing out.

The first requests encounter the configured timeout and contribute to the failure rate. Once the threshold is crossed, the breaker opens. Subsequent requests fail locally without creating additional inventory calls.

Thirty seconds later, a small number of half-open probes are allowed through. If inventory has recovered, the circuit closes. If the probes still fail, the circuit opens again.

The resulting behavior is significantly different from endlessly forwarding production traffic to a dependency that is already known to be unhealthy.

Monitoring Circuit Breakers

A circuit breaker changes application behavior during failures, so its state should be observable.

Useful metrics include:

Metric	Why It Matters
Circuit state	Shows whether a dependency is currently blocked
State transitions	Reveals repeated opening and recovery cycles
Failure rate	Shows why the breaker may be opening
Rejected calls	Measures traffic prevented from reaching the dependency
Half-open probe results	Shows whether recovery attempts succeed
Dependency latency	Detects degradation before complete failure

Logs should include the dependency, breaker name, previous state, new state, and reason for the transition.

circuit=inventory
state_from=closed
state_to=open
failure_rate=0.64
window_requests=50

An occasional circuit opening may be normal during dependency incidents. Frequent transitions between open and closed can indicate an unstable dependency or poorly tuned thresholds.

Observability should therefore focus not only on whether a breaker is open but also why it opened and how frequently it changes state.

Common Circuit Breaker Mistakes

Circuit breakers can reduce cascading failures, but incorrect configuration can create new availability problems.

No timeout. Calls can remain blocked too long before the breaker receives failure information.
Opening after one failure. A single transient error can unnecessarily disable a healthy dependency.
Counting business errors as service failures. Validation failures or expected 404 responses should not normally indicate infrastructure failure.
A threshold that is too slow. Thousands of failing calls may reach the dependency before the breaker reacts.
A threshold that is too sensitive. Normal error variation can repeatedly open the circuit.
Unlimited half-open traffic. Full traffic can overwhelm a recovering dependency immediately.
A global breaker for unrelated dependencies. One failing service can unnecessarily disable healthy integrations.
A fallback that hides correctness failures. Returning stale or fabricated results may be unacceptable for payments, authorization, or inventory operations.
No metrics. Operators cannot distinguish dependency failures from requests rejected by an already-open circuit.

A circuit breaker should be tuned from actual traffic, latency, and failure characteristics rather than copied as a fixed configuration across every dependency.

When a Circuit Breaker Is Useful

Circuit breakers are most useful around remote operations where repeated failure consumes meaningful resources or contributes to cascading failure.

Typical examples include:

HTTP calls between microservices;
third-party APIs;
database connections;
remote caches;
payment providers;
search clusters;
external identity services.

Not every function call needs a circuit breaker. Local deterministic operations generally do not benefit from one.

A breaker is also only one isolation mechanism. Bulkheads can limit how many resources one dependency is allowed to consume, while load shedding rejects excess work when the system itself approaches saturation.

Circuit Breaker vs Bulkhead vs Load Shedding compares these patterns and the different failure modes they address.

Conclusion

A circuit breaker protects distributed systems by temporarily stopping calls to a dependency that appears unhealthy. Instead of repeatedly waiting for predictable failures, the caller fails fast, preserves resources, and gives the dependency time to recover.

Production circuit breakers need sensible failure classification, minimum sample sizes, thresholds, timeouts, controlled half-open probes, fallback behavior, and observability. They also need to work coherently with retries rather than allowing retries to amplify an outage.

The core principle is simple: once a dependency is known to be failing, continuing to send it the same production traffic is often worse than temporarily refusing to call it at all.