Reliability in Software
Reliability is the ability of a software system to keep doing the correct thing over time, even when parts of the system fail.
A reliable system does not need to be perfect. Servers crash. Networks become slow. Databases restart. Third-party APIs return errors. Deployments introduce bugs. Traffic suddenly increases.
The real question is:
What happens when something goes wrong?
If one failed dependency causes the entire application to stop working, the system is fragile.
If the system detects the problem, limits the damage, recovers automatically, and continues serving most users, it is reliable.
This article explains reliability using simple examples and practical patterns commonly used in production systems.
Table of Contents
- What Is Reliability?
- Failures Are Normal
- Redundancy
- Timeouts
- Retries
- Circuit Breakers
- Bulkheads
- Graceful Degradation
- Queues and Failure Isolation
- Health Checks
- Reliable Data Storage
- Observability and Alerting
- Example: Making an Online Store Reliable
- Common Reliability Mistakes
- Production Checklist
- Conclusion
What Is Reliability?
Imagine a very small application:
User │ ▼ Application │ ▼ Database
When everything is healthy, the system works.
But suppose the database becomes unavailable for 30 seconds.
A fragile application may behave like this:
Database unavailable
↓
Every request fails
↓
Users retry manually
↓
Traffic increases
↓
Application becomes overloaded
A more reliable system may behave differently:
Database temporarily unavailable
↓
Requests fail fast
↓
Retries are controlled
↓
Some cached data remains available
↓
Database recovers
↓
System returns to normal
The failure still happened. Reliability comes from how the system reacted to it.
A useful way to think about reliability is:
Reliable system = fewer failures + smaller failure impact + faster recovery
Reliability vs Availability
Availability usually describes whether the system can serve requests at a particular moment.
Reliability is broader. It includes whether the system behaves correctly and consistently over time.
For example, imagine a payment API that is always reachable but occasionally charges customers twice.
Availability: high Reliability: poor
Now imagine another service that experiences a short outage once a year but otherwise processes every request correctly.
Its reliability may still be much better.
Availability is therefore an important part of reliability, but reliability also includes correctness, recovery, durability, and predictable behavior during failures.
Reliability vs Performance
A system can be fast but unreliable.
Consider two APIs:
API A Latency: 20 ms Failure rate: 8% API B Latency: 60 ms Failure rate: 0.01%
API A is faster when it works, but API B is likely more useful in production.
Reliability engineering often accepts a little extra latency or infrastructure cost in exchange for fewer failures.
Examples include:
- replicating data
- writing to durable queues
- performing health checks
- keeping spare capacity
- validating requests
Failures Are Normal
Reliable systems are designed with one assumption:
Something will eventually fail.
Possible failures include:
Server crashes Network packets disappear Database becomes overloaded DNS lookup fails Disk fills up Memory is exhausted Deployment contains a bug Certificate expires Third-party API slows down Cloud region experiences problems Message is delivered twice Worker crashes halfway through a job
A small system may experience these failures rarely. A large distributed system experiences some form of failure almost continuously because it contains many more components.
Suppose one server has a 0.1% chance of failure during a particular period.
With one server, failure is uncommon.
With thousands of servers, the probability that some server is failing becomes much higher.
This is why production architectures should not depend on every component always working perfectly.
Redundancy
One of the simplest reliability techniques is redundancy: do not depend on only one copy of a critical component.
Consider one application server:
Users │ ▼ App Server
If that server crashes:
Users │ ▼ X
The application becomes unavailable.
Now consider three servers:
┌──► App 1
Users ──► Load Balancer
├──► App 2
└──► App 3
If App 2 fails:
App 1 ✓ App 2 ✗ App 3 ✓
The load balancer can stop sending traffic to App 2 while the other servers continue processing requests.
The same idea applies to databases:
Primary Database
│
├──► Replica 1
└──► Replica 2
If the primary fails, one replica may be promoted depending on the database architecture.
Redundancy reduces single points of failure, but it introduces new questions:
How is failure detected? How quickly can traffic move? Is replicated data current? What happens during failover? Can both nodes accidentally become primary?
Redundancy helps only when the failover mechanism itself is reliable.
Timeouts
One of the most important reliability rules in distributed systems is:
Do not wait forever.
Imagine Service A calls Service B:
Service A ─────► Service B
Normally Service B responds in 100 ms.
Then Service B becomes overloaded and requests start taking 60 seconds.
Without a timeout:
Request 1 waiting... Request 2 waiting... Request 3 waiting... Request 4 waiting... Request 5 waiting... ...
Threads, connections, memory, and request slots remain occupied.
Eventually Service A can fail even though Service A itself was healthy.
With a timeout:
response = http_client.get(
"https://inventory-service/items/123",
timeout=2.0
)Service A decides that after two seconds, waiting is no longer useful.
The request fails quickly and resources are released.
Timeouts therefore prevent slow dependencies from consuming unlimited resources.
But timeout values matter.
Timeout too short → healthy requests fail unnecessarily Timeout too long → resources remain blocked too long
A reasonable timeout should be based on expected dependency latency and the latency budget of the caller.
For a deeper explanation, see Timeouts, Retries, and Exponential Backoff.
Retries
Some failures are temporary.
For example:
Request │ ▼ Database connection fails │ ▼ Connection recovers 200 ms later
Retrying may make the request succeed.
A simple retry flow:
Attempt 1 → failed Wait Attempt 2 → failed Wait Attempt 3 → success
Retries are useful for transient failures such as:
- temporary network errors
- short service interruptions
- rate limiting
- leader elections
- temporary resource contention
But uncontrolled retries are dangerous.
Suppose a service normally receives 10,000 requests per second.
The dependency starts failing and every request retries three times:
Original traffic: 10,000 req/sec 3 retries: 10,000 + 30,000 Total potential traffic: 40,000 req/sec
The retry mechanism can turn a partial outage into a much larger outage.
Exponential Backoff
Instead of retrying immediately:
retry retry retry retry
increase the delay between attempts:
Attempt 1 ↓ wait 100 ms Attempt 2 ↓ wait 200 ms Attempt 3 ↓ wait 400 ms Attempt 4 ↓ wait 800 ms
This gives the failing dependency time to recover.
Random jitter is often added so that thousands of clients do not retry at exactly the same moment.
import random
import time
delay = 0.2
for attempt in range(4):
try:
call_service()
break
except TemporaryError:
jitter = random.uniform(0, 0.1)
time.sleep(delay + jitter)
delay *= 2Idempotency
Retries become dangerous when an operation has side effects.
Consider a payment request:
Client │ ▼ Charge $100 │ ▼ Payment succeeds │ X response lost
The client does not know that the payment succeeded.
It retries:
Charge $100 again
Without protection, the customer may be charged twice.
An idempotency key allows the server to recognize repeated attempts:
headers = {
"Idempotency-Key": "order-81293-payment"
}The server stores the result of the first successful operation:
order-81293-payment
↓
already processed
↓
return previous result
This turns retrying into a safe operation.
Circuit Breakers
A timeout stops one request from waiting forever.
A circuit breaker stops the application from repeatedly calling a dependency that is already known to be unhealthy.
Imagine:
Recommendation Service
│
▼
External ML API
The external API becomes unavailable.
Without a circuit breaker:
Request 1 → wait → fail Request 2 → wait → fail Request 3 → wait → fail Request 4 → wait → fail ...
Every request pays the timeout cost.
With a circuit breaker:
Several failures detected
↓
Circuit opens
↓
New requests fail immediately
↓
Wait for recovery period
↓
Try dependency again
The states are usually described as:
CLOSED │ │ too many failures ▼ OPEN │ │ recovery timeout ▼ HALF-OPEN │ ├── success → CLOSED └── failure → OPEN
This protects both systems:
The caller avoids wasting resources, while the unhealthy dependency receives fewer requests while recovering.
Bulkheads
A bulkhead prevents one failing workload from consuming all resources.
The name comes from ships. A ship is divided into compartments so that flooding one compartment does not sink the entire ship.
The same concept applies to software.
Suppose one worker pool processes:
Emails Invoices Image processing Webhooks Reports
A huge report suddenly consumes every worker:
Worker 1 → report Worker 2 → report Worker 3 → report Worker 4 → report Worker 5 → report
Now emails and webhooks cannot run.
Instead, isolate workloads:
Email Workers ├── Worker 1 └── Worker 2 Report Workers ├── Worker 3 └── Worker 4 Webhook Workers └── Worker 5
A report overload now affects report processing, but other workloads remain operational.
Bulkheads can be implemented using:
- separate thread pools
- separate connection pools
- separate queues
- separate worker groups
- separate service instances
- resource quotas
Graceful Degradation
Reliable systems do not always need every feature to work.
Consider an online store:
Product page ├── Product details ├── Price ├── Inventory ├── Reviews ├── Recommendations └── Recently viewed items
Suppose the recommendation service fails.
A fragile implementation may return:
500 Internal Server Error
for the entire product page.
A better implementation may show:
Product details ✓ Price ✓ Inventory ✓ Reviews ✓ Recommendations unavailable Recently viewed ✓
The important part of the application continues working.
This is called graceful degradation.
Other examples:
Search ranking service fails → return lexical search results Personalization unavailable → show popular products Analytics system unavailable → buffer events and continue checkout Image resize service unavailable → serve original image Real-time inventory temporarily unavailable → display last known value with warning
The goal is to identify which features are critical and which can temporarily degrade.
Queues and Failure Isolation
Queues can dramatically improve reliability by separating request processing from background work.
Consider order creation:
Create Order │ ├──► Send email ├──► Update CRM ├──► Generate invoice ├──► Send analytics └──► Notify warehouse
If all operations happen synchronously, one failing dependency can break the whole request.
A queue changes the design:
Client
│
▼
Order API
│
├──► Database
│
└──► Queue
│
├──► Email Worker
├──► Invoice Worker
├──► CRM Worker
└──► Warehouse Worker
If the CRM system is unavailable:
CRM job ↓ fails ↓ retry later
The order itself still exists.
Queues also provide a buffer during temporary overload.
Incoming: 20,000 jobs/min Workers: 10,000 jobs/min Difference: 10,000 jobs/min temporarily stored in queue
Instead of failing immediately, the system accumulates a backlog and processes it when capacity becomes available.
However, queues require their own reliability mechanisms:
Retry policy Dead-letter queue Duplicate handling Visibility timeout Idempotent consumers Queue-depth monitoring
Health Checks
A system cannot route around failures unless it knows which instances are healthy.
Health checks usually answer questions such as:
Is the process alive? Can it accept traffic? Are critical dependencies usable?
A load balancer may check:
GET /health
and receive:
{
"status": "healthy"
}If one instance fails:
App 1 → healthy App 2 → unhealthy App 3 → healthy
the load balancer removes App 2 from traffic.
There are often different kinds of checks.
Liveness asks:
Should this process be restarted?
Readiness asks:
Should this instance receive requests right now?
An application may be alive but not ready while warming caches, applying migrations, or waiting for an important dependency.
For a deeper discussion, see Health Checks, Readiness, and Liveness Probes.
Reliable Data Storage
Reliability is not only about keeping APIs online. Data must also survive failures.
Consider a database with a single disk:
Database │ ▼ Single Disk
If the disk fails and no backup exists, the application may recover while the data does not.
Reliable storage usually combines multiple mechanisms:
Replication + Backups + Snapshots + Transaction logs + Recovery testing
Replication protects against some infrastructure failures:
Primary │ ├──► Replica A └──► Replica B
But replication is not a replacement for backups.
If application code accidentally executes:
DELETE FROM customers;replication may quickly copy that deletion to every replica.
A backup provides a separate recovery point.
The practical question is therefore not:
Do backups exist?
but:
Can the system actually restore from them? How long does recovery take? How much data could be lost?
These ideas are commonly expressed as:
RPO = acceptable data loss RTO = acceptable recovery time
For example:
RPO: 5 minutes RTO: 30 minutes
means the architecture should normally lose no more than about five minutes of data and should recover within about thirty minutes after a qualifying disaster.
Observability and Alerting
A reliability mechanism is useful only if failures can be detected.
Imagine an API returning 500 errors for 20% of requests.
If nobody notices for six hours, recovery mechanisms are incomplete.
Useful reliability signals include:
Request rate Error rate Latency CPU utilization Memory utilization Database connections Queue depth Retry rate Circuit breaker state Replication lag Cache hit rate Dependency failures
For example:
Normal: Error rate = 0.05% Incident: Error rate = 12%
An alert should identify that something meaningful changed.
Latency percentiles are also important.
p50 = 60 ms p95 = 120 ms p99 = 4,500 ms
The average may look acceptable while a small but important group of users experiences severe delays.
Reliable systems therefore monitor both successful operation and degraded behavior.
Example: Making an Online Store Reliable
Consider a small online store.
Stage 1: Simple architecture.
Users │ ▼ Application │ ▼ Database
It works, but both components are single points of failure.
Stage 2: Add multiple application instances.
┌──► App 1
Users ──► LB ─────┼──► App 2
└──► App 3
One application instance can now fail without taking down the entire service.
Stage 3: Add database replication.
Application
│
▼
Primary DB
│
├──► Replica 1
└──► Replica 2
Database infrastructure now has redundancy.
Stage 4: Add timeouts.
The product page calls several services:
Inventory Reviews Recommendations Pricing
Each call receives a bounded timeout.
Inventory 500 ms Pricing 500 ms Reviews 800 ms Recommendations 300 ms
A slow recommendation service can no longer block requests indefinitely.
Stage 5: Add graceful degradation.
Recommendations unavailable
↓
Product page still works
```
The recommendation section simply disappears temporarily.
Stage 6: Add circuit breakers.
If the review provider starts failing continuously:
Failures increase
↓
Circuit opens
↓
Stop calling provider temporarily
↓
Serve product page without reviews
Stage 7: Move secondary work to queues.
Checkout
│
├──► Save Order
│
└──► Queue
│
├──► Confirmation Email
├──► Analytics
├──► Warehouse Notification
└──► CRM Synchronization
The CRM being down no longer prevents checkout.
Stage 8: Add idempotency to payments.
Payment request
│
▼
Idempotency key
│
▼
Exactly one logical payment
A network retry cannot accidentally create a second charge.
Stage 9: Add monitoring and automatic recovery.
Health checks
│
├──► remove unhealthy instances
│
Metrics
├──► detect rising errors
│
Alerts
└──► notify operators
The final architecture may look like this:
┌──────────────┐
Users ──► Load Balancer ─┤ App Servers │
└──────┬───────┘
│
┌──────────────┼───────────────┐
│ │ │
▼ ▼ ▼
Cache Primary DB Services
│ │
┌──────┴──────┐ │
▼ ▼ │
Replica 1 Replica 2 │
│
Timeouts
Retries
Circuit Breakers
│
▼
Queue
│
┌───────────┼───────────┐
▼ ▼ ▼
Email Invoice CRM
Worker Worker Worker
No individual technique makes the system reliable.
Reliability comes from layers of protection:
Redundancy + Timeouts + Controlled retries + Circuit breakers + Isolation + Graceful degradation + Durable queues + Reliable storage + Monitoring + Recovery
Common Reliability Mistakes
Calling Services Without Timeouts
A dependency that never responds can consume connections and workers until the caller also becomes unavailable.
Every remote dependency should normally have a bounded waiting time.
Retrying Everything
Retries should not be automatic for every error.
Validation error → retry? No Authentication error → retry? Usually no Temporary network failure → retry? Often yes Rate limit → retry? Maybe, after delay Payment timeout → retry? Only with idempotency
Retry policy must understand the type of failure.
Keeping Hidden Single Points of Failure
An architecture may have ten application servers but still depend on:
One Redis instance One database One queue broker One NAT gateway One configuration service
Reliability must be evaluated across the entire request path.
Treating Every Feature as Critical
A recommendation widget should not normally have the same reliability requirements as checkout.
Classifying features helps decide what should fail open, fail closed, retry, degrade, or remain strictly consistent.
Having Backups but Never Testing Restore
A backup that cannot be restored is not a recovery strategy.
Restore procedures should be tested before an actual incident.
Running at 100% Capacity
A system operating permanently near maximum capacity has little room for failures.
Consider:
Normal traffic: 90% CPU One server fails: remaining servers must absorb its traffic Result: 100% CPU timeouts retries larger outage
Reliable systems usually keep some spare capacity so that they can tolerate component failures or sudden load increases.
Production Checklist
- Identify single points of failure.
- Run critical services with redundancy.
- Set explicit timeouts for remote calls.
- Retry only transient failures.
- Use exponential backoff and jitter.
- Make retried side-effecting operations idempotent.
- Use circuit breakers for repeatedly failing dependencies.
- Isolate unrelated workloads with bulkheads.
- Design non-critical features for graceful degradation.
- Use durable queues for asynchronous work.
- Monitor queue depth, retries, errors, and latency.
- Replicate critical data and maintain independent backups.
- Test restoration and failover procedures.
- Keep enough capacity to survive expected failures.
Conclusion
Reliability is not the absence of failures.
Reliability is the ability to continue operating when failures happen.
A reliable architecture assumes that servers will crash, networks will become slow, dependencies will fail, and software bugs will eventually reach production.
It then limits the consequences.
Failure happens
↓
Detect it
↓
Contain it
↓
Keep critical functionality working
↓
Recover
↓
Return to normal
The most reliable systems are usually built from multiple simple protections rather than one perfect mechanism.
Key Takeaway: design software so that the failure of one component becomes a small incident instead of a system-wide outage.
Comments (0)