Observability Explained: Logs, Metrics, and Traces

Observability is what makes complex production systems understandable when behavior no longer fits inside one process or one server. A single request may cross an API gateway, several microservices, databases, caches, queues, and external providers, while failures can emerge from interactions between components that appear healthy in isolation.

The practical problem is not collecting more telemetry. It is collecting the right signals with enough context to move from a user-visible symptom to the responsible component and underlying cause. Logs, metrics, and traces solve different parts of that problem and become significantly more useful when they are correlated.

A production observability architecture must also scale like any other distributed system. Telemetry volume, cardinality, retention, sampling, backpressure, storage cost, and collector failures all affect whether observability remains useful during the incidents when it matters most.

Table of Contents

• Observability vs Monitoring
• Logs, Metrics, and Traces
  • Logs
  • Metrics
  • Traces
• Correlating Observability Signals
• Production Observability Architecture
  • Telemetry Collection
  • Sampling, Cardinality, and Retention
• Failure Scenarios and Degraded Operation
• Production Design Example
• Ready-to-Use Example
• Common Mistakes
• Production Checklist
• Conclusion
• More Articles to Read

Observability vs Monitoring

Monitoring answers questions that engineers already know how to ask. Observability helps investigate behavior that was not predicted in advance.

A monitoring system can continuously evaluate conditions such as:

API availability < 99.9%

checkout error rate > 2%

p99 latency > 800 ms

queue oldest-message age > 5 minutes

database connection pool utilization > 90%

These are predefined signals. They are useful because they turn known failure modes into dashboards and alerts.

Observability becomes important after one of those conditions fires. A latency alert does not explain whether the cause is a deployment, database lock, connection pool saturation, network degradation, queue backlog, retry storm, or slow external provider.

Alert:
checkout p99 latency = 2.4 sec
           |
           v
Which region?
           |
           v
Which service?
           |
           v
Which dependency?
           |
           v
Which requests?
           |
           v
What changed?
           |
           v
Root cause

The distinction is operational rather than philosophical. The same telemetry infrastructure can support both monitoring and observability. Monitoring detects known conditions; observability provides enough context to investigate unknown ones.

Logs, Metrics, and Traces

Logs, metrics, and traces represent different views of production behavior. No single signal is ideal for every debugging or monitoring problem.

Metrics are efficient for answering how much and how often. Logs are useful for detailed event context. Traces explain how one operation moved through a distributed system.

Logs

A log represents an event. It is usually the most detailed observability signal and the easiest place to attach business context, error details, identifiers, state transitions, and diagnostic information.

Production logs should normally be structured:

{
  "timestamp": "2026-08-22T20:41:18Z",
  "level": "error",
  "service": "payment-service",
  "version": "2026.08.22.4",
  "environment": "production",
  "region": "us-east",
  "event": "payment_provider_timeout",
  "order_id": "ORD-918271",
  "provider": "payment-provider",
  "duration_ms": 3012,
  "trace_id": "3f8c149af8104bd4"
}

Structured fields make production queries predictable:

service = "payment-service"
AND region = "us-east"
AND event = "payment_provider_timeout"
AND version = "2026.08.22.4"

Advantages:

• rich event-level detail;
• good for exceptions and unexpected states;
• supports high-cardinality identifiers such as order IDs and trace IDs;
• useful for audit and business-event investigation.

Disadvantages:

• high ingestion and indexing cost;
• large volumes can make queries slow;
• unstructured formats become difficult to aggregate;
• sensitive information can leak into telemetry;
• verbose debug logging can consume significant storage.

Suppose 800 service instances generate an average of 1.5 MB of logs per minute:

800 × 1.5 MB/min
= 1.2 GB/min

≈ 1.73 TB/day

≈ 52 TB/month

That is logical payload before indexing, replication, and storage overhead. Log design is therefore also capacity and cost engineering.

Use logs when detailed context matters, especially for failures, state transitions, security-relevant events, and high-cardinality debugging. Avoid converting every successful low-value operation into an expensive indexed event.

For a deeper treatment of schema design, correlation fields, sensitive data, and production logging patterns, see Structured Logging for Distributed Systems.

Metrics

Metrics aggregate numeric system behavior over time. They trade event-level detail for extremely efficient monitoring of large workloads.

Typical service metrics include:

http_requests_total

http_request_duration_seconds

http_errors_total

active_connections

db_connection_pool_active

queue_oldest_message_age_seconds

cache_hit_ratio

Instead of retaining one record for every request, metrics can summarize millions of operations:

request rate = 41,000/sec

error rate = 0.8%

p50 latency = 52 ms
p95 latency = 210 ms
p99 latency = 870 ms

connection pool utilization = 78%

For request-driven systems, four categories form a practical baseline:

Percentiles are usually more meaningful than averages for latency. A healthy average can hide a badly degraded tail.

9,900 requests = 50 ms
  100 requests = 5 sec

Average ≈ 100 ms

p99 = 5 sec

Advantages:

• efficient storage for high-volume behavior;
• fast aggregation and dashboard queries;
• ideal for alerts, trends, SLOs, and capacity planning;
• good for comparing versions, regions, and services.

Disadvantages:

• aggregation loses individual request detail;
• high-cardinality labels can overwhelm the metrics backend;
• poorly selected metrics create noisy dashboards without operational value;
• incorrect histogram boundaries can reduce latency insight.

A critical production rule is to keep unbounded identifiers out of metric dimensions.

SAFE:

http_requests_total{
    service="checkout",
    region="us-east",
    status="500"
}


DANGEROUS:

http_requests_total{
    user_id="9182718",
    request_id="a81f..."
}

Millions of users multiplied by routes, regions, statuses, versions, and other labels can create millions of time series. High-cardinality identifiers usually belong in logs or traces instead.

For metric selection and alert-oriented telemetry design, see Metrics That Actually Matter in Production Systems.

Traces

A distributed trace represents one logical operation as it crosses service boundaries. Each operation within the request becomes a span.

Client
  |
  v
API Gateway
  |
  v
Order Service
  |
  +------> Inventory Service
  |
  +------> Payment Service
                  |
                  +------> Database
                  |
                  +------> External Provider

A trace can show exactly where latency accumulates:

[ POST /checkout                      1.92 sec ]
   |
   +-- [ order-service                1.88 sec ]
          |
          +-- [ inventory-service       38 ms ]
          |
          +-- [ payment-service        1.72 sec ]
                 |
                 +-- [ acquire DB conn  1.41 sec ]
                 |
                 +-- [ SELECT payment    84 ms ]

The trace demonstrates that the database query is not the main problem. Waiting for a database connection consumes most of the time.

Useful span attributes include:

{
  "trace_id": "3f8c149af8104bd4",
  "span_id": "ab1290cc",
  "service": "payment-service",
  "operation": "acquire_db_connection",
  "region": "us-east",
  "version": "2026.08.22.4",
  "duration_ms": 1412,
  "status": "error"
}

Tracing depends on propagation. The same trace context must survive HTTP, RPC, queues, and background jobs:

API
 |
 | trace=abc
 v
Service A
 |
 | trace=abc
 v
Message Queue
 |
 | trace=abc
 v
Worker
 |
 | trace=abc
 v
External API

Advantages:

• reveals cross-service request dependencies;
• shows latency contribution by component;
• helps identify retry amplification and fan-out behavior;
• connects synchronous and asynchronous workflows when context propagates correctly.

Disadvantages:

• high span volume at large request rates;
• instrumentation must propagate context correctly;
• sampling can discard rare failures;
• high-cardinality attributes can increase storage cost;
• very large traces can become expensive to process and query.

Detailed propagation and sampling strategies are covered in Distributed Tracing Across Microservices.

Correlating Observability Signals

The real production value appears when logs, metrics, and traces share enough context to support one investigation workflow.

A useful correlation model includes common attributes such as:

service.name
service.version
environment
region
availability_zone
trace_id
request_id
deployment_id

Consider an incident where checkout latency increases immediately after a deployment.

METRIC

checkout p99:
280 ms --> 2.1 sec

error rate:
0.4% --> 5.2%

affected version:
payment-service v43

A trace from the affected version reveals:

checkout                    2.05 sec
 |
 +-- order-service          2.01 sec
      |
      +-- inventory           42 ms
      |
      +-- payment-service    1.89 sec
           |
           +-- DB pool wait  1.63 sec

The associated structured log provides request-level context:

{
  "level": "warning",
  "service": "payment-service",
  "version": "v43",
  "region": "us-east",
  "event": "db_connection_wait",
  "trace_id": "3f8c149af8104bd4",
  "wait_ms": 1632,
  "pool_active": 100,
  "pool_max": 100
}

Metrics then confirm whether this is one request or a systemic capacity problem:

db_pool_active = 100
db_pool_max = 100
db_pool_waiters = 2,182

The investigation path becomes:

Metrics
"What changed?"
    |
    v
Traces
"Where is the time going?"
    |
    v
Logs
"What happened inside that operation?"
    |
    v
Metrics
"How widespread is the condition?"

There is no required starting signal. An engineer may begin with an error log, a trace, a customer report, or a dashboard anomaly. The architecture should make movement between signals cheap.

Production Observability Architecture

Telemetry generation should be isolated from telemetry storage. An application should not synchronously depend on a central logging or tracing platform to serve production traffic.

The observability pipeline should absorb normal bursts, tolerate backend outages, normalize attributes, control telemetry volume, and route different signals to specialized storage systems.

                      Production Services
                   /         |          \
                  /          |           \
               Logs       Metrics       Traces
                  \          |           /
                   \         |          /
                    v        v         v
                 Local Collectors
                        |
                        v
                 Central Collectors
                        |
               +--------+---------+
               |        |         |
               v        v         v
             Logs     Metrics    Traces
              |         |          |
              v         v          v
          Log Store  Metric DB  Trace Store
               \        |         /
                \       |        /
                 +------+-------+
                        |
                        v
                Query / Dashboards
                        |
                +-------+-------+
                |               |
                v               v
             Alerts         Engineers

Telemetry Collection

Collectors or agents provide an isolation layer between applications and telemetry storage.

Useful collector responsibilities include:

• batching small telemetry payloads;
• buffering temporary backend slowdowns;
• normalizing resource attributes;
• redacting sensitive fields;
• sampling traces;
• dropping low-value telemetry under overload;
• routing signals to different storage tiers.

Applications should avoid making telemetry export part of request success.

BAD

Request
  |
  v
Business logic
  |
  v
Synchronous log export
  |
  X telemetry backend slow
  |
  v
User request delayed


BETTER

Request
  |
  v
Business logic
  |
  v
Local telemetry buffer
  |
  +------> response
  |
  +------> async export

This does not mean telemetry must never be durable. Audit logs may require stronger guarantees than ordinary diagnostic logs. Different signal classes can have different loss policies.

Sampling, Cardinality, and Retention

Observability cost is usually driven by three dimensions: volume, cardinality, and retention.

Suppose a platform processes 150,000 requests per second and produces 12 spans per request:

150,000 × 12
= 1.8 million spans/sec

At 5% sampling:

1.8M × 0.05
= 90,000 spans/sec

Sampling must preserve useful incident data. A simple random policy may discard exactly the failed or unusually slow request needed for debugging.

A more useful policy can retain different classes differently:

Fast success
    |
    +--> sample 1%

Normal success
    |
    +--> sample 5%

Slow trace
    |
    +--> retain 100%

Error trace
    |
    +--> retain 100%

Tail-based sampling can make decisions after enough of the trace is available to determine whether it failed or exceeded latency thresholds. The cost is additional buffering and collector complexity.

Failure Scenarios and Degraded Operation

An observability system must continue behaving safely when components fail. Telemetry is secondary to the production workload, but losing all telemetry during an incident is also dangerous.

Collector failure. Applications should reconnect to another collector or continue using bounded local buffering. Collectors themselves should be deployed redundantly when telemetry availability matters.

Telemetry backend slowdown. Export queues begin accumulating data. The system needs explicit limits so telemetry cannot consume all local memory or disk.

Applications
    |
    v
Collector
    |
    v
Export Queue
    |
    X backend slow
    |
    v
Queue grows
    |
    +--> buffer
    +--> sample
    +--> spill to disk
    +--> drop low-priority telemetry

Storage fills. Retention policies, storage alerts, and ingestion limits should activate before the platform exhausts capacity. A log platform that fills its disks can lose the exact events needed to investigate the incident that created the log spike.

Cardinality explosion. A deployment accidentally adds request_id as a metric label. The number of series increases rapidly, driving memory usage and query latency. Cardinality monitoring and label controls should detect this before the metrics backend becomes unstable.

Trace propagation breaks. A service deployment stops forwarding trace context. Traces fragment even though requests still succeed. Instrumentation health should therefore be treated as a measurable production concern.

Deployment creates debug-log amplification. An accidental debug statement on a high-volume path can multiply log traffic. Rate limits, log sampling, and telemetry ingestion alerts protect the central platform.

Network partition. Local agents may lose connectivity to central collectors. Bounded buffering can absorb short outages, but an explicit drop or spill policy is required for long partitions.

The core principle is that telemetry failure should degrade visibility predictably rather than destabilize the application being observed.

Production Design Example

Consider a multi-service logistics platform handling shipment creation, quoting, booking, tracking, and carrier integrations.

Requests enter through an API gateway and may cross several services:

                             Client
                               |
                               v
                         API Gateway
                               |
                               v
                       Shipment Service
                         /           \
                        v             v
                 Rating Service   Booking Service
                      |                 |
                      v                 v
                  Rate Cache       Message Queue
                                        |
                                        v
                                  Booking Worker
                                        |
                                        v
                                  Carrier API
                                        |
                                        v
                                   PostgreSQL

The system needs observability across both synchronous HTTP requests and asynchronous carrier bookings.

Request flow. A shipment request receives a trace ID at the API gateway. That context propagates through Rating Service and Booking Service.

Write flow. Booking Service stores the booking request and publishes a queue message containing the trace context. The worker creates a child span when processing the event.

Read flow. Dashboards use metrics for throughput, latency, error rate, queue age, carrier performance, and database saturation. Engineers use traces and logs only when deeper investigation is required.

Failure flow. If one carrier starts timing out, metrics detect increased carrier-specific latency and retry rate. Traces show that most time is spent in the carrier span. Structured logs expose timeout and retry details.

Metrics:
carrier_request_p99 = 4.8 sec
retry_rate = 19%

        |
        v

Trace:
booking-worker
  |
  +-- carrier-api = 4.6 sec

        |
        v

Log:
event=carrier_timeout
carrier=carrier_a
attempt=3
trace_id=...

Monitoring. The platform watches user-visible request latency as well as queue oldest-message age, booking retry rate, carrier error rate, database pool utilization, collector queue saturation, telemetry drop count, and cardinality growth.

Scaling. Application and worker capacity scale according to workload metrics. Observability collectors scale independently according to telemetry ingestion rate and processing backlog.

Deployment considerations. Every telemetry event includes the service version. Dashboards compare the current version with the previous deployment so regression detection does not depend on manually matching release timestamps.

Suppose Booking Service version v58 introduces a connection leak.

bookings p99:
v57 = 320 ms
v58 = 1.9 sec

db_pool_active:
v57 instances = 35-55%
v58 instances = 100%

Tracing confirms requests wait on database connection acquisition rather than query execution. Logs correlate those requests with v58. The release can be rolled back while the root cause is investigated.

This example demonstrates why telemetry should carry both request context and deployment context. A trace ID explains one operation; service version and region explain why a whole population behaves differently.

Ready-to-Use Example

A small FastAPI service can expose production metrics, produce structured logs, and preserve trace context without coupling request success to telemetry storage.

import json
import logging
import time
from contextvars import ContextVar
from dataclasses import dataclass
from typing import Any
from uuid import uuid4

from fastapi import FastAPI, Request, Response

app = FastAPI()

trace_id_context: ContextVar[str] = ContextVar(
    "trace_id",
    default="",
)


@dataclass(frozen=True)
class ServiceContext:
    service: str
    version: str
    environment: str
    region: str


SERVICE = ServiceContext(
    service="shipment-service",
    version="2026.08.22.4",
    environment="production",
    region="us-east",
)


class JsonFormatter(logging.Formatter):
    def format(self, record: logging.LogRecord) -> str:
        payload: dict[str, Any] = {
            "timestamp": time.time(),
            "level": record.levelname.lower(),
            "service": SERVICE.service,
            "version": SERVICE.version,
            "environment": SERVICE.environment,
            "region": SERVICE.region,
            "trace_id": trace_id_context.get(),
            "message": record.getMessage(),
        }

        event = getattr(record, "event", None)
        if event:
            payload["event"] = event

        return json.dumps(payload)


handler = logging.StreamHandler()
handler.setFormatter(JsonFormatter())

logger = logging.getLogger("shipment-service")
logger.setLevel(logging.INFO)
logger.handlers.clear()
logger.addHandler(handler)


@app.middleware("http")
async def observability_context(
    request: Request,
    call_next,
) -> Response:
    trace_id = request.headers.get(
        "traceparent",
        str(uuid4()),
    )

    token = trace_id_context.set(trace_id)
    started = time.perf_counter()

    try:
        response = await call_next(request)

        duration_ms = (
            time.perf_counter() - started
        ) * 1000

        logger.info(
            "request_completed",
            extra={"event": "request_completed"},
        )

        response.headers["x-trace-id"] = trace_id
        response.headers["server-timing"] = (
            f"app;dur={duration_ms:.2f}"
        )

        return response

    except Exception:
        logger.exception(
            "request_failed",
            extra={"event": "request_failed"},
        )
        raise

    finally:
        trace_id_context.reset(token)

This example keeps the application concern small: establish request context and emit structured telemetry. Production deployments would normally export traces and metrics through instrumentation libraries and a collector rather than implementing telemetry protocols directly in application code.

A collector configuration can batch telemetry and prevent every service from connecting directly to backend storage:

receivers:
  otlp:
    protocols:
      grpc:
      http:

processors:
  memory_limiter:
    check_interval: 1s
    limit_mib: 1024

  batch:
    send_batch_size: 4096
    timeout: 2s

  attributes:
    actions:
      # Prevent accidental propagation of sensitive fields.
      - key: authorization
        action: delete

exporters:
  otlp:
    endpoint: telemetry-backend:4317

service:
  pipelines:
    traces:
      receivers: [otlp]
      processors:
        - memory_limiter
        - attributes
        - batch
      exporters: [otlp]

    metrics:
      receivers: [otlp]
      processors:
        - memory_limiter
        - batch
      exporters: [otlp]

The memory limiter is important because telemetry backpressure must not consume unlimited collector memory. Batching reduces exporter overhead, while centralized attribute processing provides a consistent place for redaction and normalization.

Infrastructure should deploy multiple collectors so one process is not responsible for all production telemetry:

AWSTemplateFormatVersion: "2010-09-09"

Resources:
  TelemetryCluster:
    Type: AWS::ECS::Cluster

  TelemetryService:
    Type: AWS::ECS::Service
    Properties:
      Cluster: !Ref TelemetryCluster
      DesiredCount: 3

      # Collectors remain horizontally replaceable.
      DeploymentConfiguration:
        MinimumHealthyPercent: 66
        MaximumPercent: 200

      NetworkConfiguration:
        AwsvpcConfiguration:
          AssignPublicIp: DISABLED
          Subnets:
            - subnet-a
            - subnet-b
            - subnet-c

The collector fleet should scale from telemetry ingestion rate, queue utilization, CPU, and memory rather than from application request rate alone. A debug-log deployment can multiply observability traffic without changing business traffic.

Common Mistakes

Production Checklist

• Instrument user-visible paths: cover critical APIs, jobs, queues, and external dependencies.
• Standardize service identity: emit service name, environment, region, and version consistently.
• Use structured logs: define stable event names and machine-readable fields.
• Include trace IDs in logs: make transitions between trace and log investigation immediate.
• Track traffic: measure request, job, event, and message throughput by major workload.
• Track error rates: distinguish application errors, dependency failures, timeouts, and rejected work.
• Track latency distributions: monitor p50, p95, and p99 rather than averages alone.
• Track saturation: monitor connection pools, queue age, worker utilization, memory, CPU, and storage.
• Control metric cardinality: reject or review labels containing unbounded identifiers.
• Propagate trace context: preserve context through HTTP, RPC, queues, scheduled jobs, and workers.
• Sample deliberately: use higher retention for errors and slow traces.
• Protect application latency: keep telemetry export outside synchronous business paths.
• Bound collector queues: prevent observability outages from consuming unlimited resources.
• Monitor telemetry drops: alert when logs, metrics, or spans are rejected or discarded.
• Monitor collector saturation: watch CPU, memory, queue utilization, and exporter latency.
• Monitor cardinality growth: detect unexpected time-series expansion after deployments.
• Monitor telemetry ingestion volume: catch accidental debug logging and instrumentation explosions.
• Annotate deployments: correlate regressions with service versions automatically.
• Redact secrets: remove authentication headers, tokens, credentials, and unnecessary personal data.
• Define retention tiers: keep recent searchable telemetry longer only where operationally useful.
• Plan storage growth: include indexes, replicas, and retention in capacity forecasts.
• Test collector failure: verify application behavior when telemetry infrastructure is unavailable.
• Test backend backpressure: verify buffering and drop policies under slow ingestion.
• Test trace propagation: verify end-to-end traces across synchronous and asynchronous boundaries.
• Measure query latency: observability data must remain searchable during incidents.
• Review telemetry cost: monitor cost per service, signal, environment, and retention tier.
• Document incident workflows: engineers should know how to move from alert to trace to logs and supporting metrics.

Conclusion

Production observability is not a logging project or a monitoring dashboard. It is an architecture for reducing uncertainty when distributed systems behave unexpectedly. Metrics efficiently show changes across large populations of requests, traces expose where distributed operations spend time, and logs provide detailed context around individual events.

The difficult production problems appear around the telemetry itself: high-cardinality metrics, excessive log volume, trace sampling, backend backpressure, sensitive data, broken context propagation, collector capacity, and long-term storage cost. Those concerns should be designed deliberately rather than treated as an afterthought.

Key Takeaway: Logs, metrics, and traces are most valuable when they form one correlated investigation model. Use metrics to identify system-wide symptoms, traces to narrow failures to specific distributed operations, and logs to explain detailed events. Keep telemetry outside the critical application path, control cardinality and volume, preserve context across service boundaries, and operate the observability pipeline as a production distributed system.

More Articles to Read

• Structured Logging for Distributed Systems
• Metrics That Actually Matter in Production Systems
• Distributed Tracing Across Microservices
• Designing Monitoring and Alerting Pipelines
• Production Troubleshooting and Root Cause Analysis
• Observability Best Practices for Production Systems