Distributed Tracing Across Microservices

Distributed systems make latency and failures harder to explain because a single user request can cross several services, queues, databases, caches, and external providers. Each component may look healthy in isolation while the complete request is slow, repeatedly retried, or partially failing.

Distributed tracing solves this by connecting related operations into one causal request graph. A trace represents the complete operation, while spans represent individual units of work such as HTTP requests, database queries, message consumption, cache access, or external API calls.

In production, the challenge is not simply creating spans. A useful tracing system must propagate context across synchronous and asynchronous boundaries, choose meaningful span boundaries, control sampling and cardinality, protect sensitive data, and operate a trace pipeline that can handle failures without affecting application traffic.

Table of Contents

• How Distributed Tracing Works
  • Traces and Spans
  • Understanding the Critical Path
• Context Propagation Across Services
  • HTTP and RPC Propagation
  • Queues and Background Workers
• Designing Useful Spans
• Sampling Strategies at Scale
• Production Tracing Architecture
• Failure Scenarios and Trace Analysis
• Production Design Example
• Ready-to-Use Example
• Common Mistakes
• Production Checklist
• Conclusion
• More Articles to Read

How Distributed Tracing Works

A trace models one logical operation from its entry point through the downstream work caused by that operation.

Consider a checkout request:

Client
  |
  v
API Gateway
  |
  v
Checkout Service
  |
  +------> Inventory Service
  |              |
  |              +------> PostgreSQL
  |
  +------> Payment Service
                 |
                 +------> Payment Provider

Metrics can reveal that checkout latency increased. Logs can show payment timeouts. A trace connects the full path and shows where time was actually spent.

Traces and Spans

A trace identifies the complete distributed operation. A span represents one unit of work inside that operation.

Trace: 47bd...

[ POST /checkout                    1.84 sec ]
   |
   +-- [ checkout-service           1.81 sec ]
          |
          +-- [ inventory-service     61 ms ]
          |      |
          |      +-- [ SELECT stock   17 ms ]
          |
          +-- [ payment-service      1.64 sec ]
                 |
                 +-- [ provider API  1.51 sec ]

A span commonly contains:

{
  "trace_id": "47bd24bf416f44eb967acc738016ca21",
  "span_id": "832aca7c12dc1901",
  "parent_span_id": "e84ac819178f9211",
  "service": "payment-service",
  "operation": "POST provider-payment",
  "duration_ms": 1514,
  "status": "ok",
  "region": "us-east",
  "version": "v84"
}

The trace ID stays stable across the complete operation. Each span receives its own span ID and links back to its parent.

This relationship matters because distributed operations are rarely a flat list of timings.

Checkout
 |
 +-- Inventory
 |
 +-- Payment
      |
      +-- Token Service
      |
      +-- Provider API

The hierarchy makes causality explicit.

Understanding the Critical Path

One important tracing concept is the critical path: the sequence of operations determining total end-to-end latency.

Suppose three services execute concurrently:

Checkout                      900 ms
 |
 +-- Inventory                120 ms
 |
 +-- Pricing                   80 ms
 |
 +-- Payment                  760 ms

The child durations add up to 960 ms, but the request still completes in 900 ms because operations overlap.

The payment path dominates end-to-end latency.

This distinction prevents a common debugging mistake: adding span durations together as if every operation executed sequentially.

Tracing is especially useful for discovering:

• slow downstream dependencies;
• unexpected sequential calls;
• large service fan-out;
• retries hidden inside request latency;
• queue waiting time;
• connection pool waits;
• cross-region calls;
• slow external providers.

Context Propagation Across Services

Tracing works only when services preserve the active trace context as requests move between processes.

Each downstream service extracts the incoming trace information, creates a child span, and forwards updated context when it calls another component.

HTTP and RPC Propagation

A synchronous request flow looks like:

Client
  |
  | trace=A
  v
API Gateway
  |
  | trace=A, parent=span-1
  v
Service A
  |
  | trace=A, parent=span-2
  v
Service B
  |
  | trace=A, parent=span-3
  v
Service C

The trace ID stays unchanged. Each service creates its own span and passes that span as the parent of the next operation.

Context propagation should normally be handled by tracing instrumentation rather than manually assembling headers throughout application code.

Advantages:

• creates one end-to-end request graph;
• works across independently deployed services;
• allows traces and structured logs to share the same trace ID;
• makes service boundaries visible without requiring centralized application code.

Disadvantages:

• every supported protocol must preserve context;
• custom clients may require additional instrumentation;
• one broken propagation boundary fragments the trace;
• cross-company boundaries may intentionally stop propagation.

A fragmented trace often looks like:

Trace A

API Gateway
   |
Order Service


Trace B

Payment Service
   |
Provider API

If Payment Service should be part of the original request, this shape usually indicates missing propagation.

Queues and Background Workers

Asynchronous workflows require special attention because the downstream operation may happen seconds, minutes, or hours after the original request.

API Request
    |
    v
Booking Service
    |
    v
Queue
    |
    | 45 seconds
    v
Booking Worker
    |
    v
Carrier API

The message should carry tracing metadata in headers or another propagation carrier:

{
  "headers": {
    "traceparent": "..."
  },
  "payload": {
    "shipment_id": "SHP-91821"
  }
}

The worker extracts the context and starts a processing span associated with the originating workflow.

Queue tracing should distinguish several different latency components:

Producer
   |
   | publish = 8 ms
   v
Queue
   |
   | waiting = 45 sec
   v
Worker
   |
   | processing = 120 ms
   v
Carrier API

If only processing time is recorded, the worker appears healthy while the user waits almost a minute.

For queued systems, traces should help expose:

• publish latency;
• queue waiting time;
• consumer processing latency;
• retry attempts;
• dead-letter transitions;
• downstream dependency latency.

Asynchronous tracing is particularly valuable when combined with queue age metrics. See Metrics That Actually Matter.

Designing Useful Spans

Tracing every local function produces expensive, noisy traces. A span should normally represent work that matters architecturally, operationally, or from a latency perspective.

Good span boundaries include:

• incoming HTTP requests;
• outgoing service calls;
• database operations;
• cache operations when they materially affect latency;
• message publication;
• message consumption;
• external provider calls;
• important workflow steps;
• expensive internal computations.

A noisy trace might look like:

POST /shipments
 |
 +-- parse_json
 |
 +-- validate_string
 |
 +-- normalize_address
 |
 +-- convert_boolean
 |
 +-- build_dict
 |
 +-- serialize_model
 |
 +-- save

A better production trace would be:

POST /shipments
 |
 +-- validate shipment
 |
 +-- INSERT shipment
 |
 +-- rating-service
 |
 +-- publish booking event

The second trace preserves architectural information while avoiding low-value implementation noise.

Attributes should follow the same principle.

Trace attributes are often indexed or searched. Large payloads and highly sensitive values therefore create both cost and security problems.

Sampling Strategies at Scale

Trace volume grows with both request throughput and the number of spans per request.

Suppose:

Requests/sec = 150,000
Average spans/request = 12

Full tracing produces:

150,000 × 12
= 1,800,000 spans/sec

If an average encoded span is roughly 1 KB:

≈ 1.8 GB/sec

≈ 155 TB/day

The real storage footprint depends on compression, indexing, attributes, replicas, and retention, but tracing every request indefinitely can clearly become expensive.

Sampling reduces trace volume.

Head vs Tail Sampling

Two common approaches are head and tail sampling.

Head sampling might keep 5% of all traces:

100 traces
   |
   v
random sampling
   |
   +--> 5 retained
   |
   +--> 95 dropped

The problem is that an important failed request may be discarded before its failure is known.

Tail sampling waits long enough to inspect outcomes:

Trace Buffer
    |
    +-- success 80 ms ------> sample 1%
    |
    +-- success 5 sec ------> retain
    |
    +-- HTTP 500 -----------> retain
    |
    +-- provider timeout ---> retain

A practical production policy might be:

Errors                    100%
Very slow traces          100%
Rare workflows             50%
Normal requests             5%
Health checks             0.1%

Healthy traces should still be retained because debugging often requires comparing a failing request with a normal request.

Advantages of head sampling:

• simple;
• predictable ingestion volume;
• minimal collector memory;
• easy horizontal scaling.

Disadvantages of head sampling:

• cannot know final trace outcome;
• rare failures can be discarded;
• high-value latency outliers may disappear.

Advantages of tail sampling:

• retains failed traces selectively;
• retains latency outliers;
• provides better diagnostic value per stored trace.

Disadvantages of tail sampling:

• requires buffering;
• higher memory requirements;
• more operationally complex;
• collector affinity may be needed so spans from the same trace meet at the same sampling layer.

Production Tracing Architecture

Tracing should not synchronously depend on the storage backend. Applications should record spans and export them asynchronously through a collector layer.

                         Services
                 /         |          \
                v          v           v
              API       Service     Worker
                \          |          /
                 \         |         /
                  v        v        v
                 Local Collectors
                        |
                        v
                 Gateway Collectors
                        |
                +-------+-------+
                |               |
                v               v
            Processing       Sampling
                \               /
                 \             /
                  v           v
                   Trace Storage
                        |
                        v
                Query / Visualization

The collector layer can handle:

• batching;
• retry policies;
• memory limiting;
• attribute enrichment;
• redaction;
• sampling;
• routing to multiple backends.

The critical reliability rule is:

Application availability
        >
trace completeness

If trace storage is unavailable, losing some spans is normally safer than allowing telemetry buffers to consume all application memory or block requests.

The tracing infrastructure should itself expose:

spans_received_total
spans_exported_total
spans_dropped_total

collector_queue_size
collector_queue_capacity

export_errors_total

sampling_rate

collector_memory
collector_cpu

These metrics reveal whether missing traces are caused by healthy traffic or a broken telemetry pipeline.

Failure Scenarios and Trace Analysis

Tracing is particularly valuable when production failures emerge from interactions between systems.

Slow external dependency.

POST /checkout                   3.2 sec
 |
 +-- inventory                    60 ms
 |
 +-- payment                     3.0 sec
      |
      +-- provider               2.9 sec

The latency source is clear.

Database connection saturation.

POST /shipments                 2.0 sec
 |
 +-- acquire DB connection      1.7 sec
 |
 +-- INSERT shipment             42 ms

The SQL statement is not slow. Connection acquisition is.

Retry amplification.

Payment Service
 |
 +-- attempt 1 ---- 3 sec timeout
 |
 +-- backoff ------- 1 sec
 |
 +-- attempt 2 ---- 3 sec timeout
 |
 +-- backoff ------- 2 sec
 |
 +-- attempt 3 ---- 3 sec timeout

The trace reveals that much of the total request latency comes from retry policy rather than one slow operation.

Queue backlog.

Publish booking
      |
      | 14 minutes waiting
      v
Booking Worker
      |
      | 85 ms processing
      v
Carrier API

The worker is fast. The queue is the actual bottleneck.

Broken propagation.

Trace A:
API --> Service A

Trace B:
Service B --> Database

If Service B belongs to the request, propagation is broken.

Collector outage. Span buffers grow, exporter failures increase, and eventually spans may be dropped according to bounded policies. Business traffic should continue.

Trace backend slowdown. Collector queues increase. Scaling collectors alone may not solve the problem if the storage backend is the bottleneck.

Deployment failure. A new version can produce longer spans or more downstream calls. Service version should therefore be included in resource metadata so trace populations can be compared.

Production Design Example

Consider a logistics platform that creates shipments, calculates rates, books freight through carriers, and updates shipment status asynchronously.

                              Client
                                |
                                v
                          API Gateway
                                |
                                v
                        Shipment Service
                         /             \
                        v               v
                 Rating Service    Booking Service
                     /   \              |
                    v     v             v
                 Cache Carrier      Booking Queue
                                       |
                                       v
                                 Booking Worker
                                  /           \
                                 v             v
                           PostgreSQL      Carrier API

The synchronous shipment request creates one trace.

POST /shipments                       620 ms
 |
 +-- validate shipment                 12 ms
 |
 +-- rating-service                   310 ms
 |     |
 |     +-- cache GET                    8 ms
 |     |
 |     +-- carrier rates              270 ms
 |
 +-- INSERT shipment                   31 ms
 |
 +-- publish booking event             16 ms

Request flow. The API gateway creates or continues trace context. Shipment Service propagates it to Rating Service and downstream carrier requests.

Write flow. Shipment Service persists the shipment and publishes the booking event with trace context attached.

Asynchronous flow. Booking Worker extracts the context and creates a processing span.

Original request
      |
      v
publish booking
      |
      v
message broker
      |
      | waiting 4 sec
      v
booking worker
      |
      +-- SELECT shipment
      |
      +-- carrier booking
      |
      +-- UPDATE status

Failure flow. Carrier A starts timing out:

Metrics:

carrier_error_rate = 28%
booking_queue_age = 7 min

          |
          v

Trace:

booking-worker
 |
 +-- carrier attempt 1 ---- timeout
 |
 +-- retry delay
 |
 +-- carrier attempt 2 ---- timeout
 |
 +-- retry delay
 |
 +-- carrier attempt 3 ---- timeout

The trace explains why worker throughput fell: each booking now occupies a worker for multiple timeout windows and retry delays.

Monitoring. Application metrics track carrier errors, booking latency, queue age, retries, and worker throughput. Trace-platform metrics track ingestion, collector memory, sampling, dropped spans, and export failures.

Scaling. Booking workers scale based on queue age and processing rate. Tracing collectors scale independently based on spans per second and buffered trace state.

Deployment. Every span includes service version. During canary deployment, traces from the candidate version can be compared with stable instances.

If version v85 starts making one additional database query per request:

v84:

booking-worker
 |
 +-- SELECT shipment
 |
 +-- carrier
 |
 +-- UPDATE shipment


v85:

booking-worker
 |
 +-- SELECT shipment
 |
 +-- SELECT account
 |
 +-- carrier
 |
 +-- UPDATE shipment

The extra dependency becomes immediately visible in traces even before it causes a major latency regression.

Ready-to-Use Example

A Python service can use OpenTelemetry instrumentation for FastAPI while exporting spans asynchronously through an OTLP collector.

from dataclasses import dataclass

from fastapi import FastAPI
from opentelemetry import trace
from opentelemetry.exporter.otlp.proto.grpc.trace_exporter import (
    OTLPSpanExporter,
)
from opentelemetry.instrumentation.fastapi import (
    FastAPIInstrumentor,
)
from opentelemetry.sdk.resources import Resource
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.trace.export import BatchSpanProcessor


@dataclass(frozen=True)
class ServiceConfig:
    name: str
    version: str
    environment: str
    region: str


CONFIG = ServiceConfig(
    name="shipment-service",
    version="v85",
    environment="production",
    region="us-east",
)


resource = Resource.create({
    "service.name": CONFIG.name,
    "service.version": CONFIG.version,
    "deployment.environment.name": CONFIG.environment,
    "cloud.region": CONFIG.region,
})

provider = TracerProvider(
    resource=resource,
)

exporter = OTLPSpanExporter(
    endpoint="http://otel-collector:4317",
    insecure=True,
)

provider.add_span_processor(
    BatchSpanProcessor(exporter)
)

trace.set_tracer_provider(provider)

app = FastAPI()

FastAPIInstrumentor.instrument_app(app)

tracer = trace.get_tracer(__name__)

The batch span processor avoids synchronously exporting every span from request handlers.

Custom spans should wrap important operations:

from dataclasses import dataclass


@dataclass(frozen=True)
class CarrierBooking:
    shipment_id: str
    carrier: str


async def book_with_carrier(
    booking: CarrierBooking,
) -> None:
    with tracer.start_as_current_span(
        "carrier.booking"
    ) as span:
        span.set_attribute(
            "carrier.name",
            booking.carrier,
        )

        try:
            await carrier_client.book(booking)

        except TimeoutError as exc:
            span.record_exception(exc)
            span.set_attribute(
                "error.type",
                "carrier_timeout",
            )
            raise

The carrier name is useful if the set of carriers is bounded. Authentication tokens, complete payloads, and sensitive customer data should not be attached to the span.

Context can be injected into asynchronous messages:

from typing import Any

from opentelemetry.propagate import inject


def build_booking_message(
    payload: dict[str, Any],
) -> dict[str, Any]:
    headers: dict[str, str] = {}

    inject(headers)

    return {
        "headers": headers,
        "payload": payload,
    }

The consumer restores the context:

from typing import Any

from opentelemetry.propagate import extract


async def consume_booking(
    message: dict[str, Any],
) -> None:
    context = extract(
        message["headers"]
    )

    with tracer.start_as_current_span(
        "booking.process",
        context=context,
    ):
        await process_booking(
            message["payload"]
        )

A collector can apply tail sampling and memory protection:

receivers:
  otlp:
    protocols:
      grpc:
      http:

processors:
  memory_limiter:
    check_interval: 1s
    limit_mib: 2048

  tail_sampling:
    decision_wait: 10s
    num_traces: 100000
    policies:
      - name: errors
        type: status_code
        status_code:
          status_codes:
            - ERROR

      - name: slow-traces
        type: latency
        latency:
          threshold_ms: 2000

  batch:
    send_batch_size: 4096
    timeout: 2s

exporters:
  otlp:
    endpoint: trace-storage:4317

service:
  pipelines:
    traces:
      receivers:
        - otlp
      processors:
        - memory_limiter
        - tail_sampling
        - batch
      exporters:
        - otlp

Tail sampling requires temporary trace state, so collector memory and trace arrival patterns must be capacity-planned. The memory limiter protects the collector itself, but production configuration should also define what happens when limits are reached.

Common Mistakes

Production Checklist

• Trace inbound requests: continue or create context at service entry points.
• Trace outbound requests: instrument meaningful service dependencies.
• Use standard propagation: avoid custom trace-header schemes where unnecessary.
• Propagate through queues: preserve context across asynchronous workflows.
• Instrument databases: distinguish connection waits from query execution.
• Instrument external providers: isolate third-party latency and failures.
• Instrument retries: make repeated attempts visible inside traces.
• Track queue wait time: separate backlog delay from worker latency.
• Choose meaningful spans: avoid tracing trivial local functions.
• Include service name: make ownership clear.
• Include service version: support deployment comparison.
• Include environment: separate production from non-production traces.
• Include region: expose localized failures.
• Use bounded attributes: avoid unnecessary high-cardinality indexing.
• Never attach secrets: exclude credentials and authentication headers.
• Review sensitive data: keep customer payloads out of spans by default.
• Use asynchronous exporters: isolate tracing from request latency.
• Batch spans: reduce exporter overhead.
• Bound collector memory: prevent telemetry pressure from destabilizing collectors.
• Monitor collector queues: detect downstream trace-storage pressure.
• Monitor dropped spans: visibility loss must itself be visible.
• Monitor exporter errors: detect storage and networking failures.
• Monitor trace ingestion: catch instrumentation explosions after deployment.
• Choose sampling intentionally: balance cost with diagnostic value.
• Retain failed traces: preserve important incidents.
• Retain slow traces: preserve latency outliers.
• Retain healthy traces: provide comparison baselines.
• Capacity-plan tail sampling: account for buffered trace state.
• Test propagation: verify complete traces across every protocol boundary.
• Test tracing outages: ensure telemetry failure cannot break application traffic.

Conclusion

Distributed tracing provides a causal view of production requests that cannot be reconstructed efficiently from isolated service logs or aggregate metrics alone. It reveals where latency accumulates, how services depend on one another, how retries amplify failures, and where asynchronous workflows spend time waiting rather than processing.

The production trade-off is telemetry volume and operational complexity. Context propagation must work across all relevant boundaries, spans must remain meaningful, sampling must preserve high-value requests, and collectors must be able to absorb failures without affecting the application being observed.

Key Takeaway: Treat traces as causal maps of meaningful distributed work. Propagate context through HTTP, RPC, queues, and workers; instrument service and dependency boundaries rather than every function; preserve failed, slow, and representative healthy traces; correlate traces with logs and metrics; and operate the tracing pipeline with explicit sampling, capacity, security, and failure policies.

More Articles to Read

• Observability Explained: Logs, Metrics, and Traces
• Structured Logging for Distributed Systems
• Metrics That Actually Matter
• Designing Monitoring and Alerting Pipelines
• Root Cause Analysis in Production
• Observability Best Practices for Production Systems