Metrics That Actually Matter

Production systems can expose thousands of metrics while still failing to answer the most important operational question: is the system working correctly for users? CPU usage, memory consumption, thread counts, cache statistics, database metrics, and container telemetry all provide useful context, but collecting everything does not automatically create useful monitoring.

The strongest metric strategy starts with system behavior and works inward. Request traffic, errors, latency, and saturation reveal whether a service is healthy; dependency and infrastructure metrics then explain why that behavior is changing. This keeps dashboards and alerts focused on production outcomes rather than arbitrary measurements.

Metrics also have architectural costs. Cardinality consumes memory and storage, incorrect aggregation hides failures, averages conceal tail latency, and badly designed alerts create operational noise. Production metric design therefore requires the same attention to scalability, failure modes, and cost as the application being monitored.

Table of Contents

• Metrics Should Answer Production Questions
• Four Core Service Signals
  • Traffic
  • Errors
  • Latency
  • Saturation
• Application vs Infrastructure Metrics
• Counters, Gauges, and Histograms
• Metric Cardinality and Dimensional Design
• Metrics During Production Failures
• Production Design Example
• Ready-to-Use Example
• Common Mistakes
• Production Checklist
• Conclusion
• More Articles to Read

Metrics Should Answer Production Questions

A metric is useful when it helps answer a production question or supports a decision. Recording a measurement simply because it is available usually produces dashboards full of data that nobody uses during an incident.

Useful questions include:

Are users successfully completing requests?

Is the service becoming slower?

Which operation is failing?

Which region is affected?

Is a dependency degrading?

Is the queue accumulating work?

Is the database approaching capacity?

Did the latest deployment change behavior?

How much traffic can the current capacity handle?

This creates a hierarchy of signals:

                 User Experience
                       |
                       v
          Traffic / Errors / Latency
                       |
                       v
               Service Saturation
                       |
                       v
              Dependencies
                       |
                       v
              Infrastructure

Metrics near the top describe symptoms experienced by users. Metrics near the bottom usually help diagnose causes.

For example:

SYMPTOM

checkout p99 latency = 2.8 sec
           |
           v
CAUSE INDICATOR

db_pool_waiters = 1,920
           |
           v
RESOURCE CONDITION

database connections = 100%

Alerting only on the database connection count can create noise because high utilization is not necessarily a problem. Alerting on degraded checkout latency identifies a user-visible symptom; connection-pool metrics then explain the cause.

This symptom-versus-cause distinction is central to practical production monitoring. Google SRE similarly recommends focusing user-facing monitoring on latency, traffic, errors, and saturation, while using lower-level signals to understand causes. :contentReference[oaicite:0]{index=0}

Four Core Service Signals

Most request-driven services need a small group of metrics before they need hundreds of infrastructure measurements. Traffic, errors, latency, and saturation provide a useful baseline because together they describe demand, correctness, performance, and remaining capacity.

Traffic

Traffic measures demand placed on the system. The correct unit depends on the workload.

API:
requests / second

Message consumer:
messages / second

Payment platform:
payments / second

Logistics platform:
shipments / minute

Database:
queries / second

Streaming pipeline:
events / second

Traffic should normally be segmented by bounded dimensions that materially change system behavior:

service
operation
region
status
version

Raw request rate alone can be misleading when requests have different computational costs. Ten thousand lightweight reads may consume fewer resources than one thousand expensive search requests.

Production systems should therefore identify the unit of work that best represents actual demand.

Advantages:

• provides the denominator for error rates;
• supports capacity planning;
• reveals traffic shifts between regions and operations;
• helps correlate resource saturation with workload growth.

Disadvantages:

• request counts may hide workload complexity;
• too many dimensions create cardinality problems;
• traffic alone says nothing about correctness or performance.

Errors

Error metrics should represent failed outcomes rather than only exceptions.

A request can fail without throwing an exception:

HTTP 429
HTTP 503
payment declined because dependency unavailable
queue message moved to DLQ
request rejected by overload protection
deadline exceeded
partial response returned

Absolute error counts are useful, but error rate usually provides better context:

errors
---------------- = error rate
total requests

Consider:

Service A:
100 failures / 10,000,000 requests
= 0.001%

Service B:
100 failures / 1,000 requests
= 10%

The same number of errors represents radically different production conditions.

Error metrics should also distinguish failures by meaningful bounded categories:

timeout
dependency_error
validation_error
rate_limited
internal_error
overloaded

Do not put raw exception messages into metric attributes. Exception messages can contain IDs, URLs, dynamic values, or user input and therefore create uncontrolled cardinality.

Latency

Latency is a distribution, not a single number.

An average can hide the requests that matter most operationally:

9,900 requests = 50 ms
  100 requests = 5 sec

The average is approximately 100 ms, which looks reasonable. The tail tells a different story:

p50 = 50 ms
p95 = 50 ms
p99 = 5 sec

Production latency monitoring should therefore use histograms or another distribution-preserving representation rather than recording only averages. Histograms aggregate populations of measurements while retaining enough information to analyze their distribution. :contentReference[oaicite:1]{index=1}

Latency should also be separated by outcome when useful:

Successful request:
2.4 sec

Failed validation:
4 ms

If both are aggregated together, a spike in fast failures can make average latency improve while the service is becoming less reliable.

Advantages:

• directly represents user experience;
• tail latency exposes saturation early;
• supports SLOs and regression detection;
• helps compare deployments and regions.

Disadvantages:

• histograms require sensible bucket or aggregation design;
• percentiles become noisy with very small sample sizes;
• high-dimensional latency metrics can become expensive.

Saturation

Saturation measures how close a constrained resource is to preventing additional work from being processed efficiently.

CPU utilization is one example, but many application bottlenecks appear elsewhere:

database connection pool
thread pool
worker concurrency
queue backlog
queue oldest-message age
memory
disk capacity
network bandwidth
file descriptors
rate-limit quota
external API concurrency

The most useful saturation metric is often the waiting work, not just resource utilization.

Database pool:

active connections = 100 / 100
waiting requests = 0

vs

active connections = 100 / 100
waiting requests = 2,400

Both report 100% utilization, but only the second clearly demonstrates insufficient capacity.

For asynchronous systems, queue age is often more operationally meaningful than queue depth:

Queue A:
100,000 messages
oldest = 4 sec

Queue B:
2,000 messages
oldest = 45 min

Queue A may simply process enormous throughput. Queue B is clearly failing to keep up.

Google's SRE guidance similarly treats saturation as a measure of constrained resources and notes that latency increases can be an early indicator of approaching saturation. :contentReference[oaicite:2]{index=2}

Application vs Infrastructure Metrics

Infrastructure metrics are important, but they should not replace application metrics. A service can have healthy CPU and memory while every request fails because a dependency is unavailable.

CPU                 42%
Memory              58%
Disk                31%

Looks healthy
     |
     v

Payment success rate
     12%

System is broken

A useful monitoring hierarchy separates three layers:

All three are useful, but they answer different questions.

For a payment service:

BUSINESS

payment_success_rate
        |
        v
APPLICATION

payment_request_p99
provider_error_rate
db_pool_waiters
        |
        v
INFRASTRUCTURE

cpu_utilization
memory_usage
network_errors

Alerts should generally begin as close to the user-visible symptom as practical. Lower-level metrics then support diagnosis and capacity prediction.

Counters, Gauges, and Histograms

Metric type should match the behavior being measured. Using the wrong representation can make queries confusing or produce misleading results.

Counters represent values that accumulate:

requests_total
payments_failed_total
bytes_sent_total
jobs_completed_total

Operational queries normally calculate their rate of change:

requests_total

1000 --> 1600 over 60 sec

request rate = 10/sec

Gauges represent a current value:

active_connections
queue_depth
memory_bytes
workers_running

Histograms represent distributions:

request_duration
database_query_duration
message_processing_duration
request_size
response_size

OpenTelemetry's metric model provides counters, up/down counters, gauges, and histograms for these different measurement semantics. :contentReference[oaicite:3]{index=3}

Metrics should describe stable concepts. A metric named around one implementation detail may become useless when the implementation changes, while a metric representing a service outcome remains valuable.

Metric Cardinality and Dimensional Design

Cardinality is one of the most important scalability properties of a metrics system. Each unique combination of metric attributes can create another time series that must be maintained, stored, and queried.

Suppose an HTTP metric contains:

method:
5 values

route:
100 values

status:
10 values

region:
4 values

version:
3 values

The theoretical number of combinations is:

5 × 100 × 10 × 4 × 3
= 60,000 time series

Add a million possible user IDs:

60,000 × 1,000,000

= potentially enormous cardinality

Actual combinations may be lower, but the dimensional model is fundamentally unsafe.

OpenTelemetry describes cardinality as the number of unique attribute combinations associated with a metric and notes that each combination requires separate aggregation state. High-cardinality values such as user IDs or raw URL paths can therefore create substantial memory and storage pressure. :contentReference[oaicite:4]{index=4}

Good metric attributes are normally bounded:

GOOD

method = GET | POST | PUT | DELETE

region = us-east | us-west | eu-west

status_class = 2xx | 4xx | 5xx

operation = create_shipment | get_shipment

Dangerous attributes are usually unbounded:

BAD

user_id
request_id
trace_id
order_id
shipment_id
raw_url
exception_message
SQL query text

Those values belong naturally in logs or traces, where high-cardinality event context is expected.

Modern metric SDKs may enforce cardinality limits to protect application memory, but hitting the limit can degrade dimensional accuracy. In OpenTelemetry, overflowed combinations can be aggregated into an overflow data point; totals remain available, while queries grouped or filtered by the discarded attributes can undercount. :contentReference[oaicite:5]{index=5}

That means cardinality protection is a safety mechanism, not a substitute for good metric design.

Metrics During Production Failures

Metrics should be designed around how systems actually fail. The most useful signals often describe queues, waiting work, dependency behavior, and successful outcomes rather than only infrastructure utilization.

Database slowdown.

request_p99
220 ms --> 2.4 sec

db_query_p99
40 ms --> 70 ms

db_pool_wait_p99
5 ms --> 1.9 sec

The database queries themselves remain reasonably fast. The actual bottleneck is acquiring connections.

Worker capacity failure.

incoming messages = 12,000/sec
processed messages = 9,000/sec

queue depth:
10K --> 500K

oldest message:
2 sec --> 11 min

Queue age demonstrates the user-visible processing delay more clearly than CPU alone.

Dependency outage with retries.

incoming API traffic = stable

provider requests:
8,000/sec --> 21,000/sec

provider error rate:
0.2% --> 34%

retry rate:
0.1% --> 162%

The retry metric reveals traffic amplification that could turn a dependency failure into a cascading failure.

Deployment regression.

version=v71
p99=240 ms
errors=0.3%

version=v72
p99=1.8 sec
errors=4.9%

Deployment version should be a bounded dimension so releases can be compared directly during rollout.

Monitoring pipeline failure. Missing data is not automatically zero. Dashboards and alerts must distinguish a healthy zero from telemetry that stopped arriving.

Production Design Example

Consider a logistics platform where shipment requests trigger rating and carrier booking operations.

                         Clients
                            |
                            v
                       API Gateway
                            |
                            v
                    Shipment Service
                     /            \
                    v              v
             Rating Service   Booking Service
                    |              |
                    v              v
                 Cache          Queue
                                   |
                                   v
                            Booking Workers
                                   |
                                   v
                              Carrier APIs
                                   |
                                   v
                              PostgreSQL

The metric strategy starts at the user-facing boundary.

Shipment API:

shipment_requests_total
shipment_request_duration
shipment_errors_total
active_requests

Rating:

rating_requests_total
rating_duration
rating_errors_total
cache_hit_ratio
carrier_rate_request_duration

Booking:

booking_requests_total
booking_success_total
booking_failure_total
booking_duration
booking_retry_total

Queue:

booking_queue_depth
booking_queue_oldest_message_seconds
booking_messages_processed_total
booking_messages_failed_total

Database:

db_pool_active
db_pool_waiters
db_pool_wait_duration
db_query_duration
db_errors_total

Suppose carrier A begins responding slowly.

carrier latency increases
          |
          v
worker throughput falls
          |
          v
queue age increases
          |
          v
booking completion latency increases
          |
          v
customer-facing SLO degrades

The metrics show the failure propagating through the architecture.

Capacity planning can use the same signals. Assume peak workload is:

12,000 bookings/sec

worker throughput:
80 bookings/sec/worker

The theoretical minimum is:

12,000 / 80
= 150 workers

Running exactly 150 workers leaves no room for bursts, failures, slower carriers, deployments, or uneven workload distribution.

At 30% headroom:

150 × 1.30
= 195 workers

Production capacity should still be validated with queue age and latency rather than assuming throughput remains linear as concurrency increases.

Monitoring. Alerts focus on booking success rate, user-facing latency, and queue age. CPU, database pools, carrier latency, retries, and worker utilization provide diagnostic context.

Scaling. Worker autoscaling should consider backlog and queue age in addition to CPU because I/O-bound workers can be overloaded while CPU remains low.

Deployment. Service version is included as a bounded metric dimension during rollout so new and old instances can be compared.

Ready-to-Use Example

A FastAPI service can instrument a small set of high-value metrics without exposing request-specific identifiers as dimensions.

import time
from dataclasses import dataclass

from fastapi import FastAPI, Request, Response
from prometheus_client import Counter, Gauge, Histogram

app = FastAPI()


@dataclass(frozen=True)
class MetricLabels:
    service: str
    region: str


LABELS = MetricLabels(
    service="shipment-service",
    region="us-east",
)


REQUESTS = Counter(
    "http_requests_total",
    "Total HTTP requests",
    ["service", "region", "method", "route", "status_class"],
)

REQUEST_DURATION = Histogram(
    "http_request_duration_seconds",
    "HTTP request duration",
    ["service", "region", "method", "route"],
    buckets=(
        0.025,
        0.05,
        0.1,
        0.25,
        0.5,
        1.0,
        2.5,
        5.0,
    ),
)

ACTIVE_REQUESTS = Gauge(
    "http_active_requests",
    "Currently executing HTTP requests",
    ["service", "region"],
)


@app.middleware("http")
async def metrics_middleware(
    request: Request,
    call_next,
) -> Response:
    started = time.perf_counter()

    ACTIVE_REQUESTS.labels(
        service=LABELS.service,
        region=LABELS.region,
    ).inc()

    try:
        response = await call_next(request)

        status_class = f"{response.status_code // 100}xx"

        REQUESTS.labels(
            service=LABELS.service,
            region=LABELS.region,
            method=request.method,
            route=request.url.path,
            status_class=status_class,
        ).inc()

        return response

    finally:
        duration = time.perf_counter() - started

        REQUEST_DURATION.labels(
            service=LABELS.service,
            region=LABELS.region,
            method=request.method,
            route=request.url.path,
        ).observe(duration)

        ACTIVE_REQUESTS.labels(
            service=LABELS.service,
            region=LABELS.region,
        ).dec()

There is one important production issue in this implementation: request.url.path may contain dynamic identifiers.

/shipments/1001
/shipments/1002
/shipments/1003
...

That creates one label value per shipment. Metrics should use the normalized route template instead:

/shipments/{shipment_id}

The production implementation should therefore record framework route names or normalized route templates rather than raw URLs.

Business outcomes should also have dedicated counters:

BOOKINGS = Counter(
    "shipment_bookings_total",
    "Shipment booking outcomes",
    ["carrier", "result"],
)


def record_booking(
    carrier: str,
    success: bool,
) -> None:
    BOOKING_RESULT = (
        "success"
        if success
        else "failure"
    )

    BOOKINGS.labels(
        carrier=carrier,
        result=BOOKING_RESULT,
    ).inc()

The carrier label is appropriate only if the set of carriers is reasonably bounded. Shipment ID would not be appropriate.

Queue monitoring should expose both depth and age:

from dataclasses import dataclass


@dataclass(frozen=True)
class QueueState:
    depth: int
    oldest_message_age_seconds: float


QUEUE_DEPTH = Gauge(
    "booking_queue_depth",
    "Number of pending booking messages",
)

QUEUE_AGE = Gauge(
    "booking_queue_oldest_message_seconds",
    "Age of the oldest pending booking message",
)


def record_queue_state(state: QueueState) -> None:
    QUEUE_DEPTH.set(state.depth)
    QUEUE_AGE.set(
        state.oldest_message_age_seconds
    )

Depth describes accumulated work; age describes how long that backlog affects processing latency. Monitoring both prevents misleading conclusions for naturally high-volume queues.

Common Mistakes

Production Checklist

• Measure traffic: identify the workload unit that represents actual system demand.
• Measure errors: include failed outcomes beyond exceptions.
• Calculate error rates: relate failures to total traffic.
• Measure latency distributions: preserve p50, p95, and p99 behavior.
• Separate important outcomes: avoid hiding fast failures inside successful latency.
• Measure saturation: track constrained resources and waiting work.
• Monitor queue age: do not rely on queue depth alone.
• Monitor connection-pool waiting: pool utilization alone can miss contention severity.
• Measure dependency latency: identify external bottlenecks independently from application latency.
• Measure dependency errors: distinguish internal failures from downstream failures.
• Measure retries: detect traffic amplification during incidents.
• Measure business outcomes: track critical operations such as successful bookings or payments.
• Use bounded labels: review every metric dimension for expected cardinality.
• Normalize routes: never expose dynamic IDs through URL labels.
• Keep IDs out of metrics: request, trace, user, shipment, and order IDs belong elsewhere.
• Use stable error categories: avoid exception messages as dimensions.
• Track service versions: compare rollout behavior between deployments.
• Track regions: make localized failures visible.
• Monitor cardinality: detect unexpected series growth.
• Monitor metric overflow: ensure cardinality limits are not silently degrading dimensional queries.
• Detect stale telemetry: distinguish missing metrics from legitimate zero values.
• Monitor ingestion health: track exporter failures, rejected data, and collection lag.
• Load-test metrics: validate instrumentation under production-level cardinality and throughput.
• Review dashboard usage: remove metrics and panels that no longer support operational decisions.
• Review alerts: every page should correspond to an expected human action.
• Use metrics for capacity planning: correlate throughput with latency and saturation.
• Preserve capacity headroom: do not size production systems only for observed average traffic.
• Track telemetry cost: measure storage and ingestion growth caused by new dimensions.

Conclusion

Useful production metrics describe system behavior rather than merely exposing everything that can be measured. Traffic shows demand, errors show correctness, latency shows performance, and saturation shows how close the system is to its limits. Business and application signals identify symptoms, while dependency and infrastructure metrics provide the context needed to diagnose them.

The difficult engineering work is choosing representations and dimensions that remain meaningful at scale. Histograms preserve latency distributions, bounded labels make segmentation practical, queue age exposes processing delay, and carefully selected saturation metrics reveal bottlenecks before resources completely fail.

Key Takeaway: Measure outcomes first and causes second. Start with traffic, errors, latency, and saturation, add business metrics for critical workflows, and use infrastructure metrics to explain degraded behavior. Keep dimensions bounded, preserve latency distributions, measure waiting work and retries, and treat metric cardinality and telemetry health as production capacity concerns.

More Articles to Read

• Observability Explained: Logs, Metrics, and Traces
• Structured Logging for Distributed Systems
• Distributed Tracing Across Microservices
• Designing Monitoring and Alerting Pipelines
• Production Troubleshooting and Root Cause Analysis
• Observability Best Practices for Production Systems