Category: Observability Tags: amazon-cloudwatch monitoring observability

What Is CloudWatch?

By Oleksandr Andrushchenko — Published on Oct 24 — Modified on Oct 24

Amazon CloudWatch is a monitoring and observability service from AWS. It collects metrics, logs, and other operational signals from AWS infrastructure and applications, allowing engineers to understand system health, investigate failures, create dashboards, and configure automated alarms. Services such as EC2, Lambda, ECS, RDS, SQS, and Application Load Balancers publish metrics that CloudWatch can use to reveal resource utilization, traffic, errors, and other behavior.

CloudWatch is more than a place to inspect CPU utilization or read application logs. It can detect elevated error rates, measure request latency, monitor background processing, query structured logs, and trigger notifications or supported automated responses. However, enabling CloudWatch does not automatically make an application observable. Engineers must decide which signals matter, instrument application-level behavior, manage metric dimensions, configure useful alarms, and control the cost of telemetry collection.

What Amazon CloudWatch Provides
- Monitoring vs. Observability
- CloudWatch vs. CloudTrail, X-Ray, and Prometheus
CloudWatch Metrics, Logs, and Traces
Alarms, Dashboards, and Automated Responses
Production Design Example: Observability for a SaaS API
- API, Database, and Worker Monitoring
- Failure Scenario: Latency Increases but CPU Looks Normal
Ready-to-Use Example: CloudWatch with Terraform and Python
Performance, Cost, and Production Practices
Frequently Asked Questions
Conclusion
- More Articles to Read

What Amazon CloudWatch Provides

Distributed applications produce information about their behavior continuously. A load balancer records HTTP traffic, Lambda reports invocation errors, RDS exposes database resource metrics, and application code records business events. Without a shared monitoring approach, this information remains fragmented across services.

CloudWatch provides a place to collect and investigate many of these signals. AWS-managed service integrations reduce instrumentation work for infrastructure metrics, while custom metrics, log ingestion, and tracing integrations make application-specific monitoring possible.

Monitoring vs. Observability

Monitoring answers questions that engineers have already defined, such as whether an API's error rate exceeds 2% or whether database CPU utilization is unusually high.

Observability goes further. It provides enough information to investigate unexpected behavior, including failure modes that were not anticipated when dashboards and alarms were designed.

Metrics: Reveal trends, error rates, resource utilization, and service performance.
Logs: Provide detailed records of application execution and operational events.
Traces: Connect operations across multiple services and dependencies.
Alarms: Identify conditions requiring investigation or action.
Dashboards: Present related operational signals in one place.

Consider an API reporting an elevated error rate. A metric identifies the problem, logs reveal database connection failures, and a distributed trace helps show where requests spend time.

No single signal tells the complete story. Reliable observability depends on collecting signals that can be correlated using service names, request identifiers, timestamps, deployment versions, and trace context.

CloudWatch vs. CloudTrail, X-Ray, and Prometheus

Several AWS and open-source services participate in observability and auditing, but their responsibilities are different.

Service	Primary Purpose	Example
Amazon CloudWatch	Metrics, logs, alarms, dashboards, and operational observability	API latency and error rate
AWS CloudTrail	Audit supported AWS API and account activity	Who changed an IAM policy
AWS X-Ray	Distributed tracing and request analysis	Identify a slow downstream call
Prometheus	Time-series monitoring and query ecosystem	Container and application metrics
OpenTelemetry	Vendor-neutral telemetry instrumentation and collection	Generate metrics, logs, and traces

CloudWatch and CloudTrail are complementary. CloudWatch helps detect and diagnose operational problems, while CloudTrail helps investigate supported AWS API activity. A deployment failure might require both: CloudWatch reveals an unavailable service, and CloudTrail may show a relevant configuration change.

OpenTelemetry can instrument applications independently of a specific monitoring backend. AWS supports OpenTelemetry-based integrations, making it useful for teams that want consistent instrumentation across CloudWatch and other observability systems.

CloudWatch Metrics, Logs, and Traces

CloudWatch stores different forms of telemetry using different data models. A metric is an aggregated time series, a log event is a timestamped record, and a trace represents the execution path of an operation across components.

Choosing the right signal matters. Request IDs belong in logs and traces, not in custom metric dimensions. Error counts belong in metrics when engineers need cheap aggregation and alarms.

Metrics, Namespaces, and Dimensions

A CloudWatch metric is identified by its namespace, metric name, and complete set of dimensions. Namespaces organize metrics, while dimensions identify specific resource or application variants.

A custom API metric might have these properties:

{
  "Namespace": "SysDesPro/Demo",
  "MetricName": "RequestCount",
  "Dimensions": [
    {
      "Name": "Service",
      "Value": "orders-api"
    },
    {
      "Name": "Environment",
      "Value": "production"
    }
  ],
  "Value": 1,
  "Unit": "Count"
}

CloudWatch stores metrics separately for different dimension combinations. A metric for Service=orders-api is not automatically the same time series as a metric for Service=payments-api.

Common AWS namespaces include AWS/EC2, AWS/Lambda, AWS/SQS, and AWS/ApplicationELB. Custom applications should use their own namespace rather than publishing application data into AWS-reserved namespaces.

CloudWatch supports standard-resolution metrics and high-resolution custom metrics. Standard resolution uses one-minute granularity, while high-resolution metrics can be recorded with one-second granularity. Sub-minute collection and alarming may increase costs.

Choose statistics carefully. Sum is useful for request and error counts; Average is useful for some utilization measurements; Maximum can identify peaks. For request latency, percentile measurements such as p95 and p99 reveal slow-tail behavior that averages may hide.

Metric dimensions must remain bounded. Do not create a unique time series for every user ID, request ID, session, or full URL path. High-cardinality dimensions can dramatically increase the number of billable custom metrics.

CloudWatch Logs and Logs Insights

CloudWatch Logs organizes log events into log groups and log streams. A log group commonly represents one application or workload, while streams identify individual execution environments or log producers.

For example, Lambda typically writes execution logs into a group named /aws/lambda/function-name. ECS applications can send container output to CloudWatch Logs through the awslogs log driver.

Structured JSON logs are particularly useful because engineers can filter and aggregate named attributes without relying exclusively on fragile text searches.

{
  "level": "ERROR",
  "service": "orders-api",
  "environment": "production",
  "request_id": "req-1042",
  "event": "database_timeout",
  "route": "/orders/{order_id}",
  "duration_ms": 2100,
  "dependency": "postgresql",
  "deployment_version": "v42"
}

CloudWatch Logs Insights allows engineers to query log groups interactively. The following query counts error events by application event type over five-minute intervals:

fields @timestamp, level, event, service
| filter level = "ERROR"
| stats count(*) as errors by bin(5m), service, event
| sort bin(5m) desc

Logs Insights has its own query languages and operators; the example above uses its query syntax rather than PostgreSQL SQL. JSON fields can be automatically discovered for supported log events.

Good logs explain what happened without exposing sensitive data. Avoid logging authentication tokens, passwords, AWS credentials, raw payment information, and unnecessary personal information.

Log retention is configurable. Short-lived development logs may need only days of retention, while operational or regulatory requirements may justify longer periods. The correct policy depends on incident-response and compliance needs.

Application Signals, Tracing, and Synthetic Monitoring

CloudWatch includes integrations and capabilities for application performance monitoring beyond traditional infrastructure metrics. CloudWatch Application Signals helps teams observe supported application services and operations, including service-level indicators related to availability, latency, and dependencies.

Distributed tracing connects an incoming request to downstream database calls, HTTP requests, or asynchronous work. This is particularly useful when an API spans several services and the slow component is not obvious from top-level metrics.

CloudWatch Synthetics provides canaries that periodically test defined application behavior, such as loading a page or verifying an HTTP endpoint.

CloudWatch RUM can collect real-user monitoring information for supported web applications. Synthetic monitoring answers whether a controlled test succeeds; real-user monitoring provides information about actual browser experiences.

These capabilities have different instrumentation requirements and charges. Enable them because they answer specific operational questions, not simply because they are available.

Alarms, Dashboards, and Automated Responses

Telemetry becomes operationally valuable when engineers can use it to detect failures early and decide what to do next. CloudWatch alarms evaluate defined conditions against metric data and can transition between states.

A well-designed alarm should indicate an actionable problem, not merely that a resource has changed from its usual behavior.

Metric Alarms and Missing Data

CloudWatch metric alarms commonly have three states:

OK: The monitored condition is not currently breaching according to alarm evaluation.
ALARM: The alarm's configured conditions have been met.
INSUFFICIENT_DATA: Available data is not sufficient for a normal evaluation.

For example, an alarm may enter ALARM when five-minute API error rate exceeds a threshold over a configured number of evaluation periods.

CloudWatch allows an alarm to require M breaching periods out of N evaluated periods. This can reduce noise from isolated spikes while still detecting sustained degradation.

The treat_missing_data setting is important:

breaching: Missing data can contribute to an alarm condition.
notBreaching: Missing data is considered acceptable when used for evaluation.
ignore: Retain the current alarm state in applicable missing-data scenarios.
missing: Allow an insufficient-data state when appropriate.

For a sparse error-count metric, missing data might be normal because no errors occurred. For a heartbeat metric expected every minute, missing data can indicate that the application or telemetry pipeline stopped.

This distinction matters because a broken telemetry publisher can make an otherwise serious incident invisible if all missing data is treated as healthy.

Composite Alarms and Alert Noise

CloudWatch composite alarms combine the states of other alarms using Boolean expressions. They can represent application health more effectively than sending separate pages for every infrastructure metric.

For example, an application alarm can depend on combinations of elevated API errors, high latency, and dependency failures.

Composite alarms help reduce alert noise, but they do not create meaningful monitoring by themselves. The underlying alarms still need suitable thresholds and reliable data.

A production paging strategy should distinguish customer-impacting failures from diagnostic resource warnings. High CPU alone may not justify waking an engineer if service performance remains healthy.

Alarms can invoke configured actions, including SNS notifications and supported automated scaling or incident workflows. An SNS topic without an active subscriber does not automatically notify anyone.

Dashboards and Operational Investigation

CloudWatch dashboards can combine metrics, alarm status, log-derived information, and other supported widgets into a shared operational view.

A useful API dashboard might present:

Requests per minute.
HTTP 5xx error count and error percentage.
p50, p95, and p99 latency.
Running application instances or tasks.
Database connections and query latency.
SQS oldest message age.
Active alarms and relevant deployment markers.

Dashboards should help responders answer whether users are affected, which component is failing, and what changed recently.

A dashboard containing dozens of unrelated CPU and memory charts may look comprehensive while making real incidents harder to investigate.

Production Design Example: Observability for a SaaS API

Consider a SaaS application with a FastAPI service running on ECS Fargate, PostgreSQL on RDS, Redis caching, and SQS workers. The application handles user requests and generates reports asynchronously.

The platform needs to detect user-visible failures, correlate them with infrastructure issues, and distinguish API problems from background-processing delays.

API, Database, and Worker Monitoring

A production observability architecture can use the following signals:

Component	Metrics	Logs or Traces
Application Load Balancer	Request count, 5xx errors, target response time	Access logs where enabled
FastAPI on ECS	Request latency, errors, CPU, memory	Structured application logs and request traces
Amazon RDS	Connections, CPU, storage, I/O	Database logs and query diagnostics
Amazon SQS	Visible messages, oldest message age	Worker processing and retry logs
Report workers	Jobs completed, failures, processing duration	Job and workflow correlation records

The API should use consistent fields for service, environment, version, route template, and request or trace identifiers. Workers should preserve the originating workflow identifier when processing messages.

For example, a report-generation request can create a workflow ID that appears in the API log, SQS message, worker logs, and report completion event.

A practical alarm strategy begins with user-visible outcomes: sustained elevated 5xx rates, critical API latency, and report jobs exceeding their maximum acceptable processing delay. Infrastructure metrics then provide diagnostic evidence.

Failure Scenario: Latency Increases but CPU Looks Normal

Suppose API p99 latency rises from 250 milliseconds to four seconds after a deployment. ECS task CPU remains around 40%, and memory utilization stays stable.

If engineers monitor only CPU and memory, the application appears healthy. But database connection pool metrics show that all connections are occupied and requests are waiting for available connections.

CloudWatch metrics reveal the latency regression, structured logs reveal connection acquisition timeouts, and traces show the delay before database queries begin.

The correct response may be to investigate a new query pattern or connection leak rather than adding more ECS tasks. Blindly scaling the service could create even more concurrent database connection attempts.

This illustrates why application observability must include queueing, dependency performance, and resource saturation—not only host-level utilization.

For more detailed architectural guidance, see Observability Best Practices for Production Systems.

Ready-to-Use Example: CloudWatch with Terraform and Python

This example deploys a Python Lambda function that writes structured JSON logs and publishes three application metrics: request count, error count, and duration. Terraform creates a log group, IAM execution role, CloudWatch alarm, SNS topic, and dashboard.

Calling the function with a simulated HTTP status of 500 publishes an error metric. A CloudWatch alarm watches for errors and can send notifications after an SNS subscription is configured and confirmed.

1. Project Structure

Install Terraform, Python 3, and the AWS CLI. Configure AWS credentials with appropriate permissions for Lambda, IAM, CloudWatch, CloudWatch Logs, and SNS.

Create this project structure:

cloudwatch-demo/
  main.tf
  outputs.tf
  lambda/
    handler.py

The sample uses us-west-2 and a dedicated custom metric namespace. It deliberately uses direct PutMetricData calls to make the metric publication mechanism easy to understand. For high-throughput production applications, OpenTelemetry or CloudWatch Embedded Metric Format may provide a more suitable instrumentation pipeline.

2. Provision Lambda, Logs, Metrics, and Alarms

Create main.tf:

terraform {
  required_version = ">= 1.5.0"

  required_providers {
    aws = {
      source  = "hashicorp/aws"
      version = "~> 6.0"
    }

    archive = {
      source  = "hashicorp/archive"
      version = "~> 2.0"
    }
  }
}

provider "aws" {
  region = "us-west-2"
}

locals {
  service_name = "cloudwatch-demo"
  namespace    = "SysDesPro/Demo"
}

# -----------------------------------
# CloudWatch Logs
# -----------------------------------

resource "aws_cloudwatch_log_group" "app" {
  name              = "/aws/lambda/cloudwatch-demo-api"
  retention_in_days = 7
}

# -----------------------------------
# Lambda execution role
# -----------------------------------

data "aws_iam_policy_document" "lambda_trust" {
  statement {
    actions = ["sts:AssumeRole"]

    principals {
      type        = "Service"
      identifiers = ["lambda.amazonaws.com"]
    }
  }
}

resource "aws_iam_role" "lambda" {
  name_prefix = "cloudwatch-demo-"

  assume_role_policy = (
    data.aws_iam_policy_document.lambda_trust.json
  )
}

resource "aws_iam_role_policy_attachment" "logging" {
  role = aws_iam_role.lambda.name

  policy_arn = (
    "arn:aws:iam::aws:policy/service-role/AWSLambdaBasicExecutionRole"
  )
}

data "aws_iam_policy_document" "metrics" {
  statement {
    sid = "PublishApplicationMetrics"

    actions = [
      "cloudwatch:PutMetricData"
    ]

    resources = ["*"]

    condition {
      test     = "StringEquals"
      variable = "cloudwatch:namespace"
      values   = [local.namespace]
    }
  }
}

resource "aws_iam_role_policy" "metrics" {
  name   = "publish-demo-metrics"
  role   = aws_iam_role.lambda.id
  policy = data.aws_iam_policy_document.metrics.json
}

# -----------------------------------
# Lambda package and function
# -----------------------------------

data "archive_file" "lambda" {
  type        = "zip"
  source_file = "${path.module}/lambda/handler.py"
  output_path = "${path.module}/lambda.zip"
}

resource "aws_lambda_function" "app" {
  function_name = "cloudwatch-demo-api"

  role    = aws_iam_role.lambda.arn
  runtime = "python3.12"
  handler = "handler.lambda_handler"

  filename         = data.archive_file.lambda.output_path
  source_code_hash = data.archive_file.lambda.output_base64sha256

  timeout     = 15
  memory_size = 128

  environment {
    variables = {
      METRIC_NAMESPACE = local.namespace
      SERVICE_NAME     = local.service_name
    }
  }

  depends_on = [
    aws_cloudwatch_log_group.app,
    aws_iam_role_policy_attachment.logging,
    aws_iam_role_policy.metrics
  ]
}

# -----------------------------------
# Alarm notification topic
# -----------------------------------

resource "aws_sns_topic" "alerts" {
  name = "cloudwatch-demo-alerts"
}

# -----------------------------------
# Custom metric alarm
# -----------------------------------

resource "aws_cloudwatch_metric_alarm" "errors" {
  alarm_name = "cloudwatch-demo-errors"

  alarm_description = (
    "Triggers when the demo API reports errors."
  )

  namespace   = local.namespace
  metric_name = "ErrorCount"

  statistic = "Sum"
  period    = 60

  comparison_operator = "GreaterThanThreshold"
  threshold           = 0

  evaluation_periods  = 1
  datapoints_to_alarm = 1

  # ErrorCount is sparse: no value is published
  # in periods without an invocation.
  treat_missing_data = "notBreaching"

  dimensions = {
    Service = local.service_name
  }

  alarm_actions = [
    aws_sns_topic.alerts.arn
  ]

  ok_actions = [
    aws_sns_topic.alerts.arn
  ]
}

# -----------------------------------
# CloudWatch dashboard
# -----------------------------------

resource "aws_cloudwatch_dashboard" "app" {
  dashboard_name = "cloudwatch-demo"

  dashboard_body = jsonencode({
    widgets = [
      {
        type   = "metric"
        x      = 0
        y      = 0
        width  = 12
        height = 6

        properties = {
          title  = "Application Requests and Errors"
          region = "us-west-2"
          period = 60
          stat   = "Sum"

          metrics = [
            [
              local.namespace,
              "RequestCount",
              "Service",
              local.service_name
            ],
            [
              local.namespace,
              "ErrorCount",
              "Service",
              local.service_name
            ]
          ]
        }
      },
      {
        type   = "metric"
        x      = 0
        y      = 6
        width  = 12
        height = 6

        properties = {
          title  = "Application Duration"
          region = "us-west-2"
          period = 60
          stat   = "Average"

          metrics = [
            [
              local.namespace,
              "DurationMs",
              "Service",
              local.service_name
            ]
          ]
        }
      }
    ]
  })
}

The IAM policy allows cloudwatch:PutMetricData only for the custom namespace. That action uses a wildcard resource because it does not support ordinary metric-level resource ARN scoping; the namespace condition provides a narrower boundary.

The Lambda execution role also receives standard CloudWatch Logs permissions. The application does not need administrative access to CloudWatch or SNS.

The error alarm uses treat_missing_data = "notBreaching" because the example does not publish a value of zero for every idle minute. This is appropriate for demonstrating a sparse error metric, but it is not sufficient to monitor application heartbeat or telemetry delivery failure.

Create outputs.tf:

output "function_name" {
  value = aws_lambda_function.app.function_name
}

output "log_group_name" {
  value = aws_cloudwatch_log_group.app.name
}

output "alarm_name" {
  value = aws_cloudwatch_metric_alarm.errors.alarm_name
}

output "sns_topic_arn" {
  value = aws_sns_topic.alerts.arn
}

output "dashboard_name" {
  value = aws_cloudwatch_dashboard.app.dashboard_name
}

3. Instrument the Application with Python

Create lambda/handler.py:

import json
import os
import time
import uuid
from datetime import datetime, timezone

import boto3


NAMESPACE = os.environ["METRIC_NAMESPACE"]
SERVICE = os.environ["SERVICE_NAME"]

cloudwatch = boto3.client("cloudwatch")


def write_log(
    level: str,
    event_name: str,
    request_id: str,
    **fields,
) -> None:
    record = {
        "timestamp": datetime.now(
            timezone.utc
        ).isoformat(),
        "level": level,
        "service": SERVICE,
        "event": event_name,
        "request_id": request_id,
        **fields,
    }

    # Lambda forwards stdout to CloudWatch Logs.
    print(json.dumps(record))


def publish_metrics(
    duration_ms: float,
    status_code: int,
) -> None:
    dimensions = [
        {
            "Name": "Service",
            "Value": SERVICE,
        }
    ]

    metric_data = [
        {
            "MetricName": "RequestCount",
            "Dimensions": dimensions,
            "Value": 1,
            "Unit": "Count",
        },
        {
            "MetricName": "DurationMs",
            "Dimensions": dimensions,
            "Value": duration_ms,
            "Unit": "Milliseconds",
        },
    ]

    if status_code >= 500:
        metric_data.append({
            "MetricName": "ErrorCount",
            "Dimensions": dimensions,
            "Value": 1,
            "Unit": "Count",
        })

    cloudwatch.put_metric_data(
        Namespace=NAMESPACE,
        MetricData=metric_data,
    )


def lambda_handler(event, context):
    started = time.perf_counter()

    request_id = (
        getattr(context, "aws_request_id", None)
        or uuid.uuid4().hex
    )

    status_code = int(event.get("status_code", 200))

    if status_code not in (200, 500):
        raise ValueError(
            "Demo supports status_code 200 or 500"
        )

    # Simulate a small amount of application work.
    time.sleep(0.05)

    duration_ms = (
        time.perf_counter() - started
    ) * 1000

    level = (
        "ERROR" if status_code >= 500
        else "INFO"
    )

    write_log(
        level=level,
        event_name="request_completed",
        request_id=request_id,
        status_code=status_code,
        duration_ms=round(duration_ms, 2),
        route="/demo",
    )

    publish_metrics(
        duration_ms=duration_ms,
        status_code=status_code,
    )

    return {
        "statusCode": status_code,
        "body": json.dumps({
            "request_id": request_id,
            "status": (
                "error" if status_code >= 500
                else "ok"
            ),
        }),
    }

This Lambda function simulates successful and failed API requests. For every invocation, it writes a structured log and publishes request count and duration. For simulated 500 responses, it also publishes an error-count data point.

The demonstration uses one PutMetricData request per invocation. That is easy to understand, but at high request rates it creates API overhead, cost, and another possible source of request failures. In a production FastAPI or Lambda service, evaluate OpenTelemetry, Embedded Metric Format, batching, or suitable telemetry extensions.

Another limitation is that the sample publishes individual duration values, so the dashboard displays an average. A production latency dashboard should include tail latency using an appropriate metric representation that supports percentile calculations.

A failure while publishing telemetry also raises an exception in this simple example. Production instrumentation should normally avoid turning an otherwise successful business request into a failure solely because the monitoring backend is unavailable.

4. Deploy, Generate Telemetry, and Investigate

Run Terraform from the project directory:

# Configure your AWS credentials beforehand.
export AWS_REGION="us-west-2"

terraform init
terraform fmt
terraform validate
terraform plan
terraform apply

# Read the deployed resources.
export FUNCTION_NAME="$(
  terraform output -raw function_name
)"

export LOG_GROUP="$(
  terraform output -raw log_group_name
)"

export ALARM_NAME="$(
  terraform output -raw alarm_name
)"

# Invoke a successful request.
aws lambda invoke \
  --function-name "$FUNCTION_NAME" \
  --payload '{"status_code":200}' \
  --cli-binary-format raw-in-base64-out \
  /tmp/cloudwatch-success.json

cat /tmp/cloudwatch-success.json

# Invoke a simulated server failure.
aws lambda invoke \
  --function-name "$FUNCTION_NAME" \
  --payload '{"status_code":500}' \
  --cli-binary-format raw-in-base64-out \
  /tmp/cloudwatch-error.json

cat /tmp/cloudwatch-error.json

# View recent structured logs.
aws logs tail "$LOG_GROUP" --since 15m

# Inspect CloudWatch alarm state.
aws cloudwatch describe-alarms \
  --alarm-names "$ALARM_NAME" \
  --query "MetricAlarms[].{Name:AlarmName,State:StateValue,Reason:StateReason}"

# Inspect published custom metrics.
aws cloudwatch list-metrics \
  --namespace "SysDesPro/Demo"

# Delete the demonstration infrastructure.
terraform destroy

The first invocation produces a successful request metric and log entry. The second produces an error data point. Once CloudWatch has received and evaluated the metric, the alarm can transition to ALARM.

New custom metrics may take time to appear in metric listings, and alarm transitions are not instantaneous. A successful Lambda invocation also does not prove the SNS notification was delivered to a human recipient.

To receive actual email notifications, create an SNS email subscription for the Terraform-created topic and confirm it using the email sent by AWS:

# Substitute an email address you control.
export ALERT_EMAIL="you@example.com"

aws sns subscribe \
  --topic-arn "$(terraform output -raw sns_topic_arn)" \
  --protocol email \
  --notification-endpoint "$ALERT_EMAIL"

Replace the placeholder address with your own email, and confirm the subscription before testing notifications. The example creates only the topic by default; without a confirmed subscriber, alarm transitions will not produce an email notification.

The console provides a useful way to open the dashboard and inspect data over the same interval as the test invocations. For log analysis, open Logs Insights, select the log group, and filter for level = "ERROR".

After testing, destroy the disposable infrastructure and check that the Lambda function, alarm, log group, dashboard, and SNS topic have been removed. Do not run this cleanup against production monitoring resources.

Performance, Cost, and Production Practices

Observability systems have operating costs of their own. A poorly designed logging or metric strategy can generate substantial expenses while still failing to answer important incident questions.

CloudWatch deployments should define which signals are operationally valuable, how long they need to be retained, and how they will be used during incidents.

Retention, Cardinality, and Telemetry Cost

CloudWatch charges vary by telemetry type, usage, Region, and feature. Relevant cost drivers can include custom metric series, log ingestion and storage, Logs Insights queries, dashboards, alarms, high-resolution metrics, and advanced observability features.

For example, suppose an application publishes a metric with these dimensions:

Service = orders-api
Environment = production
Region = us-west-2
Route = /orders/{order_id}
StatusClass = 5xx

These dimensions are relatively bounded. But replacing the route template with the full URL or adding a unique request ID creates an enormous number of potential combinations.

For large applications, use logs and traces for unique identifiers, and metrics for aggregated labels with controlled cardinality.

Logging volume requires similar discipline. Writing full HTTP request and response bodies for every request can create excessive cost and may expose sensitive information. Use structured fields, sampling where appropriate, and documented retention periods.

CloudWatch Logs supports retention configuration and additional storage and query capabilities. Choose the correct log class and storage strategy based on access frequency, required features, and cost rather than assuming every log must remain in one expensive searchable location indefinitely.

Common Mistakes

Common CloudWatch mistakes include:

Monitoring only CPU and memory: Application failures go unnoticed while infrastructure looks healthy.
Using average latency alone: Slow-tail requests remain hidden.
Creating high-cardinality metric dimensions: Custom metric counts and costs grow unexpectedly.
Logging sensitive information: Credentials or private user data become accessible in log storage.
Using incorrect metric statistics: Request counts are averaged instead of summed.
Ignoring missing metric data: An absent heartbeat is mistaken for a healthy service.
Sending every infrastructure warning to on-call: Alert fatigue obscures user-impacting incidents.
Assuming an alarm automatically sends email: SNS has no confirmed subscriber or lacks required permissions.
Ignoring application request IDs: Logs cannot be correlated across distributed services.
Collecting unlimited logs: Ingestion and retention charges grow without clear operational value.
Making telemetry synchronous and mandatory: A monitoring failure becomes an application outage.
Confusing log delivery with business completion: A logged event is treated as proof that the transaction succeeded.

Production Checklist

Before relying on CloudWatch for critical applications, verify that:

Important services expose traffic, error, latency, and saturation metrics.
Custom metrics use documented names and bounded dimensions.
Critical workflows have business-level success and failure signals.
Application logs are structured and include correlation identifiers.
Logs exclude passwords, access tokens, and unnecessary sensitive data.
Log retention matches operational and compliance requirements.
Alarms use appropriate statistics, periods, and evaluation thresholds.
Missing-data behavior is deliberate for every important alarm.
Notification destinations have confirmed and tested subscriptions.
Paging alerts reflect actual service degradation or imminent operational risk.
Dashboards support real incident investigation, not only resource inventory.
Tracing context is propagated across service and queue boundaries.
Telemetry ingestion, query, and retention costs are monitored.
Failures in the telemetry pipeline do not automatically bring down the application.
Incident runbooks explain how to investigate and respond to significant alarms.

Frequently Asked Questions

CloudWatch covers infrastructure monitoring, logs, custom metrics, alarms, and application performance capabilities. The following questions clarify its most common use cases and limitations.

What Is Amazon CloudWatch Used For?

Amazon CloudWatch is used to monitor AWS resources and applications. It collects metrics and logs, supports dashboards and alarms, and provides integrations for application performance monitoring and operational troubleshooting.

What Is the Difference Between CloudWatch Logs and Metrics?

CloudWatch metrics are numerical time series intended for aggregation, visualization, and alarms. CloudWatch Logs stores timestamped events containing execution details and contextual information.

Metrics are usually better for detecting that a problem exists. Logs are useful for investigating what happened in individual operations.

What Is the Difference Between CloudWatch and CloudTrail?

CloudWatch focuses on operational monitoring and observability. CloudTrail records supported AWS account and API activity for auditing, security analysis, and investigation.

CloudTrail records can be delivered to supported destinations and analyzed alongside CloudWatch operational signals, but the services have different primary responsibilities.

Can Applications Send Custom Metrics to CloudWatch?

Yes. Applications can publish custom metrics using APIs such as PutMetricData, CloudWatch Embedded Metric Format, or supported telemetry pipelines.

Custom metrics allow engineers to measure business outcomes and application behavior not covered by default AWS service metrics.

Can CloudWatch Automatically Notify Me About Failures?

Yes. CloudWatch alarms can invoke configured actions, including SNS notifications and supported automated responses.

The alarm needs an appropriate threshold and notification configuration. For email alerts through SNS, the subscription must be confirmed before messages can reach the intended recipient.

Is Amazon CloudWatch Free?

Amazon CloudWatch has applicable free allowances or credits for eligible accounts and supported usage. Costs beyond those allowances depend on metrics, log ingestion and retention, dashboards, alarms, queries, and optional features.

High-cardinality custom metrics and excessive log ingestion are common causes of unexpectedly high observability costs.

Conclusion

Amazon CloudWatch is AWS's integrated monitoring and observability service for infrastructure and applications. It helps engineering teams track operational metrics, query logs, monitor dependencies, create dashboards, and react to problems through alarms and supported automation.

CloudWatch becomes most valuable when telemetry is designed around real operational questions. Metrics should reveal service health, logs should explain failures, and distributed tracing should connect related operations. Alerting should emphasize meaningful customer impact rather than every temporary resource fluctuation.

Production engineering also requires careful attention to metric cardinality, missing data, logging security, retention, and cost. A large volume of telemetry is not a substitute for a small set of trustworthy, actionable signals.

Key Takeaway: CloudWatch provides the tools to observe AWS workloads, but effective monitoring requires deliberate instrumentation and alert design. Measure user-visible outcomes, correlate metrics with logs and traces, and build alarms that help engineers respond to actual failures.

Table of Contents