Amazon ECS Best Practices for Production

Amazon ECS production best practices are less about memorizing service options and more about designing reliable container workloads. A production ECS service needs safe deployments, private networking, least-privilege IAM, useful observability, correct scaling signals, and predictable failure handling.

ECS removes much of the orchestration burden, but it does not remove system design responsibility. Tasks still need to be stateless, health checks must be meaningful, containers need proper CPU and memory limits, secrets must be handled safely, and deployment failures must be recoverable.

Table of Contents

• Production ECS Model
• Task Design
  • Stateless Tasks
  • Immutable Images
  • CPU and Memory Sizing
• Service Reliability
  • Desired Count
  • Health Checks
  • Graceful Shutdown
• Networking Best Practices
  • Private Tasks
  • Security Group Boundaries
  • Outbound Connectivity
• Security Best Practices
  • Task Roles
  • Secrets Management
  • Container Permissions
• Deployment Best Practices
  • Rolling Deployments
  • Deployment Circuit Breaker
  • Backward-Compatible Changes
• Scaling Best Practices
  • API Scaling
  • Worker Scaling
  • Scale-In Safety
• Observability Best Practices
  • Logs
  • Metrics
  • Alarms
• Ready-to-Use Example
  • CloudFormation Example
  • Deployment Script
• Common Mistakes
• Production Checklist
• Conclusion
• More Articles to Read

Production ECS Model

A production ECS service should be treated as a complete system, not just a container running somewhere on AWS. The task definition, service settings, load balancer, subnets, security groups, IAM roles, logs, metrics, and deployment behavior all affect reliability.

The strongest ECS architectures assume that tasks are disposable. A task can stop during deployment, health replacement, scaling, infrastructure maintenance, or application failure.

Production ECS service

Container image
  -> Task definition
  -> ECS service
  -> Private subnets
  -> Security groups
  -> Load balancer
  -> Logs, metrics, alarms
  -> Auto scaling
  -> Safe deployments

Task Design

ECS tasks are runtime units, not permanent servers. Good task design makes replacement safe and predictable.

The most important task-level practices are statelessness, immutable images, and measured resource sizing.

Stateless Tasks

ECS tasks should usually be stateless. Durable data should not live inside the container filesystem because the task can be replaced at any time.

Rule of Thumb: if losing a task loses important data, the task is holding state that belongs somewhere else.

Immutable Images

Production deployments should use immutable image tags. Tags such as Git SHA, release version, or build number make deployments traceable and rollbacks predictable.

Good:
  orders-api:8f31c2a
  orders-api:v1.42.0
  orders-api:2026-07-08-1432

Bad:
  orders-api:latest
  orders-api:prod
  orders-api:stable

Important: the same image tag should not point to different image contents over time. Mutable tags make incident debugging and rollback much harder.

CPU and Memory Sizing

CPU and memory values affect cost, placement, performance, and failure behavior. A memory limit that is too low can kill a task. CPU that is too low can create latency during traffic spikes.

Production Note: task size should come from load testing and production metrics, not from initial guesses.

Service Reliability

ECS services keep tasks running, but service reliability depends on desired count, health checks, deployment settings, and application behavior.

A reliable ECS service should survive normal task replacement without user-visible downtime.

Desired Count

Desired count defines how many tasks ECS should keep running. For production APIs, one task is usually not enough because deployments and failures need redundancy.

Health Checks

Health checks decide whether a task is safe to receive traffic. A task can be running but not ready to serve requests.

GET /health

Expected:
  200 OK
  {"status":"ok"}

Important: health checks should be lightweight and reliable. Expensive database queries or slow external API calls can turn health checks into a failure source.

Graceful Shutdown

During deployments and scale-in events, ECS stops tasks. The application should stop accepting new work, finish in-flight work, and exit cleanly.

Task shutdown flow:
  ECS starts draining task
  -> Load balancer stops sending new requests
  -> Application receives termination signal
  -> Application finishes in-flight work
  -> Application exits cleanly

Production Note: graceful shutdown is critical for APIs, queue workers, payment workflows, file processing, and any non-idempotent operation.

Networking Best Practices

ECS networking should expose only the traffic paths that the application actually needs. Public traffic should normally reach a load balancer, not ECS tasks directly.

Strong network design reduces attack surface and makes service communication easier to reason about.

Private Tasks

ECS tasks should usually run in private subnets. The Application Load Balancer can run in public subnets and forward requests to private task IPs.

Internet
  -> Public ALB
  -> ECS tasks in private subnets
  -> Database / cache / queue

Rule of Thumb: if users need public access, expose the load balancer. If tasks need outbound internet access, use NAT gateways or VPC endpoints.

Security Group Boundaries

Security groups should describe real application communication paths. Avoid broad access rules that allow services to talk to everything.

Outbound Connectivity

Private ECS tasks often need outbound access to pull images, write logs, fetch secrets, call external APIs, or reach AWS services.

Trade-off: NAT gateways are simple but can become expensive. VPC endpoints reduce public internet dependency but add configuration.

Security Best Practices

ECS security depends on IAM boundaries, network boundaries, secret handling, and container runtime behavior.

The safest ECS systems use least privilege at every layer.

Task Roles

ECS separates task execution role and task role. These roles should not be mixed.

Important: application permissions belong in the task role. The execution role should not become a broad application role.

Secrets Management

Secrets should not be baked into images, committed to source control, or stored in plain environment files. Use AWS Secrets Manager or SSM Parameter Store.

{
  "secrets": [
    {
      "name": "DATABASE_URL",
      "valueFrom": "arn:aws:ssm:us-east-1:123456789012:parameter/prod/orders/database-url"
    }
  ]
}

Secret access should be scoped to the specific task role and environment. A staging service should not be able to read production secrets.

Container Permissions

Container images should run with the minimum permissions needed. Avoid unnecessary packages, root execution when possible, writable filesystem assumptions, and broad runtime privileges.

• Use small base images to reduce attack surface.
• Run as non-root when the application supports it.
• Avoid baking secrets into images or layers.
• Scan images as part of CI/CD.
• Remove unused tools that are not needed at runtime.

Deployment Best Practices

ECS deployments should be predictable, traceable, and recoverable. A deployment should not depend on manual console steps or mutable image tags.

The release process should create a new image, register a new task definition revision, update the service, wait for stability, and stop if the service does not become healthy.

Rolling Deployments

Rolling deployments are a strong default for most ECS services. ECS starts new tasks, waits for them to become healthy, and stops old tasks according to deployment settings.

Revision 21 running
  -> Deploy revision 22
  -> Start new tasks
  -> Wait for healthy targets
  -> Stop old tasks
  -> Service reaches steady state

Deployment Circuit Breaker

The deployment circuit breaker helps detect deployments that cannot reach a stable healthy state. With rollback enabled, ECS can return to the previous working revision.

Production Note: circuit breaker rollback is only as good as the health signals. Weak health checks can allow broken versions to pass.

Backward-Compatible Changes

Rolling deployments may run old and new task versions at the same time. Database migrations, message formats, API contracts, and cache keys should be compatible during that overlap.

Scaling Best Practices

ECS service scaling changes the number of running tasks. Scaling should follow the real workload bottleneck, not only default CPU metrics.

APIs and background workers usually need different scaling signals.

API Scaling

APIs should scale to protect response latency and availability. ALB request count per target is often a better signal than CPU for HTTP services.

Worker Scaling

Workers should scale based on backlog, processing delay, and job duration. CPU can be misleading if workers wait on databases, third-party APIs, or file systems.

Key Point: APIs scale to protect latency. Workers scale to protect backlog age.

Scale-In Safety

Scale-out should usually happen faster than scale-in. Adding capacity protects availability. Removing capacity too quickly can cause oscillation and repeated scaling events.

Rule of Thumb: use shorter scale-out cooldowns and longer scale-in cooldowns.

Observability Best Practices

A production ECS service should be observable from multiple angles: logs, metrics, alarms, deployment events, task restarts, and application-level signals.

Container-level metrics are useful, but they are not enough. Application behavior must be visible.

Logs

Containers should write logs to stdout and stderr. ECS can route those logs to CloudWatch Logs through the awslogs driver.

{
  "logConfiguration": {
    "logDriver": "awslogs",
    "options": {
      "awslogs-group": "/ecs/orders-api",
      "awslogs-region": "us-east-1",
      "awslogs-stream-prefix": "ecs"
    }
  }
}

Logs should include request IDs, job IDs, user-safe error messages, deployment version, and important business events. Secrets, passwords, tokens, and sensitive payloads should not be logged.

Metrics

Metrics should answer whether the service is healthy, fast, and keeping up with work.

Alarms

Alarms should represent user impact or operational risk. CPU alarms alone are not enough.

• API latency is high.
• 5xx error rate increased.
• ECS deployment failed.
• Task restart count increased.
• Worker queue age is growing.
• DLQ has messages.
• Memory utilization is near limit.

Ready-to-Use Example

This example shows a production-oriented ECS Fargate service with private networking, immutable image input, CloudWatch Logs, deployment circuit breaker, and safe rolling deployment settings.

The example assumes the VPC, private subnets, ALB target group, security group, task execution role, and task role already exist.

CloudFormation Example

The CloudFormation snippet below focuses on ECS resources that are commonly needed for a production API service.

AWSTemplateFormatVersion: "2010-09-09"
Description: Production-oriented ECS Fargate service example

Parameters:
  ClusterName:
    Type: String
    Description: Existing ECS cluster name.

  ServiceName:
    Type: String
    Default: orders-api
    Description: ECS service name.

  ContainerImage:
    Type: String
    Description: Immutable image URI, usually tagged with Git SHA.

  PrivateSubnetIds:
    Type: List<AWS::EC2::Subnet::Id>
    Description: Private subnets where ECS tasks should run.

  TaskSecurityGroupId:
    Type: AWS::EC2::SecurityGroup::Id
    Description: Security group that allows inbound traffic only from the ALB.

  TargetGroupArn:
    Type: String
    Description: ALB target group used to route traffic to ECS tasks.

  TaskExecutionRoleArn:
    Type: String
    Description: IAM role used by ECS to pull images, fetch secrets, and write logs.

  TaskRoleArn:
    Type: String
    Description: IAM role used by application code inside the container.

Resources:
  ApiLogGroup:
    Type: AWS::Logs::LogGroup
    Properties:
      # Keep logs long enough for debugging while controlling storage cost.
      LogGroupName: !Sub "/ecs/${ServiceName}"
      RetentionInDays: 30

  ApiTaskDefinition:
    Type: AWS::ECS::TaskDefinition
    Properties:
      # Family groups task definition revisions for this service.
      Family: !Ref ServiceName

      # awsvpc gives each task its own private IP and security group boundary.
      NetworkMode: awsvpc

      # Fargate removes EC2 container instance management.
      RequiresCompatibilities:
        - FARGATE

      # Tune these values from load testing and production metrics.
      Cpu: "512"
      Memory: "1024"

      # Execution role is for ECS platform actions, not application permissions.
      ExecutionRoleArn: !Ref TaskExecutionRoleArn

      # Task role is what application code uses to access AWS services.
      TaskRoleArn: !Ref TaskRoleArn

      ContainerDefinitions:
        - Name: api
          Image: !Ref ContainerImage
          Essential: true

          # This port must match the ALB target group configuration.
          PortMappings:
            - ContainerPort: 8000
              Protocol: tcp

          # Non-sensitive runtime configuration can use environment variables.
          Environment:
            - Name: APP_ENV
              Value: production

          # Send application logs to CloudWatch Logs through the awslogs driver.
          LogConfiguration:
            LogDriver: awslogs
            Options:
              awslogs-group: !Ref ApiLogGroup
              awslogs-region: !Ref AWS::Region
              awslogs-stream-prefix: ecs

  ApiService:
    Type: AWS::ECS::Service
    Properties:
      ServiceName: !Ref ServiceName
      Cluster: !Ref ClusterName
      LaunchType: FARGATE
      TaskDefinition: !Ref ApiTaskDefinition

      # Run at least two tasks for basic production redundancy.
      DesiredCount: 2

      # Give the app time to start before target group health checks count.
      HealthCheckGracePeriodSeconds: 60

      DeploymentConfiguration:
        # Keep all desired tasks healthy during rolling deployments.
        MinimumHealthyPercent: 100

        # Allow ECS to temporarily run extra tasks during deployment.
        MaximumPercent: 200

        DeploymentCircuitBreaker:
          # Detect deployments that cannot reach a healthy steady state.
          Enable: true

          # Automatically roll back to the previous working task definition.
          Rollback: true

      NetworkConfiguration:
        AwsvpcConfiguration:
          # Keep tasks private. Public traffic should enter through the ALB.
          AssignPublicIp: DISABLED

          # Use private subnets across multiple Availability Zones.
          Subnets: !Ref PrivateSubnetIds

          # This security group should allow inbound traffic only from the ALB.
          SecurityGroups:
            - !Ref TaskSecurityGroupId

      LoadBalancers:
        - ContainerName: api
          ContainerPort: 8000
          TargetGroupArn: !Ref TargetGroupArn

Production Note: this example intentionally receives existing network and IAM resources as parameters. In real systems, networking, IAM, ALB, and ECS services are often managed as separate infrastructure modules.

Deployment Script

A safe ECS deployment script should build an immutable image, push it to ECR, deploy the infrastructure update, and wait until the ECS service reaches steady state.

#!/usr/bin/env bash
set -euo pipefail

AWS_REGION="us-east-1"
AWS_ACCOUNT_ID="123456789012"
ECR_REPOSITORY="orders-api"
STACK_NAME="orders-api-prod"
CLUSTER_NAME="prod-ecs"
SERVICE_NAME="orders-api"

# Use the Git commit SHA as an immutable tag for auditability and rollback.
IMAGE_TAG="$(git rev-parse --short HEAD)"
IMAGE_URI="${AWS_ACCOUNT_ID}.dkr.ecr.${AWS_REGION}.amazonaws.com/${ECR_REPOSITORY}:${IMAGE_TAG}"

echo "Building image: ${IMAGE_URI}"

# Authenticate Docker to Amazon ECR in the target account and region.
aws ecr get-login-password --region "${AWS_REGION}" \
  | docker login \
      --username AWS \
      --password-stdin "${AWS_ACCOUNT_ID}.dkr.ecr.${AWS_REGION}.amazonaws.com"

# Build the image from the current repository state.
docker build -t "${IMAGE_URI}" .

# Push the immutable image so ECS tasks can pull it during deployment.
docker push "${IMAGE_URI}"

echo "Deploying CloudFormation stack: ${STACK_NAME}"

# Updating ContainerImage creates a new task definition revision.
aws cloudformation deploy \
  --region "${AWS_REGION}" \
  --stack-name "${STACK_NAME}" \
  --template-file ecs-production-service.yaml \
  --capabilities CAPABILITY_NAMED_IAM \
  --parameter-overrides \
      ClusterName="${CLUSTER_NAME}" \
      ServiceName="${SERVICE_NAME}" \
      ContainerImage="${IMAGE_URI}"

echo "Waiting for ECS service to stabilize..."

# Stop the pipeline if ECS cannot reach steady state.
aws ecs wait services-stable \
  --region "${AWS_REGION}" \
  --cluster "${CLUSTER_NAME}" \
  --services "${SERVICE_NAME}"

echo "Deployment completed successfully: ${IMAGE_URI}"

Important: a deployment pipeline should fail when the service does not stabilize. Continuing after a partial deployment hides production risk.

Common Mistakes

Most ECS production mistakes come from treating containers as isolated deployment units instead of part of a larger production system.

• Using latest image tags instead of immutable release tags.
• Running only one task for a production API.
• Putting ECS tasks in public subnets when only the ALB should be public.
• Using broad security group rules instead of service-to-service access rules.
• Confusing task execution role and task role.
• Storing secrets in container images or plain environment files.
• Keeping durable state inside containers.
• Skipping graceful shutdown and breaking in-flight requests or jobs.
• Scaling workers by CPU only while queue age grows.
• Using weak health checks that let broken tasks receive traffic.
• Using heavy health checks that fail during temporary dependency issues.
• Ignoring rollback behavior until a production incident.

Production Checklist

A production ECS service should be safe to deploy, easy to observe, hard to expose accidentally, and designed for task replacement.

• Use immutable image tags tied to Git commits or release versions.
• Keep tasks stateless and store durable data externally.
• Run production APIs with at least two tasks.
• Place ECS tasks in private subnets by default.
• Expose HTTP traffic through an ALB or another controlled ingress layer.
• Use security-group references instead of broad CIDR rules where possible.
• Separate execution role and task role permissions.
• Store secrets in Secrets Manager or SSM Parameter Store.
• Configure CloudWatch Logs for every container.
• Use meaningful application metrics, not only CPU and memory.
• Alert on latency, 5xx rate, task restarts, deployment failures, and queue age.
• Enable deployment circuit breaker and rollback.
• Implement graceful shutdown for APIs and workers.
• Scale APIs by request pressure or resource limits.
• Scale workers by queue backlog or oldest message age.
• Use infrastructure as code for repeatable environments.
• Test deployment and rollback behavior before relying on it.

Conclusion

Amazon ECS can run production services reliably, but reliability comes from the surrounding engineering decisions. Task design, networking, IAM, secrets, deployments, scaling, logs, metrics, and alarms all matter.

The strongest ECS systems treat tasks as disposable, keep workloads private, expose traffic through controlled entry points, use least-privilege permissions, deploy immutable images, scale from meaningful signals, and make failures visible through observability.

Key Takeaway: ECS production best practices are about predictable operations: stateless tasks, private networking, safe deployments, least-privilege IAM, useful observability, and scaling policies based on real workload pressure.

More Articles to Read

Continue with ECS articles that explain the runtime model, compute choices, networking, workload patterns, deployment behavior, and scaling.

• Amazon ECS Task Definitions, Services, and Deployments Explained
• AWS Fargate vs EC2 for ECS: Architecture Trade-Offs
• Amazon ECS Networking Explained: VPC, ALB, Subnets, and Security Groups
• Running Production APIs and Background Workers on Amazon ECS
• Amazon ECS Deployment Strategies and Auto Scaling