Running Production APIs and Background Workers on Amazon ECS

Amazon ECS is often used for two common production workload types: HTTP APIs and background workers. Both run as containers, but they have different traffic patterns, scaling signals, failure modes, and operational requirements.

A production API usually receives requests through an Application Load Balancer and must optimize for latency, availability, and safe deployments. A background worker usually consumes messages from a queue and must optimize for throughput, retries, idempotency, and backlog control.

Table of Contents

• ECS Workload Model
• Production API Architecture
  • API Request Flow
  • API Task Design
  • API Scaling Signals
• Background Worker Architecture
  • Worker Processing Flow
  • Worker Task Design
  • Worker Scaling Signals
• API vs Worker Trade-Offs
  • Latency vs Throughput
  • Synchronous vs Asynchronous Failure
  • Capacity Planning
• Shared Production Concerns
  • Stateless Containers
  • Configuration and Secrets
  • Logs and Metrics
  • Graceful Shutdown
• Combined API and Worker Pattern
  • When to Split Work
  • Queue-Based Decoupling
• Ready-to-Use Example
  • CloudFormation Example
  • Deployment Script
• Common Mistakes
• Production Checklist
• Conclusion
• More Articles to Read

ECS Workload Model

ECS runs containers as tasks. A long-running workload is usually managed by an ECS service, which keeps the desired number of tasks running and replaces failed tasks.

APIs and workers both use this model, but the service behavior is interpreted differently. For APIs, task count represents request-serving capacity. For workers, task count represents processing capacity.

Key Point: APIs and workers may use the same container platform, but they should not be designed with the same operational assumptions.

Production API Architecture

A production API on ECS is usually deployed as an ECS service behind an Application Load Balancer. The ALB provides a stable entry point, checks task health, and routes traffic only to healthy targets.

ECS tasks should usually run in private subnets. Public traffic should terminate at the load balancer, not directly at task IP addresses.

API Request Flow

The common request path starts with DNS, reaches the load balancer, and then forwards traffic to healthy ECS task IPs.

Client
  -> Route 53
  -> Application Load Balancer
  -> Target Group
  -> ECS Service: orders-api
       -> Task A
       -> Task B
       -> Task C
  -> Database / Cache / Queue

The application should not depend on a specific task IP. Tasks are replaceable and can change during deployments, health recovery, and scaling.

API Task Design

API tasks should be stateless. Request state, uploaded files, sessions, and durable records should live outside the container.

Production Note: stateless APIs are easier to scale, replace, deploy, and recover. Stateful containers make deployments fragile.

API Scaling Signals

API scaling should be tied to user-facing pressure. CPU and memory are useful, but they may not be the best primary signals.

Rule of Thumb: scale APIs by request pressure or resource pressure, but alert on latency and error rate.

Background Worker Architecture

A background worker on ECS usually runs as an ECS service without public ingress. It consumes messages from a queue, processes work, writes results, and acknowledges messages after successful completion.

Workers are often used to keep APIs fast. Instead of doing expensive work inside a request, the API places a message on a queue and returns quickly.

Worker Processing Flow

The common worker flow starts with a queue and ends when the worker successfully acknowledges or deletes the message.

API Service
  -> Send message to SQS

SQS Queue
  -> ECS Worker Service
       -> Worker Task A
       -> Worker Task B
  -> Process job
  -> Write result
  -> Delete message from queue

If processing fails, the message should become visible again or eventually move to a dead-letter queue. This prevents silent data loss.

Worker Task Design

Worker tasks should be designed for failure. A task may stop during deployment, scaling, host replacement, memory pressure, or application crash.

• Idempotency prevents duplicate processing from corrupting data.
• Visibility timeout gives a worker time to finish before another worker receives the same message.
• Dead-letter queues isolate repeatedly failing messages.
• Batch size controls throughput and failure blast radius.
• Backoff prevents retry storms during dependency outages.

Important: queue workers should acknowledge messages only after the work is safely completed.

Worker Scaling Signals

Worker scaling should usually follow backlog, not CPU alone. A worker fleet can have low CPU and still be behind if messages are slow because of external dependencies.

Production Note: the best worker metric is often oldest message age, because it shows how long users or downstream systems wait for work to complete.

API vs Worker Trade-Offs

APIs and workers solve different production problems. APIs provide synchronous request/response behavior. Workers provide asynchronous processing and decoupling.

A strong ECS architecture often uses both: APIs for fast user-facing interactions and workers for slow, expensive, or retryable work.

Latency vs Throughput

APIs are latency-sensitive. A slow API response directly affects clients. Workers are throughput-sensitive. A worker system can tolerate delayed processing if backlog remains controlled.

Synchronous vs Asynchronous Failure

API failures are usually visible immediately as 4xx, 5xx, timeout, or degraded latency. Worker failures may be hidden until backlog grows or messages land in a dead-letter queue.

This changes monitoring strategy. APIs need latency and error alerts. Workers need backlog, oldest message age, retry, and DLQ alerts.

Capacity Planning

API capacity planning starts from traffic rate, latency target, and per-task throughput. Worker capacity planning starts from message arrival rate, processing time, and acceptable delay.

API capacity:
  requests per second
  / safe requests per task
  = required task count

Worker capacity:
  messages per minute * average processing time
  / acceptable backlog window
  = required worker capacity

Key Point: APIs scale to protect response time. Workers scale to protect backlog age.

Shared Production Concerns

APIs and workers have different workload shapes, but they share several ECS production concerns. Both need stateless containers, safe configuration, centralized logs, metrics, and graceful shutdown.

These concerns should be designed before traffic grows, not after the first incident.

Stateless Containers

ECS tasks should be disposable. A task may stop during deployment, health replacement, scaling, or infrastructure changes.

Durable state should be stored outside the task.

Configuration and Secrets

Runtime configuration should be explicit in the task definition. Non-sensitive values can use environment variables. Sensitive values should use Secrets Manager or SSM Parameter Store.

{
  "environment": [
    {
      "name": "APP_ENV",
      "value": "production"
    }
  ],
  "secrets": [
    {
      "name": "DATABASE_URL",
      "valueFrom": "arn:aws:ssm:us-east-1:123456789012:parameter/orders/database-url"
    }
  ]
}

Important: secrets should not be baked into container images or stored in plain environment files committed to source control.

Logs and Metrics

Logs should go to a centralized destination such as CloudWatch Logs. Metrics should describe application behavior, not only container resource usage.

Logs should include request IDs or job IDs. Without correlation IDs, debugging distributed ECS workloads becomes much harder.

Graceful Shutdown

ECS may stop tasks during deployments, scale-in events, or health recovery. Applications should handle shutdown signals correctly.

API shutdown:
  Stop accepting new requests
  Finish in-flight requests
  Close connections
  Exit cleanly

Worker shutdown:
  Stop polling new messages
  Finish or safely release current message
  Exit cleanly

Production Note: graceful shutdown is more than clean process exit. It prevents partial writes, duplicate work, failed requests, and message loss.

Combined API and Worker Pattern

Many production systems use ECS APIs and ECS workers together. The API handles client requests and delegates slow work to a queue. The worker processes that queue independently.

This pattern improves latency, isolates failures, and allows the API and worker fleets to scale independently.

When to Split Work

Work should move from an API request to a background worker when it is slow, retryable, resource-heavy, or not required before responding to the client.

Queue-Based Decoupling

A queue creates a buffer between the API and the worker. If workers slow down, the API can continue accepting requests until backlog limits are reached.

Client
  -> API Service
  -> Store request metadata
  -> Send message to SQS
  -> Return 202 Accepted

Worker Service
  -> Poll SQS
  -> Process message
  -> Update status
  -> Delete message

Trade-off: queue-based architecture improves resilience but introduces eventual consistency. Clients may need a status endpoint, webhook, or notification mechanism.

Ready-to-Use Example

A practical ECS setup usually contains two services: one API service behind an Application Load Balancer and one worker service consuming messages from SQS. The API is optimized for request latency, while the worker is optimized for reliable asynchronous processing.

The example below shows the core CloudFormation resources for this pattern. It intentionally focuses on ECS-specific resources and assumes the VPC, subnets, load balancer, target group, SQS queue, IAM roles, and security groups already exist.

CloudFormation Example

The API service uses an ECS service with a target group. The worker service runs without a load balancer and receives queue configuration through environment variables.

AWSTemplateFormatVersion: "2010-09-09"
Description: ECS API and worker services example

Parameters:
  ClusterName:
    Type: String
  ApiServiceName:
    Type: String
    Default: orders-api
  WorkerServiceName:
    Type: String
    Default: orders-worker
  ContainerImage:
    Type: String
  PrivateSubnetIds:
    Type: List
  ApiSecurityGroupId:
    Type: AWS::EC2::SecurityGroup::Id
  WorkerSecurityGroupId:
    Type: AWS::EC2::SecurityGroup::Id
  ApiTargetGroupArn:
    Type: String
  TaskExecutionRoleArn:
    Type: String
  ApiTaskRoleArn:
    Type: String
  WorkerTaskRoleArn:
    Type: String
  QueueUrl:
    Type: String

Resources:
  ApiLogGroup:
    Type: AWS::Logs::LogGroup
    Properties:
      LogGroupName: !Sub "/ecs/${ApiServiceName}"
      RetentionInDays: 14

  WorkerLogGroup:
    Type: AWS::Logs::LogGroup
    Properties:
      LogGroupName: !Sub "/ecs/${WorkerServiceName}"
      RetentionInDays: 14

  ApiTaskDefinition:
    Type: AWS::ECS::TaskDefinition
    Properties:
      Family: !Ref ApiServiceName
      NetworkMode: awsvpc
      RequiresCompatibilities:
        - FARGATE
      Cpu: "512"
      Memory: "1024"
      ExecutionRoleArn: !Ref TaskExecutionRoleArn
      TaskRoleArn: !Ref ApiTaskRoleArn
      ContainerDefinitions:
        - Name: api
          Image: !Ref ContainerImage
          Essential: true
          PortMappings:
            - ContainerPort: 8000
              Protocol: tcp
          Environment:
            - Name: APP_MODE
              Value: api
          LogConfiguration:
            LogDriver: awslogs
            Options:
              awslogs-group: !Ref ApiLogGroup
              awslogs-region: !Ref AWS::Region
              awslogs-stream-prefix: ecs

  WorkerTaskDefinition:
    Type: AWS::ECS::TaskDefinition
    Properties:
      Family: !Ref WorkerServiceName
      NetworkMode: awsvpc
      RequiresCompatibilities:
        - FARGATE
      Cpu: "512"
      Memory: "1024"
      ExecutionRoleArn: !Ref TaskExecutionRoleArn
      TaskRoleArn: !Ref WorkerTaskRoleArn
      ContainerDefinitions:
        - Name: worker
          Image: !Ref ContainerImage
          Essential: true
          Environment:
            - Name: APP_MODE
              Value: worker
            - Name: QUEUE_URL
              Value: !Ref QueueUrl
          LogConfiguration:
            LogDriver: awslogs
            Options:
              awslogs-group: !Ref WorkerLogGroup
              awslogs-region: !Ref AWS::Region
              awslogs-stream-prefix: ecs

  ApiService:
    Type: AWS::ECS::Service
    DependsOn:
      - ApiTaskDefinition
    Properties:
      ServiceName: !Ref ApiServiceName
      Cluster: !Ref ClusterName
      LaunchType: FARGATE
      DesiredCount: 2
      TaskDefinition: !Ref ApiTaskDefinition
      HealthCheckGracePeriodSeconds: 60
      DeploymentConfiguration:
        MinimumHealthyPercent: 100
        MaximumPercent: 200
      NetworkConfiguration:
        AwsvpcConfiguration:
          AssignPublicIp: DISABLED
          Subnets: !Ref PrivateSubnetIds
          SecurityGroups:
            - !Ref ApiSecurityGroupId
      LoadBalancers:
        - ContainerName: api
          ContainerPort: 8000
          TargetGroupArn: !Ref ApiTargetGroupArn

  WorkerService:
    Type: AWS::ECS::Service
    DependsOn:
      - WorkerTaskDefinition
    Properties:
      ServiceName: !Ref WorkerServiceName
      Cluster: !Ref ClusterName
      LaunchType: FARGATE
      DesiredCount: 2
      TaskDefinition: !Ref WorkerTaskDefinition
      DeploymentConfiguration:
        MinimumHealthyPercent: 50
        MaximumPercent: 200
      NetworkConfiguration:
        AwsvpcConfiguration:
          AssignPublicIp: DISABLED
          Subnets: !Ref PrivateSubnetIds
          SecurityGroups:
            - !Ref WorkerSecurityGroupId

Production Note: the API service has a load balancer because it receives HTTP traffic. The worker service does not need a load balancer because it consumes messages from SQS.

Deployment Script

A deployment should build the image, push it to ECR, update the CloudFormation stack with the new image tag, and wait until ECS stabilizes.

#!/usr/bin/env bash
set -euo pipefail

AWS_REGION="us-east-1"
AWS_ACCOUNT_ID="123456789012"
ECR_REPOSITORY="orders-service"
STACK_NAME="orders-ecs"
CLUSTER_NAME="prod-ecs"
API_SERVICE_NAME="orders-api"
WORKER_SERVICE_NAME="orders-worker"

IMAGE_TAG="$(git rev-parse --short HEAD)"
IMAGE_URI="${AWS_ACCOUNT_ID}.dkr.ecr.${AWS_REGION}.amazonaws.com/${ECR_REPOSITORY}:${IMAGE_TAG}"

echo "Building image: ${IMAGE_URI}"

aws ecr get-login-password --region "${AWS_REGION}" \
  | docker login \
      --username AWS \
      --password-stdin "${AWS_ACCOUNT_ID}.dkr.ecr.${AWS_REGION}.amazonaws.com"

docker build -t "${IMAGE_URI}" .
docker push "${IMAGE_URI}"

echo "Updating CloudFormation stack: ${STACK_NAME}"

aws cloudformation deploy \
  --region "${AWS_REGION}" \
  --stack-name "${STACK_NAME}" \
  --template-file ecs-api-worker.yaml \
  --capabilities CAPABILITY_NAMED_IAM \
  --parameter-overrides \
      ClusterName="${CLUSTER_NAME}" \
      ContainerImage="${IMAGE_URI}"

echo "Waiting for API service to stabilize..."

aws ecs wait services-stable \
  --region "${AWS_REGION}" \
  --cluster "${CLUSTER_NAME}" \
  --services "${API_SERVICE_NAME}"

echo "Waiting for worker service to stabilize..."

aws ecs wait services-stable \
  --region "${AWS_REGION}" \
  --cluster "${CLUSTER_NAME}" \
  --services "${WORKER_SERVICE_NAME}"

echo "Deployment completed: ${IMAGE_URI}"

Important: this script uses immutable image tags based on the Git commit SHA. That makes deployments easier to audit and roll back.

Common Mistakes

Most mistakes come from treating APIs and workers as the same kind of ECS service. They share the platform, but they do not share the same failure model.

• Doing slow work inside API requests instead of using workers.
• Scaling workers by CPU only while queue backlog grows.
• Deleting messages before processing succeeds.
• Ignoring idempotency and corrupting data during retries.
• Running stateful containers that lose data during task replacement.
• Using one task for production APIs and losing availability during deployment.
• Missing DLQ alerts for worker failures.
• Using heavy health checks that depend on slow downstream systems.
• Not handling graceful shutdown for in-flight requests or messages.
• Using the same scaling policy for APIs and workers.

Production Checklist

Production ECS APIs and workers should be designed around task replacement, safe failure handling, clear metrics, and independent scaling.

• Run APIs behind an Application Load Balancer.
• Keep ECS tasks in private subnets unless public networking is explicitly required.
• Run at least two API tasks for production availability.
• Keep containers stateless and store durable state externally.
• Use queues for slow or retryable work.
• Design workers for idempotency.
• Configure visibility timeouts based on real processing duration.
• Use dead-letter queues for repeatedly failing messages.
• Scale APIs by request pressure or resource usage.
• Scale workers by queue depth or oldest message age.
• Use centralized logs with request IDs and job IDs.
• Alert on API latency and 5xx errors.
• Alert on worker backlog and DLQ messages.
• Handle graceful shutdown for APIs and workers.
• Store secrets in Secrets Manager or SSM Parameter Store.

Conclusion

Running production APIs and background workers on Amazon ECS requires more than container deployment. APIs need low-latency request handling, health-based routing, stateless design, and safe scaling. Workers need durable queues, idempotency, retry handling, backlog monitoring, and safe message acknowledgement.

ECS is a strong platform for both workload types because it provides service management, task replacement, deployment control, networking integration, IAM roles, and scaling. The architecture becomes reliable when API and worker responsibilities are separated clearly.

Key Takeaway: use ECS APIs for synchronous request handling, ECS workers for asynchronous processing, and queues between them when work is slow, retryable, or resource-heavy.

More Articles to Read

Continue with ECS articles that explain runtime design, compute choices, networking, deployment behavior, scaling, and production operations.

• Amazon ECS Task Definitions, Services, and Deployments Explained
• AWS Fargate vs EC2 for ECS: Architecture Trade-Offs
• Amazon ECS Networking Explained: VPC, ALB, Subnets, and Security Groups
• Amazon ECS Deployment Strategies and Auto Scaling
• Amazon ECS Best Practices for Production