Category: Messaging & Event Streaming Tags: amazon-sqs message-queues asynchronous-processing

What Is Amazon SQS?

By Oleksandr Andrushchenko — Published on — Modified on
0 0

Amazon Simple Queue Service (Amazon SQS) is a fully managed message queuing service from AWS. It allows applications to send, store, and receive messages without operating their own message broker infrastructure. A producer sends a message to a queue, and a consumer retrieves it when ready to perform the work. The producer and consumer do not need to be running at the same time.

SQS is widely used for background jobs, order processing, email delivery, image processing, and communication between microservices. It helps absorb traffic bursts and isolate temporary failures, but reliable processing still requires careful handling of retries, duplicate messages, visibility timeouts, dead-letter queues, and consumer concurrency. A message successfully accepted by SQS is not the same as a business operation successfully completed.

Table of Contents

What Amazon SQS Provides

Amazon SQS is a managed queue between components that produce work and components that perform it. Producers use the SQS API to publish messages, and consumers retrieve messages independently. AWS operates the queue infrastructure, including storage, redundancy, and availability.

SQS does not execute application code or automatically complete the requested work. Consumers must process messages, handle failures, and acknowledge successful processing. This separation is what makes SQS useful for distributed systems where workloads have different execution times and scaling requirements.

Why Applications Use SQS

Consider an API that generates PDF reports. Producing a report can take several minutes, but the client does not need to keep an HTTP connection open throughout the operation.

Instead, the API creates a job record and publishes a message containing a job ID. A background worker eventually processes the report, uploads the result to S3, and updates the job status.

The approach provides several benefits:

  • Decoupling: Producers do not need direct connections to consumers.
  • Traffic buffering: A burst of requests can accumulate as queued work.
  • Independent scaling: Worker capacity can increase without scaling API servers.
  • Failure isolation: A temporary worker outage does not necessarily prevent producers from accepting new work.
  • Controlled concurrency: The number of active consumers can be limited to protect databases or external APIs.
  • Retry support: Failed work can become available for another attempt.

The cost is additional asynchronous state. Clients need a way to track job completion, workers need retry logic, and operators need to monitor whether the queue is processing work fast enough.

The underlying messaging model is covered in What Is a Message Queue?.

SQS vs. SNS, EventBridge, and Kafka

SQS is primarily a queue for distributing work among consumers. Other messaging services are optimized for different forms of communication.

Service Main Purpose Typical Pattern
Amazon SQS Buffer and distribute asynchronous work Queue with competing consumers
Amazon SNS Distribute notifications to subscribers Publish/subscribe fan-out
Amazon EventBridge Route events based on rules Event bus and filtering
Apache Kafka Store and process durable event streams Partitioned event log with consumer offsets

These services can complement each other. For example, an SNS topic can fan out an order event to several SQS queues, allowing email, analytics, and fulfillment consumers to process the event independently.

Multiple workers polling the same SQS queue generally compete for messages rather than each receiving a copy. To provide an independent copy to multiple services, use separate queues connected through an appropriate fan-out or event-routing mechanism.

See What Is Pub/Sub? for the difference between distributing work and broadcasting events.

How Amazon SQS Works

The core SQS API is intentionally small. Producers send messages, consumers receive them, and consumers delete messages after completing their work. An important detail is that receiving a message does not remove it from the queue.

This behavior allows messages to be processed again when a worker crashes, but it also means applications must expect duplicate processing attempts.

Message Lifecycle and Acknowledgment

A typical message lifecycle looks like this:

  1. The producer calls SendMessage.
  2. SQS accepts and stores the message.
  3. A consumer calls ReceiveMessage.
  4. SQS returns the message and a receipt handle.
  5. The message becomes temporarily invisible to other consumers.
  6. The consumer completes the required business operation.
  7. The consumer calls DeleteMessage using the receipt handle.

Deletion is the acknowledgment that processing has finished. If the consumer crashes before deleting the message, the message can become visible again after its visibility timeout expires.

Every receive attempt can return a different receipt handle. The consumer must use the receipt handle from the applicable receive operation, not the message ID, to delete the message.

Visibility Timeout and In-Flight Messages

The visibility timeout controls how long a received message remains hidden from other consumers. The default is 30 seconds, and SQS supports configurable visibility timeouts up to 12 hours.

Suppose an image-processing worker needs 90 seconds to finish, but the queue has a 30-second visibility timeout. Another worker may receive the message before the first worker completes its work.

Both workers can then process the same image concurrently. This is not necessarily an SQS failure; the message became available according to its configured visibility timeout.

Choose a visibility timeout that covers normal processing and reasonable variation. Long-running workers can extend message visibility using ChangeMessageVisibility, but should use bounded extensions and detect when processing can no longer make progress.

A message that has been received but not deleted is considered in flight. Standard queues have in-flight message limits, so failing to delete or release messages can eventually affect consumption.

Long Polling, Retention, and Message Size

SQS supports short polling and long polling. With long polling, the receive request can wait for messages to arrive rather than returning immediately when no message is available.

Setting WaitTimeSeconds to a value between 1 and 20 seconds enables long polling for an individual receive request. This usually reduces empty responses and unnecessary API calls.

Other queue configuration values include:

  • Message retention: From 1 minute to 14 days, with a default of 4 days.
  • Maximum message size: Up to 1 MiB for supported SQS messages.
  • Delivery delay: Queue-level delay up to 15 minutes.
  • Visibility timeout: Controls how long a received message remains unavailable to other consumers.
  • Receive wait time: Up to 20 seconds for long polling.

For large payloads, store the actual data in S3 and send a message containing an object key or document identifier. Messages should normally represent a command or event, not contain entire videos, archives, or large serialized data structures.

Standard vs. FIFO Queues

Amazon SQS offers Standard and FIFO queue types. They differ in ordering, duplicate-delivery behavior, and concurrency controls.

The correct choice depends on whether the application can process independent messages in any order or must preserve a defined order for related operations.

Standard Queues

Standard queues are designed for high throughput and at-least-once message delivery. Messages may be delivered more than once, and strict ordering is not guaranteed.

Standard queues are a good fit for:

  • Image and document processing.
  • Email and notification jobs.
  • Independent report-generation requests.
  • Background work where messages can safely be processed out of order.

For example, thumbnails for 1,000 independent images do not usually need strict global processing order. Multiple workers can process whichever image jobs become available.

Standard queues also support message group identifiers for fair-queue behavior, helping distribute processing more fairly among tenants with uneven traffic. This does not turn a Standard queue into an ordered FIFO queue.

FIFO Ordering and Deduplication

FIFO stands for First-In-First-Out. SQS FIFO queues preserve ordering within a message group and provide deduplication mechanisms to prevent duplicate message insertion within a defined window.

FIFO messages use two important attributes:

  • MessageGroupId: Identifies messages that must be processed sequentially relative to one another.
  • MessageDeduplicationId: Identifies duplicate sends within the five-minute deduplication interval.

Suppose an order goes through these events:

MessageGroupId = ORDER#1001

1. order.created
2. payment.authorized
3. order.shipped

MessageGroupId = ORDER#1002

1. order.created
2. payment.authorized
3. order.cancelled

Messages for each order group are processed in order, while different groups can be processed concurrently. Using one group ID for every message unnecessarily serializes the entire workload.

FIFO queues can use explicit deduplication IDs or content-based deduplication. Content-based deduplication hashes the message body, not the message attributes.

import json
import boto3

sqs = boto3.client("sqs")


def publish_order_event(
    queue_url: str,
    order_id: str,
    event_id: str,
    event_type: str,
):
    # queue_url must identify a FIFO queue.
    response = sqs.send_message(
        QueueUrl=queue_url,
        MessageBody=json.dumps({
            "event_id": event_id,
            "order_id": order_id,
            "type": event_type,
        }),
        MessageGroupId=f"ORDER#{order_id}",
        MessageDeduplicationId=event_id,
    )

    return response["MessageId"]

FIFO deduplication is not a substitute for application idempotency. If processing completes but acknowledgment fails, business side effects may still be repeated during retries. A message republished after the five-minute deduplication interval can also be treated as a new message.

Choosing the Right Queue Type

Choose based on actual business requirements rather than assuming FIFO is always safer.

Requirement Standard Queue FIFO Queue
Strict order within a message group No Yes
Delivery model At least once Ordered with send deduplication
Parallelism Independent competing consumers Across distinct message groups
Typical use case Independent background jobs Order-sensitive commands
Consumer idempotency needed Yes Yes

If the order in which image thumbnails are generated does not matter, Standard queues are usually sufficient. If account operations must occur in sequence for each account, FIFO message groups may be appropriate.

Be aware that not every AWS event source can send directly to FIFO queues. For example, native S3 Event Notifications do not directly target SQS FIFO queues; EventBridge can be used when a compatible routing design is needed.

Reliability, Retries, and Dead-Letter Queues

SQS reliably holds messages during normal consumer outages, but durable transport does not guarantee successful business processing. Consumers can crash, external APIs can time out, database transactions can fail, and poison messages can repeatedly trigger the same error.

A production queue therefore needs an explicit policy for successful acknowledgment, transient failures, permanent failures, and duplicate deliveries.

Idempotent Consumers

Consider a worker that receives an invoice.paid message and credits a customer's account. The worker commits the balance update, then crashes before calling DeleteMessage.

SQS may redeliver the message. If the worker blindly adds the amount again, the customer is credited twice.

When both the business change and deduplication record live in PostgreSQL, they can be committed in one transaction:

import psycopg


def apply_credit(
    conn: psycopg.Connection,
    event_id: str,
    account_id: int,
    amount_cents: int,
) -> bool:
    # The processed_events table must enforce
    # uniqueness on (consumer, event_id).
    with conn.transaction():
        with conn.cursor() as cursor:
            cursor.execute(
                """
                INSERT INTO processed_events (
                    consumer,
                    event_id
                )
                VALUES (%s, %s)
                ON CONFLICT DO NOTHING
                RETURNING event_id
                """,
                ("account-credit", event_id),
            )

            if cursor.fetchone() is None:
                # This logical event already completed.
                return False

            cursor.execute(
                """
                UPDATE accounts
                SET balance_cents = balance_cents + %s
                WHERE id = %s
                """,
                (amount_cents, account_id),
            )

            if cursor.rowcount != 1:
                raise ValueError("Account not found")

    return True

The transaction commits the deduplication record and balance update together. If the worker crashes after commit, redelivery finds the existing event record and skips the duplicate credit.

This assumes the connection has no already-active transaction and the two tables are in the same PostgreSQL database. External side effects, such as charging a payment provider, require additional provider idempotency support or workflow coordination.

For the underlying reliability technique, see What Is Message Deduplication?.

Dead-Letter Queues and Retry Strategies

A dead-letter queue (DLQ) stores messages that fail processing repeatedly. The source queue's redrive policy specifies which DLQ receives failed messages and the maximum receive count before redrive.

For example, if a message repeatedly fails validation or triggers a persistent application error, sending it to a DLQ prevents it from cycling through the normal queue indefinitely.

Different failure classes require different handling:

  • Temporary network failure: Retry after an appropriate delay.
  • Database outage: Apply bounded retries and protect the database from a retry storm.
  • Rate limit: Reduce concurrency and delay further attempts.
  • Invalid payload: Record the validation failure and move toward controlled DLQ handling.
  • Unsupported schema: Investigate producer-consumer compatibility before replaying.

A DLQ is not permanent storage. Messages still have a retention period, and operators need a process for investigating, fixing, and safely replaying them.

For FIFO workloads, take extra care when removing failed messages from their ordered group. Moving one message to a DLQ can allow subsequent messages to proceed, potentially violating application-level sequence assumptions.

See Dead-Letter Queues, Retries, and Poison Messages for failure classification and recovery design.

SQS with AWS Lambda

Lambda can poll SQS through an event source mapping. AWS manages the polling process and invokes the Lambda function with batches of messages.

For Standard queues, the function may process multiple messages per invocation, and the integration can scale concurrent processing. For FIFO queues, Lambda respects message-group ordering while processing separate groups concurrently.

Partial batch responses allow Lambda handlers to identify individual failed messages so that successful messages do not need to be retried with the entire batch.

Unlike a custom boto3 polling worker, a Lambda SQS consumer should not manually delete messages as its normal acknowledgment mechanism. The event source mapping manages deletion based on invocation success and configured partial batch responses.

The visibility timeout must account for the Lambda timeout, polling behavior, and potential retries. The AWS guidance commonly recommends a visibility timeout of at least six times the function timeout, with additional allowance for batch windows where applicable.

Production Design Example: Asynchronous Order Processing

Consider an e-commerce platform where customers submit orders, the system reserves inventory, and several downstream services generate invoices, update analytics, and send notifications.

These tasks have different consistency requirements. Inventory reservation is part of the critical order workflow, while sending a confirmation email may happen asynchronously.

Order Processing Architecture

A production architecture can use:

  • FastAPI order service: Validates order requests and stores order data.
  • PostgreSQL: Holds orders, transaction state, and an outbox table.
  • Outbox publisher: Reliably publishes committed order events.
  • Amazon SNS or EventBridge: Routes events to independent subscriber queues.
  • Amazon SQS: Buffers email, fulfillment, and analytics work.
  • ECS or Lambda workers: Consume messages and perform asynchronous actions.
  • Dead-letter queues: Preserve messages that cannot complete after bounded attempts.

Each downstream service should have its own queue if it must process every event independently. For example, successful email processing should not delete the fulfillment service's copy of the event.

Use small, versioned messages that identify business entities and events:

{
  "event_id": "evt_9fd71",
  "event_type": "order.created",
  "schema_version": 1,
  "order_id": "ord_10482",
  "tenant_id": "tenant_42",
  "created_at": "2026-10-20T14:00:00Z"
}

The message identifies the completed business event without copying an entire customer record, payment credentials, or unnecessary sensitive data.

Failure Scenarios and Recovery

One failure occurs between committing an order and publishing its message. If the database transaction succeeds but the application crashes before calling SQS, downstream consumers never receive the event.

The transactional outbox pattern addresses this by writing the business change and outgoing event record within one database transaction. A separate publisher then delivers committed events to the messaging infrastructure.

Read What Is a Transactional Outbox? for the dual-write problem and its implementation.

Another failure occurs when fulfillment workers become unavailable while producers continue creating orders. SQS buffers the work, but the backlog grows. Operators need to monitor the age of the oldest message, not merely the total queue length.

When workers recover, increasing concurrency can drain the backlog. However, excessive parallelism may overwhelm inventory databases, payment providers, or shipping APIs. The recovery strategy must scale consumers within the safe limits of downstream systems.

Ready-to-Use Example: SQS with Terraform and Python

This example creates a Standard SQS queue, a dead-letter queue, managed server-side encryption, and a redrive policy using Terraform. Python scripts publish messages and process them with long polling.

The worker deletes messages only after successful processing. It deliberately leaves failed messages available for retry after the visibility timeout. This is a transport-level demonstration; production side effects must additionally be idempotent.

1. Project Structure

Install Terraform, Python 3, and the AWS CLI, then configure AWS credentials with the necessary permissions for SQS queue provisioning and messaging operations.

Create the following files:

sqs-demo/
  main.tf
  outputs.tf
  requirements.txt
  producer.py
  worker.py

The example uses boto3 for messaging and the standard Python library for payload validation and logging.

2. Provision Queues with Terraform

Create main.tf:

terraform {
  required_version = ">= 1.5.0"

  required_providers {
    aws = {
      source  = "hashicorp/aws"
      version = "~> 6.0"
    }
  }
}

provider "aws" {
  region = "us-west-2"
}

# Failed messages are retained for investigation.
resource "aws_sqs_queue" "dlq" {
  name = "sysdespro-sqs-demo-dlq"

  message_retention_seconds = 1209600

  sqs_managed_sse_enabled = true

  tags = {
    Environment = "development"
    ManagedBy   = "terraform"
  }
}

# Permit only this source queue to redrive into the DLQ.
resource "aws_sqs_queue_redrive_allow_policy" "dlq" {
  queue_url = aws_sqs_queue.dlq.id

  redrive_allow_policy = jsonencode({
    redrivePermission = "byQueue"
    sourceQueueArns = [
      aws_sqs_queue.jobs.arn
    ]
  })
}

# Queue holding asynchronous jobs.
resource "aws_sqs_queue" "jobs" {
  name = "sysdespro-sqs-demo-jobs"

  # Messages can be received again if not deleted
  # within 60 seconds after receipt.
  visibility_timeout_seconds = 60

  # Keep unprocessed messages for four days.
  message_retention_seconds = 345600

  # Enable long polling at the queue level.
  receive_wait_time_seconds = 20

  # Encrypt stored messages using SQS-managed keys.
  sqs_managed_sse_enabled = true

  # Move repeatedly failing messages to the DLQ.
  redrive_policy = jsonencode({
    deadLetterTargetArn = aws_sqs_queue.dlq.arn
    maxReceiveCount     = 5
  })

  tags = {
    Environment = "development"
    ManagedBy   = "terraform"
  }
}

This configuration creates an encrypted source queue with a 60-second visibility timeout and 20-second long polling. A message that repeatedly fails processing can move to the DLQ after exceeding the configured receive threshold.

The source queue retains messages for four days, while the DLQ retains them for fourteen days. This gives operators more time to inspect failures, although retention planning should reflect actual incident-response requirements.

Create outputs.tf:

output "queue_url" {
  description = "URL of the main SQS queue."
  value       = aws_sqs_queue.jobs.id
}

output "queue_arn" {
  description = "ARN of the main SQS queue."
  value       = aws_sqs_queue.jobs.arn
}

output "dlq_url" {
  description = "URL of the dead-letter queue."
  value       = aws_sqs_queue.dlq.id
}

3. Implement a Python Producer

Create requirements.txt:

boto3

Create producer.py:

import json
import os
import uuid
from datetime import datetime, timezone

import boto3
from botocore.config import Config


QUEUE_URL = os.environ["SQS_QUEUE_URL"]
REGION = os.getenv("AWS_REGION", "us-west-2")

sqs = boto3.client(
    "sqs",
    region_name=REGION,
    config=Config(
        retries={
            "mode": "standard",
            "max_attempts": 5,
        }
    ),
)


def publish_job(
    job_id: str,
    action: str,
    attempts: int = 1,
) -> str:
    message = {
        "event_id": str(uuid.uuid4()),
        "schema_version": 1,
        "job_id": job_id,
        "action": action,
        "attempts": attempts,
        "created_at": datetime.now(
            timezone.utc
        ).isoformat(),
    }

    response = sqs.send_message(
        QueueUrl=QUEUE_URL,
        MessageBody=json.dumps(message),
        MessageAttributes={
            "event_type": {
                "DataType": "String",
                "StringValue": "job.requested",
            }
        },
    )

    return response["MessageId"]


if __name__ == "__main__":
    message_id = publish_job(
        job_id="job-1001",
        action="generate_report",
    )

    print("Published message:", message_id)

This producer publishes a JSON message with a business job ID, event ID, schema version, and creation timestamp. In a real application, the event ID should remain stable when retrying publication of the same logical event rather than being regenerated for every attempt.

The attempts property here is application metadata, not an SQS receive count. SQS exposes ApproximateReceiveCount as a system attribute when requested by consumers.

4. Implement a Reliable Worker

Create worker.py:

import json
import logging
import os
import signal
import time

import boto3
from botocore.config import Config


QUEUE_URL = os.environ["SQS_QUEUE_URL"]
REGION = os.getenv("AWS_REGION", "us-west-2")

logging.basicConfig(
    level=logging.INFO,
    format="%(asctime)s %(levelname)s %(message)s",
)

logger = logging.getLogger(__name__)

running = True

sqs = boto3.client(
    "sqs",
    region_name=REGION,
    config=Config(
        # Read timeout must exceed 20-second long polling.
        read_timeout=30,
        connect_timeout=5,
        retries={
            "mode": "standard",
            "max_attempts": 5,
        },
    ),
)


def stop_worker(signum, frame):
    global running
    running = False


signal.signal(signal.SIGTERM, stop_worker)
signal.signal(signal.SIGINT, stop_worker)


def process_message(body: dict) -> None:
    if body.get("schema_version") != 1:
        raise ValueError("Unsupported schema version")

    if body.get("action") != "generate_report":
        raise ValueError("Unsupported action")

    job_id = body.get("job_id")

    if not isinstance(job_id, str) or not job_id:
        raise ValueError("Missing job_id")

    # Replace with actual business processing.
    # Production side effects must be idempotent.
    logger.info("Processing report job %s", job_id)

    # Simulate bounded processing time.
    time.sleep(1)

    logger.info("Completed report job %s", job_id)


def receive_and_process() -> None:
    response = sqs.receive_message(
        QueueUrl=QUEUE_URL,
        MaxNumberOfMessages=5,
        WaitTimeSeconds=20,
        MessageSystemAttributeNames=[
            "ApproximateReceiveCount"
        ],
    )

    for message in response.get("Messages", []):
        message_id = message["MessageId"]
        receipt_handle = message["ReceiptHandle"]

        try:
            body = json.loads(message["Body"])

            if not isinstance(body, dict):
                raise ValueError(
                    "Expected a JSON object"
                )

            logger.info(
                "Received message=%s attempt=%s",
                message_id,
                message.get("Attributes", {}).get(
                    "ApproximateReceiveCount", "unknown"
                ),
            )

            process_message(body)

            # Acknowledge only after processing succeeds.
            sqs.delete_message(
                QueueUrl=QUEUE_URL,
                ReceiptHandle=receipt_handle,
            )

            logger.info(
                "Acknowledged message=%s",
                message_id,
            )

        except Exception:
            # Do not delete failed messages.
            # They become visible after timeout and
            # may later be moved to the DLQ.
            logger.exception(
                "Failed to process message=%s",
                message_id,
            )


def main():
    logger.info("SQS worker started")

    while running:
        try:
            receive_and_process()
        except Exception:
            # This catches polling/transport failures.
            logger.exception("SQS polling failed")
            time.sleep(2)

    logger.info("SQS worker stopped")


if __name__ == "__main__":
    main()

The worker uses 20-second long polling, receives up to five messages per request, and deletes each message individually after successful processing.

Failures leave messages in the queue for retry. The queue's visibility timeout means a message can become available again if processing takes too long or acknowledgment fails.

For real jobs, processing time should remain safely within the visibility window or use controlled visibility extensions. The loop also processes a received batch sequentially; in production, choose batch size and parallelism based on task duration and shutdown requirements.

Deleting a message is not proof that an external side effect happened exactly once. A worker can finish its database update, crash before deletion, and then receive the same logical message again. Use durable idempotency protections around non-repeatable business operations.

5. Deploy, Test, and Clean Up

Provision the queues and run the scripts:

# Initialize and validate Terraform.
terraform init
terraform fmt
terraform validate

# Review and create the queue resources.
terraform plan
terraform apply

# Configure the Python scripts.
export AWS_REGION="us-west-2"
export SQS_QUEUE_URL="$(terraform output -raw queue_url)"

# Prepare the Python environment.
python3 -m venv .venv
source .venv/bin/activate
pip install -r requirements.txt

# Start the worker in one terminal.
python worker.py

# In another terminal, publish a job.
python producer.py

# Inspect queue attributes.
aws sqs get-queue-attributes \
  --queue-url "$SQS_QUEUE_URL" \
  --attribute-names \
    ApproximateNumberOfMessages \
    ApproximateNumberOfMessagesNotVisible

# Inspect the DLQ.
DLQ_URL="$(terraform output -raw dlq_url)"

aws sqs get-queue-attributes \
  --queue-url "$DLQ_URL" \
  --attribute-names ApproximateNumberOfMessages

# Delete the disposable queues after testing.
terraform destroy

A successful test should show the worker receiving, processing, and acknowledging the message. Queue counts are approximate and may not immediately reflect the latest message state.

To test failures, publish a message with an unsupported action, or modify the worker's validation logic. The message should remain available for retries and eventually enter the DLQ according to the redrive policy.

Destroying the queues deletes their contents. Never delete production queues or DLQs without verifying that pending and failed messages are no longer needed.

Scaling, Security, and Cost

SQS scales independently from its consumers, but queue capacity and processing capacity are different concerns. A queue can continue accepting messages while workers fall behind, and a large worker fleet can overwhelm databases or external APIs even when SQS itself remains healthy.

Operational design should therefore combine backlog monitoring, bounded consumer concurrency, least-privilege access, and appropriate cost controls.

Worker Scaling and Backpressure

Queue depth is a useful signal, but the age of the oldest message often reveals more about processing health.

For example, 20,000 queued messages may represent a normal burst if workers can clear them within seconds. By contrast, 200 messages waiting for 30 minutes may indicate a consumer outage.

Useful scaling inputs include:

  • Number of visible messages.
  • Age of the oldest unprocessed message.
  • Average processing duration.
  • Incoming message rate.
  • Successful processing throughput.
  • Downstream database or API capacity.

A practical worker-scaling approximation is:

Required workers ≈ Incoming messages per second × Average processing duration in seconds

If 100 messages arrive every second and each message takes 0.5 seconds of worker time, approximately 50 concurrently processing workers are needed to keep up under ideal conditions.

Add headroom for retries, processing variance, and failures, but cap concurrency according to downstream capacity. For ECS workers, this can be implemented with Service Auto Scaling and suitable CloudWatch metrics. Lambda consumers use event source mapping concurrency controls.

IAM, Encryption, and Networking

Amazon SQS supports IAM authorization, queue resource policies, encryption, and private networking through VPC endpoints.

A worker that only receives and acknowledges messages typically needs permissions such as sqs:ReceiveMessage, sqs:DeleteMessage, sqs:ChangeMessageVisibility when required, and sqs:GetQueueAttributes for supported operations.

A producer usually needs sqs:SendMessage. Avoid giving every producer permission to purge or delete queues.

SQS provides encryption at rest using SQS-managed keys or supported AWS KMS configurations. Always use HTTPS for API communication and avoid putting sensitive data in message bodies unless it is required and properly protected.

For private VPC workloads, an SQS interface endpoint can provide connectivity without requiring traffic to use a NAT gateway for SQS access, subject to endpoint policies, routing, and DNS configuration.

SQS Pricing and Observability

Amazon SQS pricing depends on queue type, request usage, payload sizing, and additional enabled services or features. Larger message bodies can affect request-unit billing, and KMS-backed encryption can introduce additional KMS usage charges.

Long polling often reduces unnecessary empty receives and improves consumer efficiency. Batch APIs can reduce request counts when producers and consumers are designed to use them safely.

Monitor CloudWatch metrics such as:

  • ApproximateNumberOfMessagesVisible: Available messages waiting for processing.
  • ApproximateAgeOfOldestMessage: Approximate age of the oldest eligible message.
  • ApproximateNumberOfMessagesNotVisible: Messages currently in flight.
  • NumberOfMessagesSent: Producer activity.
  • NumberOfMessagesDeleted: Deletion activity, not a guaranteed count of unique completed business jobs.
  • NumberOfEmptyReceives: Receive requests returning no messages.
  • DLQ visible messages: Failed messages requiring investigation.

Combine queue metrics with application logs, processing duration, error rate, and downstream resource utilization. CloudWatch SQS metrics are useful operational approximations, not always exact real-time counts.

Common Mistakes and Production Checklist

SQS provides durable asynchronous transport, but most messaging incidents result from application-level mistakes rather than queue infrastructure failures. Visibility settings, duplicate processing, incorrect ordering assumptions, and missing DLQ procedures cause recurring problems.

The safest approach is to define what successful processing means before implementing message acknowledgment.

Common SQS Mistakes

Common problems include:

  • Deleting messages before processing: A consumer crash permanently loses unfinished work.
  • Assuming delivery is exactly once: Duplicate messages repeat business side effects.
  • Using a visibility timeout shorter than processing time: Concurrent workers receive the same message.
  • Using FIFO when ordering is unnecessary: Message groups create avoidable throughput constraints.
  • Using one FIFO group for all tenants: Independent work becomes serialized.
  • Failing to configure long polling: Consumers generate unnecessary empty receive requests.
  • Ignoring pagination or batching limits in surrounding workflows: Producers and consumers silently omit intended work.
  • Treating DLQs as permanent archives: Failed messages expire without investigation.
  • Scaling workers without protecting dependencies: The database or external API becomes overloaded.
  • Assuming published means processed: API responses claim completion before any worker finishes.
  • Putting large files into messages: Payload limits and request costs become problematic.
  • Publishing events separately from database commits: A crash creates missing or inconsistent downstream events.

Production Checklist

Before running a business-critical SQS workload, verify that:

  • The queue type matches actual ordering requirements.
  • Messages have stable business identifiers and explicit schema versions.
  • Producers handle failed or ambiguous publication attempts.
  • Consumers acknowledge messages only after required work completes.
  • Side effects are idempotent across duplicate deliveries.
  • Visibility timeouts match actual processing duration.
  • Long-running processing uses safe visibility extensions when necessary.
  • Long polling and batch sizes are tuned appropriately.
  • Retries distinguish transient failures from permanent ones.
  • Dead-letter queues have retention, alerting, and controlled replay procedures.
  • Consumer concurrency respects downstream capacity.
  • IAM permissions and queue policies follow least privilege.
  • Queue and DLQ age, depth, error rate, and processing time are monitored.
  • Producer database commits and message publishing have a reliable consistency strategy.

Frequently Asked Questions

Amazon SQS is simple to use at the API level, but its delivery model has important implications for application correctness, ordering, and failure recovery.

What Is Amazon SQS Used For?

Amazon SQS is used to decouple software components and distribute asynchronous work. Common applications include background jobs, notifications, document processing, order workflows, and integration between microservices.

Is Amazon SQS Push-Based or Pull-Based?

SQS is primarily pull-based. Consumers retrieve messages through the ReceiveMessage API, often using long polling.

AWS Lambda event source mappings manage the polling process on behalf of a Lambda consumer, so application developers do not need to implement their own polling loop.

Does SQS Delete Messages Automatically After Receiving Them?

No. Receiving a message makes it temporarily invisible. A custom consumer must explicitly delete the message after successful processing.

If the message is not deleted before its visibility timeout expires, it can become available for another receive attempt.

Does Amazon SQS Guarantee Message Ordering?

Standard queues do not guarantee strict ordering. FIFO queues preserve ordering within message groups.

Messages belonging to different FIFO groups can be processed concurrently, so ordering requirements must be defined for the correct business entity or workflow.

What Is the Maximum SQS Message Size?

Amazon SQS supports message payloads up to 1 MiB. For larger content, use object storage such as Amazon S3 and send a reference to the stored object.

This keeps messages compact and reduces unnecessary data transfer through the messaging layer.

What Happens When a Message Cannot Be Processed?

If the consumer fails without deleting the message, SQS can make it available again after the visibility timeout. With a configured redrive policy, repeated failed receives can eventually move the message to a dead-letter queue.

Applications should monitor failed messages and investigate their cause before replaying them.

Is Amazon SQS Free?

SQS has applicable AWS Free Tier allowances or credits for eligible accounts and usage. Additional requests and optional related features are billed according to the current AWS pricing model.

For production estimates, consider queue type, message volume, payload sizes, receive patterns, encryption, and associated consumers.

Conclusion

Amazon SQS is a managed message queuing service that allows producers and consumers to communicate asynchronously. It helps applications absorb traffic bursts, isolate failures, and scale background processing independently from user-facing services.

Reliable SQS applications depend on more than durable message storage. Engineers must choose suitable queue types, use safe visibility timeouts, implement idempotent consumers, configure bounded retries, monitor dead-letter queues, and manage worker concurrency according to downstream capacity.

Key Takeaway: Amazon SQS separates the creation of work from its execution. It provides the infrastructure to queue and redeliver messages, but applications are responsible for turning at-least-once delivery into correct business processing.

These additional articles explore messaging reliability, background processing, event-driven architecture, and common message broker design patterns.

Author

Oleksandr Andrushchenko

Oleksandr Andrushchenko

Software Engineer & Solution Architect
Nov 03, 2025 6 261
Want more articles like this?

Show your appreciation for Oleksandr Andrushchenko.

Your support helps them keep creating and sharing.

Related articles

Comments (0)