Dead-Letter Queues, Retries, and Poison Messages

Message processing does not always succeed on the first attempt. A database may be temporarily unavailable, an external API may return a rate-limit response, a network request may time out, or a message may contain invalid data that can never be processed successfully. Reliable messaging systems must distinguish between failures that may recover and failures that require investigation.

Retries provide recovery from temporary problems, while dead-letter queues isolate messages that exceed retry limits or cannot be processed safely. Poison messages are especially dangerous because they can fail repeatedly, consume worker capacity, block ordered queues, and create endless retry loops. A production-ready design therefore needs bounded retries, exponential backoff, idempotent consumers, clear failure classification, observability, and controlled message replay.

Table of Contents

• Why Message Processing Fails
• Understanding Retries
  • Immediate Retries
  • Delayed Retries
  • Exponential Backoff
  • Jitter
• Transient, Permanent, and Unknown Failures
• What Is a Poison Message?
• How Poison Messages Affect Systems
• What Is a Dead-Letter Queue?
• Dead-Letter Queue Architectures
  • Queue-Specific DLQ
  • Shared DLQ
  • Failure Database
• Designing a Retry Policy
• Retry Metadata
• Idempotency During Retries
• Retrying External API Calls
• Handling Rate Limits
• Retry Queues vs Delayed Delivery
• Ordering and Retries
• Dead-Letter Message Design
• Monitoring Dead-Letter Queues
• Investigating Failed Messages
• Replaying and Redriving Messages
• Safe Bulk Replay
• Retention and Cleanup
• Real-World Order Processing Example
• Common Mistakes
• Production Checklist
• Conclusion
• Key Takeaway
• More Articles to Read

Why Message Processing Fails

A consumer may fail even when the message broker works correctly. Message delivery and message processing are separate concerns. A broker can deliver a message successfully while the consumer fails during validation, database access, business logic, or communication with another service.

Common causes of message-processing failure include:

• Temporary network interruptions.
• Database connection failures.
• External API timeouts.
• Rate limits.
• Invalid or incomplete message payloads.
• Unsupported message versions.
• Missing business entities.
• Authentication or authorisation failures.
• Resource exhaustion.
• Consumer defects.
• Expired credentials.
• Unexpected business-state transitions.

Not every failure should produce the same response. Retrying a temporary timeout may succeed. Retrying a message with an invalid customer ID one hundred times will not repair the message.

def process_message(message: dict) -> None:
    validate_message(message)

    order = orders.find(message["order_id"])

    if order is None:
        raise PermanentMessageError(
            f"Order does not exist: {message['order_id']}"
        )

    payment_gateway.capture(
        payment_id=message["payment_id"],
        amount=message["amount"]
    )

A reliable consumer classifies the failure before deciding whether to acknowledge, retry, delay, or dead-letter the message.

Understanding Retries

A retry is another attempt to process a failed message. Retries are useful because many distributed-system failures are temporary. A service may be unavailable for ten seconds, a database may fail over to another node, or a provider may recover after a short rate-limit window.

However, retries increase load. A failing dependency may receive even more traffic from repeated attempts. A good retry policy therefore limits attempts, spreads retries over time, and stops retrying permanent failures.

Immediate Retries

An immediate retry occurs almost as soon as the previous attempt fails.

Advantages

• Simple to implement.
• Useful for very short transient failures.
• Can recover from occasional connection resets.

Disadvantages

• Can overload an already failing dependency.
• Consumes worker capacity while waiting.
• May create a tight failure loop.

When to Use

Use immediate retries only for failures likely to recover within milliseconds or a few seconds, and keep the number of attempts low.

Real-World Use Cases

• A single database connection reset.
• A temporary DNS resolution failure.
• An optimistic-lock conflict that can be retried immediately.

Example

def execute_with_immediate_retry(operation) -> None:
    maximum_attempts = 3

    for attempt in range(1, maximum_attempts + 1):
        try:
            operation()
            return
        except ConnectionResetError:
            if attempt == maximum_attempts:
                raise

Delayed Retries

A delayed retry returns the message to a queue or scheduler and makes it available after a waiting period.

Advantages

• Allows dependencies time to recover.
• Releases the worker to process other messages.
• Reduces repeated pressure on failing systems.

Disadvantages

• Increases end-to-end processing latency.
• Requires delayed-delivery support or retry queues.
• May affect message ordering.

When to Use

Use delayed retries for rate limits, temporary provider outages, unavailable databases, and other failures where recovery may take more than a few seconds.

Real-World Use Cases

• Retrying a failed shipment-provider request after one minute.
• Waiting before checking a pending payment again.
• Retrying an email provider after a temporary outage.

Example

def retry_later(message: dict, delay_seconds: int) -> None:
    retry_message = {
        **message,
        "retry_count": message.get("retry_count", 0) + 1
    }

    message_queue.publish_delayed(
        queue="order-retries",
        payload=retry_message,
        delay_seconds=delay_seconds
    )

Exponential Backoff

Exponential backoff increases the retry delay after every failure. A basic sequence might wait 5 seconds, 10 seconds, 20 seconds, 40 seconds, and 80 seconds.

This gives a failing dependency progressively more recovery time and prevents consumers from retrying at a constant high rate.

def calculate_backoff(
    attempt: int,
    base_delay_seconds: int = 5,
    maximum_delay_seconds: int = 900
) -> int:
    delay = base_delay_seconds * (2 ** attempt)

    return min(delay, maximum_delay_seconds)

The maximum delay prevents retry intervals from growing indefinitely.

Jitter

Jitter adds randomness to retry delays. Without jitter, thousands of failed messages may retry at exactly the same time and create another traffic spike.

import random

def calculate_backoff_with_jitter(attempt: int) -> int:
    base_delay = min(5 * (2 ** attempt), 900)
    jitter = random.randint(0, max(1, base_delay // 4))

    return base_delay + jitter

Jitter is especially important after shared failures such as a regional service outage, database restart, or provider rate limit that affects many messages simultaneously.

Transient, Permanent, and Unknown Failures

Failure classification determines what should happen next.

A permanent failure should not consume every retry attempt. The message can move directly to a dead-letter queue with a clear reason.

class TransientMessageError(Exception):
    pass


class PermanentMessageError(Exception):
    pass


def consume(message: dict) -> None:
    try:
        process_message(message)
        message_queue.acknowledge(message)

    except PermanentMessageError as error:
        send_to_dead_letter_queue(
            message=message,
            reason=str(error),
            failure_type="permanent"
        )
        message_queue.acknowledge(message)

    except TransientMessageError as error:
        schedule_retry(
            message=message,
            error=error
        )

Unknown failures require caution. A generic exception may represent a temporary infrastructure problem or a consumer bug. A limited number of retries is reasonable, but endless retries can hide defects and exhaust capacity.

What Is a Poison Message?

A poison message is a message that repeatedly fails processing and cannot progress through the normal consumer path. The problem may be in the message itself, the consumer code, or the business state required to process it.

Examples include:

• A message missing a required property.
• A payload using an unsupported schema version.
• A message that triggers a reproducible software defect.
• A message referencing a permanently deleted entity.
• A malformed date or numeric value.
• A job requiring credentials that no longer exist.
• An event that violates an unexpected database constraint.

def validate_message(message: dict) -> None:
    required_fields = {
        "message_id",
        "message_type",
        "order_id",
        "created_at"
    }

    missing_fields = required_fields - message.keys()

    if missing_fields:
        raise PermanentMessageError(
            f"Missing fields: {sorted(missing_fields)}"
        )

A poison message is not always obviously malformed. A valid-looking message can become poisonous because a particular code path always fails.

How Poison Messages Affect Systems

Without retry limits or dead-letter handling, one poison message can repeatedly return to the queue.

This can cause:

• High CPU and network use.
• Repeated database queries.
• Duplicate logs and alerts.
• Growing cloud costs.
• Reduced throughput for healthy messages.
• Consumer crash loops.
• Blocked ordered partitions.
• Repeated external API calls.

In strict-order queues, one failing message may prevent every later message in the same partition from being processed. For example, if account events must be applied in sequence, a poison event at sequence 42 can block sequences 43 through 100.

Retries must therefore be bounded. Once a message exceeds the allowed number of attempts, the system should isolate it and continue processing where business rules permit.

What Is a Dead-Letter Queue?

A dead-letter queue, commonly called a DLQ, stores messages that could not be processed successfully through the normal consumer path.

A message may enter a DLQ because:

• It exceeded the maximum retry count.
• It exceeded a maximum processing age.
• Its payload is invalid.
• Its schema version is unsupported.
• A permanent business rule rejected it.
• The consumer explicitly classified it as unrecoverable.

A DLQ is not a complete error-handling strategy by itself. It prevents repeated failures from disrupting normal processing, but the failed message still needs investigation, correction, replay, or permanent closure.

def send_to_dead_letter_queue(
    message: dict,
    reason: str,
    failure_type: str
) -> None:
    dead_letter_queue.publish({
        "original_message": message,
        "failure_reason": reason,
        "failure_type": failure_type,
        "failed_at": current_timestamp(),
        "consumer_name": "order-payment-consumer"
    })

Dead-Letter Queue Architectures

Dead-letter storage can be organised in several ways. The choice affects isolation, observability, replay procedures, and operational complexity.

Queue-Specific DLQ

Each source queue has its own dead-letter queue.

Advantages

• Clear ownership.
• Simple replay to the original queue.
• Failure volume is isolated by workload.
• Alerts can use queue-specific thresholds.

Disadvantages

• More queues must be configured and monitored.
• Operational dashboards may become fragmented.

When to Use

Use queue-specific DLQs when workloads have different owners, retry policies, schemas, or recovery procedures.

Real-World Use Cases

• Separate DLQs for payment, shipment, notification, and report queues.
• Independent services managed by different teams.
• Workloads with different retention requirements.

Example

QUEUE_CONFIGURATION = {
    "payment-jobs": {
        "dead_letter_queue": "payment-jobs-dlq",
        "maximum_attempts": 5
    },
    "notification-jobs": {
        "dead_letter_queue": "notification-jobs-dlq",
        "maximum_attempts": 8
    }
}

Shared DLQ

Several source queues send failed messages into one shared dead-letter queue.

Advantages

• Centralised monitoring.
• Fewer queues to operate.
• One investigation pipeline can process all failures.

Disadvantages

• Messages require source metadata.
• High-volume failures from one service can hide others.
• Replay routing becomes more complex.

When to Use

Use a shared DLQ for small environments or closely related workloads with common ownership and recovery tooling.

Real-World Use Cases

• A small platform with several low-volume background queues.
• A central operations team handling all message failures.
• Development or staging environments.

Example

dead_letter_queue.publish({
    "source_queue": "shipment-events",
    "source_consumer": "shipment-status-updater",
    "original_message": message,
    "failure_reason": str(error)
})

Failure Database

Failed messages are stored in a searchable database table, sometimes alongside or instead of a broker DLQ.

Advantages

• Easy filtering and reporting.
• Supports administrative interfaces.
• Can track investigation and replay status.
• Allows structured failure analytics.

Disadvantages

• Requires custom persistence code.
• The database can become another failure point.
• Replay tooling must republish messages explicitly.

When to Use

Use a failure database when operations teams need searchable history, manual review, approval, remediation, and auditability.

Real-World Use Cases

• Financial operations requiring review before replay.
• Customer-support tools for failed imports.
• Regulated workflows requiring an audit trail.

Example

CREATE TABLE failed_messages (
    failed_message_id VARCHAR(64) PRIMARY KEY,
    original_message_id VARCHAR(64) NOT NULL,
    source_queue VARCHAR(150) NOT NULL,
    consumer_name VARCHAR(150) NOT NULL,
    payload_json JSONB NOT NULL,
    failure_type VARCHAR(50) NOT NULL,
    failure_reason TEXT NOT NULL,
    attempt_count INTEGER NOT NULL,
    first_failed_at TIMESTAMP NOT NULL,
    last_failed_at TIMESTAMP NOT NULL,
    replay_status VARCHAR(30) NOT NULL DEFAULT 'pending',
    replayed_at TIMESTAMP
);

Designing a Retry Policy

A retry policy defines which failures are retried, how many attempts are allowed, how long the system waits, and what happens after retries are exhausted.

A useful policy answers these questions:

• Which exceptions are transient?
• Which failures should dead-letter immediately?
• How many attempts are allowed?
• How quickly should retries occur?
• What is the maximum retry delay?
• What is the maximum total message age?
• Should retry behaviour differ by message type?
• What happens after the final failure?

Different workloads need different policies.

RETRY_POLICIES = {
    "capture_payment": {
        "maximum_attempts": 4,
        "base_delay_seconds": 10,
        "maximum_delay_seconds": 300
    },
    "send_email": {
        "maximum_attempts": 10,
        "base_delay_seconds": 30,
        "maximum_delay_seconds": 3600
    },
    "update_search_index": {
        "maximum_attempts": 6,
        "base_delay_seconds": 5,
        "maximum_delay_seconds": 600
    }
}

The retry policy should reflect business impact, dependency behaviour, and recovery time rather than using one global setting for every queue.

Retry Metadata

Every retry should carry enough metadata for the consumer and operations team to understand message history.

Useful metadata includes:

• Original message ID.
• Current attempt number.
• First attempt time.
• Last failure time.
• Last failure reason.
• Consumer name and version.
• Original source queue.
• Correlation ID.
• Business entity ID.

def create_retry_message(
    message: dict,
    error: Exception
) -> dict:
    now = current_timestamp()

    return {
        **message,
        "retry_metadata": {
            "attempt": (
                message
                .get("retry_metadata", {})
                .get("attempt", 0)
                \+ 1
            ),
            "first_attempt_at": (
                message
                .get("retry_metadata", {})
                .get("first_attempt_at", now)
            ),
            "last_failed_at": now,
            "last_failure_reason": str(error)
        }
    }

Metadata should not grow without limits. Appending complete stack traces for every attempt can create oversized messages. Detailed stack traces belong in logs or failure storage, while the message can contain a short error code and summary.

Idempotency During Retries

At-least-once delivery means the same message may execute more than once. A consumer can finish the business operation and crash before acknowledging the message. The broker then redelivers it.

Without idempotency, retries can cause duplicate charges, duplicate emails, duplicate inventory updates, or duplicate shipments.

CREATE TABLE processed_message_operations (
    consumer_name VARCHAR(150) NOT NULL,
    operation_id VARCHAR(100) NOT NULL,
    processed_at TIMESTAMP NOT NULL,
    PRIMARY KEY (consumer_name, operation_id)
);

def apply_loyalty_points(message: dict) -> None:
    operation_id = message["loyalty_operation_id"]

    with database.transaction():
        inserted = database.execute(
            """
            INSERT INTO processed_message_operations (
                consumer_name,
                operation_id,
                processed_at
            )
            VALUES (%s, %s, NOW())
            ON CONFLICT DO NOTHING
            """,
            ["loyalty-points-consumer", operation_id]
        )

        if inserted.row_count == 0:
            return

        database.execute(
            """
            UPDATE customer_accounts
            SET loyalty_points = loyalty_points + %s
            WHERE customer_id = %s
            """,
            [
                message["points"],
                message["customer_id"]
            ]
        )

The deduplication record and business update must be committed atomically. If they use separate transactions, the consumer may record completion without applying the update or apply the update without recording completion.

External services should receive a stable idempotency key.

payment_gateway.capture(
    payment_id=message["payment_id"],
    amount=message["amount"],
    idempotency_key=message["capture_operation_id"]
)

The same idempotency key must be reused on every retry. Generating a new key per attempt defeats the protection.

Retrying External API Calls

External APIs introduce uncertainty because a timeout does not always mean the remote operation failed. The provider may have completed the request but the response may have been lost.

Consider a payment capture request:

1. The consumer sends the request.
2. The payment provider captures the payment.
3. The network connection fails before the response arrives.
4. The consumer does not know whether the payment completed.

Immediately sending the request again without idempotency may charge the customer twice.

A safe strategy uses:

• A stable idempotency key.
• A provider-status lookup.
• A reconciliation process.
• A separate state for uncertain outcomes.

def capture_payment(message: dict) -> None:
    operation_id = message["capture_operation_id"]

    try:
        result = payment_gateway.capture(
            payment_id=message["payment_id"],
            amount=message["amount"],
            idempotency_key=operation_id
        )

        payments.mark_captured(
            operation_id=operation_id,
            provider_reference=result.reference
        )

    except GatewayTimeoutError:
        existing_result = payment_gateway.find_by_idempotency_key(
            operation_id
        )

        if existing_result is not None:
            payments.mark_captured(
                operation_id=operation_id,
                provider_reference=existing_result.reference
            )
            return

        raise TransientMessageError(
            "Payment result is still unknown"
        )

Some external operations cannot be made safely idempotent. Those workflows may require manual reconciliation instead of automatic retries.

Handling Rate Limits

A rate-limited API often returns a response indicating that the request should be tried later. Retrying immediately usually makes the problem worse.

The consumer should respect a provider-supplied retry delay when available.

def call_shipping_provider(message: dict) -> None:
    response = shipping_provider.create_shipment(message)

    if response.status_code == 429:
        retry_after = response.retry_after_seconds or 60

        raise RateLimitError(
            retry_after_seconds=retry_after
        )

def handle_rate_limit(
    message: dict,
    error: RateLimitError
) -> None:
    retry_message = increment_attempt(message)

    message_queue.publish_delayed(
        queue="shipment-retries",
        payload=retry_message,
        delay_seconds=error.retry_after_seconds
    )

Per-provider and per-tenant rate limits may require shared coordination across worker instances. Local limits inside one process are not enough when dozens of workers call the same provider.

A distributed rate limiter, token bucket, or central scheduler can keep total traffic within the provider limit.

Retry Queues vs Delayed Delivery

There are two common ways to delay retries: broker-native delayed delivery and separate retry queues.

Broker-Native Delayed Delivery

The broker stores the message and makes it available after a delay.

Advantages

• Simple consumer logic.
• No additional queue-routing chain.
• Delay can often be set per message.

Disadvantages

• Not supported consistently by every broker.
• Long delays may have broker-specific limitations.
• Large delayed-message volumes may affect broker performance.

When to Use

Use native delayed delivery when the broker supports it reliably and operational behaviour is well understood.

Real-World Use Cases

• Retrying a request after a provider-supplied delay.
• Scheduling payment-status checks.
• Sending short-term reminders.

Example

message_queue.publish_delayed(
    queue="payment-events",
    payload=retry_message,
    delay_seconds=120
)

Separate Retry Queues

Messages move through dedicated queues with different delay settings before returning to the main queue.

Advantages

• Works with brokers that support queue-level expiration.
• Retry stages are easy to monitor.
• Different delay tiers can be configured.

Disadvantages

• Requires more queues and routing rules.
• Per-message delay flexibility may be limited.
• Ordering becomes more complicated.

When to Use

Use retry queues when delayed delivery is implemented through queue expiration or when distinct retry stages are operationally useful.

Real-World Use Cases

• A 30-second retry queue.
• A 5-minute retry queue.
• A 1-hour retry queue before final dead-lettering.

Example

RETRY_QUEUE_BY_ATTEMPT = {
    1: "orders-retry-30-seconds",
    2: "orders-retry-5-minutes",
    3: "orders-retry-1-hour"
}

Ordering and Retries

Retries can violate message ordering. A later message may complete while an earlier message waits for retry.

Suppose two customer-status events arrive:

1. Version 10 sets the customer to active.
2. Version 11 sets the customer to suspended.

If version 10 fails and retries after version 11, the customer could incorrectly return to active.

Version-aware consumers prevent stale updates.

def apply_customer_status(message: dict) -> None:
    customer = customers.get(message["customer_id"])

    if message["version"] <= customer.last_event_version:
        return

    customers.update_status(
        customer_id=message["customer_id"],
        status=message["status"],
        last_event_version=message["version"]
    )

Strictly ordered workflows may need to stop processing later messages in the same partition until the failed message succeeds or is explicitly skipped.

Possible strategies include:

• Partitioning by business entity.
• Processing one message at a time per partition.
• Using sequence numbers.
• Storing later messages temporarily.
• Rebuilding current state from an authoritative source.

Not every workload requires strict ordering. Relaxing unnecessary ordering improves throughput and simplifies retries.

Dead-Letter Message Design

A dead-letter message should contain enough context to support investigation and replay without depending entirely on old logs.

Useful fields include:

• Original message payload.
• Original message ID.
• Source queue or topic.
• Consumer name.
• Failure type.
• Error code.
• Short failure reason.
• Attempt count.
• First and last failure timestamps.
• Correlation ID.
• Trace ID.
• Message schema version.

def build_dead_letter_message(
    message: dict,
    error: Exception
) -> dict:
    retry_metadata = message.get("retry_metadata", {})

    return {
        "dead_letter_id": generate_id(),
        "source_queue": "order-events",
        "consumer_name": "order-payment-consumer",
        "original_message": message,
        "original_message_id": message["message_id"],
        "failure_type": classify_error(error),
        "failure_reason": str(error),
        "attempt_count": retry_metadata.get("attempt", 0),
        "first_failed_at": retry_metadata.get(
            "first_attempt_at"
        ),
        "last_failed_at": current_timestamp(),
        "correlation_id": message.get("correlation_id")
    }

Sensitive data should be removed or encrypted before storing failed messages. DLQs often retain messages longer than normal queues and may be accessible to operational tools.

Monitoring Dead-Letter Queues

A DLQ that is never monitored becomes a hidden data-loss system. Messages remain stored, but business operations never complete.

Important metrics include:

Alerts should be based on business importance and normal traffic. One invalid marketing email may not require immediate action. One failed payment-settlement message may.

Useful dashboard dimensions include:

• Source queue.
• Consumer version.
• Failure code.
• Tenant or account.
• Message type.
• Deployment version.

A sharp increase in failures after a deployment may indicate an incompatible schema change or consumer defect.

Investigating Failed Messages

Investigation should identify whether the message, consumer, dependency, or business state caused the failure.

A practical investigation process is:

1. Inspect the failure reason and attempt history.
2. Validate the message schema.
3. Check whether the referenced business entity still exists.
4. Review logs and traces using the correlation ID.
5. Check dependency health at the failure time.
6. Identify whether the failure affects other messages.
7. Determine whether replay is safe.
8. Correct data or deploy a code fix.
9. Replay a small sample.
10. Monitor the result before bulk replay.

Messages should be grouped by failure signature. If 50,000 messages fail because of the same missing database column, investigating them individually is unnecessary.

SELECT
    failure_type,
    failure_reason,
    COUNT(*) AS failed_count,
    MIN(first_failed_at) AS first_seen,
    MAX(last_failed_at) AS last_seen
FROM failed_messages
WHERE replay_status = 'pending'
GROUP BY failure_type, failure_reason
ORDER BY failed_count DESC;

Replaying and Redriving Messages

Replay means sending a failed message through processing again after the underlying problem has been corrected. Some platforms call this redrive.

A safe replay process should:

• Confirm the failure cause is fixed.
• Preserve the original business-operation ID.
• Reset retry metadata where appropriate.
• Retain a reference to the original dead-letter record.
• Limit replay rate.
• Monitor success and repeated failures.

def replay_failed_message(
    failed_message: dict
) -> None:
    replay_message = {
        **failed_message["original_message"],
        "replay_metadata": {
            "dead_letter_id": failed_message["dead_letter_id"],
            "replayed_at": current_timestamp()
        },
        "retry_metadata": {
            "attempt": 0,
            "first_attempt_at": current_timestamp()
        }
    }

    message_queue.publish(
        queue=failed_message["source_queue"],
        payload=replay_message
    )

Do not generate new business idempotency keys during replay. The replay is another attempt at the same operation, not a new operation.

Some messages should not be replayed. For example, an expired notification may no longer be useful, or an old status update may be obsolete because newer state already exists.

Safe Bulk Replay

Bulk replay can produce a sudden traffic spike against consumers, databases, and external providers. Replaying 100,000 messages at once may recreate the original outage.

A safe bulk replay uses:

• Rate limits.
• Small initial batches.
• Failure-rate thresholds.
• Pause and resume controls.
• Separate replay queues.
• Downstream capacity checks.

def replay_batch(
    failed_messages: list[dict],
    maximum_batch_size: int = 100
) -> None:
    for failed_message in failed_messages[:maximum_batch_size]:
        replay_failed_message(failed_message)

        # Prevent replay from overwhelming consumers.
        sleep(0.05)

A dedicated replay queue can use lower worker concurrency than the normal queue.

WORKER_CONFIGURATION = {
    "order-events": {
        "concurrency": 50
    },
    "order-events-replay": {
        "concurrency": 5
    }
}

Replay should stop automatically if the new failure rate exceeds an acceptable threshold.

Retention and Cleanup

DLQs need retention policies. Keeping every failed message forever creates storage cost, privacy risk, and operational noise.

Retention should consider:

• Business recovery windows.
• Legal or audit requirements.
• Message sensitivity.
• Availability of authoritative source data.
• Expected investigation time.

Resolved records may be archived or deleted after a defined period.

DELETE FROM failed_messages
WHERE replay_status IN ('completed', 'discarded')
  AND last_failed_at < NOW() - INTERVAL '90 days';

Before deleting unresolved messages, the system should produce an alert or require explicit approval. Automatic expiration without monitoring can silently remove the only copy of failed work.

Real-World Order Processing Example

Consider an order-processing service that consumes order.confirmed events and creates shipments through an external carrier API.

The consumer must handle:

• Temporary carrier outages.
• Provider rate limits.
• Invalid addresses.
• Duplicate message delivery.
• Unknown shipment-creation outcomes.
• Replay after a defect is fixed.

The incoming message contains a stable shipment-operation ID.

order_message = {
    "message_id": "msg-8f91",
    "message_type": "order.confirmed",
    "message_version": 1,
    "order_id": "order-7102",
    "shipment_operation_id": "shipment-order-7102",
    "customer_id": "customer-44",
    "correlation_id": "request-a802"
}

The consumer first checks whether the shipment operation already completed.

def create_order_shipment(message: dict) -> None:
    operation_id = message["shipment_operation_id"]

    existing_shipment = shipments.find_by_operation_id(
        operation_id
    )

    if existing_shipment is not None:
        return

    order = orders.find(message["order_id"])

    if order is None:
        raise PermanentMessageError(
            "Order does not exist"
        )

    if not order.shipping_address.is_valid():
        raise PermanentMessageError(
            "Shipping address is invalid"
        )

    create_shipment_with_provider(
        order=order,
        operation_id=operation_id
    )

The provider call uses the operation ID as an idempotency key.

def create_shipment_with_provider(
    order,
    operation_id: str
) -> None:
    try:
        response = carrier_api.create_shipment(
            address=order.shipping_address,
            parcels=order.parcels,
            idempotency_key=operation_id
        )

    except CarrierRateLimitError as error:
        raise RetryAfterError(
            retry_after_seconds=error.retry_after_seconds
        )

    except CarrierTimeoutError:
        existing = carrier_api.find_by_idempotency_key(
            operation_id
        )

        if existing is None:
            raise TransientMessageError(
                "Shipment result is unknown"
            )

        response = existing

    with database.transaction():
        shipments.insert_if_missing({
            "shipment_id": response.shipment_id,
            "shipment_operation_id": operation_id,
            "order_id": order.order_id,
            "tracking_number": response.tracking_number
        })

        orders.mark_shipment_created(
            order_id=order.order_id
        )

The top-level consumer classifies failures.

def consume_order_message(message: dict) -> None:
    try:
        create_order_shipment(message)

        message_queue.acknowledge(message)

    except RetryAfterError as error:
        schedule_retry_with_delay(
            message=message,
            delay_seconds=error.retry_after_seconds
        )

    except TransientMessageError as error:
        schedule_retry_with_backoff(
            message=message,
            error=error
        )

    except PermanentMessageError as error:
        send_to_dead_letter_queue(
            message=message,
            reason=str(error),
            failure_type="permanent"
        )

        message_queue.acknowledge(message)

    except Exception as error:
        schedule_unknown_failure_retry(
            message=message,
            error=error
        )

The retry scheduler applies bounded exponential backoff.

def schedule_retry_with_backoff(
    message: dict,
    error: Exception
) -> None:
    retry_message = create_retry_message(
        message=message,
        error=error
    )

    attempt = retry_message["retry_metadata"]["attempt"]
    maximum_attempts = 6

    if attempt >= maximum_attempts:
        send_to_dead_letter_queue(
            message=retry_message,
            reason=str(error),
            failure_type="retry_exhausted"
        )

        message_queue.acknowledge(message)
        return

    delay_seconds = calculate_backoff_with_jitter(
        attempt
    )

    message_queue.publish_delayed(
        queue="order-shipment-retries",
        payload=retry_message,
        delay_seconds=delay_seconds
    )

    message_queue.acknowledge(message)

Invalid addresses move directly to the DLQ. A support or operations tool displays the failed message, address validation result, order details, and failure reason. After the address is corrected, the message can be replayed using the same shipment-operation ID.

Carrier outages produce retries. If the outage continues beyond the retry limit, the message moves to the DLQ and creates an operational alert. Bulk replay begins only after carrier health recovers.

Useful monitoring for this workflow includes:

• Shipment queue depth.
• Oldest queued message age.
• Carrier timeout rate.
• Rate-limit response rate.
• Retry count by attempt number.
• Invalid-address DLQ count.
• Replay success rate.

Common Mistakes

Production Checklist

• Classify failures: distinguish transient, permanent, and unknown errors.
• Use bounded retries: define a maximum number of attempts.
• Use maximum message age: stop retrying obsolete work.
• Add exponential backoff: allow dependencies time to recover.
• Add jitter: prevent synchronised retry spikes.
• Respect provider retry delays: honour rate-limit guidance where available.
• Configure policies per workload: avoid one global retry rule.
• Use stable message IDs: support tracing and deduplication.
• Use stable operation IDs: protect real business effects.
• Reuse idempotency keys: preserve the same key across retries and replay.
• Commit deduplication and business updates together: avoid inconsistent state.
• Handle uncertain external outcomes: query provider state before repeating operations.
• Configure dead-letter queues: isolate unrecoverable messages.
• Store source metadata: preserve queue, topic, consumer, and routing details.
• Preserve failure context: include reason, attempt count, and timestamps.
• Protect sensitive data: redact or encrypt failed payloads.
• Monitor DLQ depth: detect accumulating failures.
• Monitor arrival rate: detect sudden regressions.
• Monitor oldest-message age: prevent unresolved failures from being forgotten.
• Group failures by signature: identify common causes efficiently.
• Define operational ownership: assign responsibility for each DLQ.
• Document investigation procedures: make diagnosis repeatable.
• Test replay in small batches: verify the fix before bulk redrive.
• Rate-limit bulk replay: protect consumers and dependencies.
• Pause replay on repeated failure: avoid recreating incidents.
• Preserve ordering where required: use partitions, versions, or sequence checks.
• Define retention: archive or delete resolved failures safely.
• Alert before unresolved expiration: prevent silent loss.
• Test poison-message scenarios: confirm retry limits and DLQ routing.
• Test consumer crashes: verify idempotency after redelivery.
• Test dependency outages: validate backoff and recovery behaviour.
• Test provider timeouts: confirm uncertain outcomes do not duplicate effects.

Conclusion

Retries, dead-letter queues, and poison-message handling are core parts of reliable asynchronous processing. Retries recover from temporary failures, but uncontrolled retries can amplify outages, waste worker capacity, and create duplicate business effects. Every retry policy should therefore be bounded, delayed appropriately, and tied to a clear failure classification.

Poison messages must be isolated before they repeatedly disrupt healthy processing. A dead-letter queue provides that isolation, but it does not resolve the underlying problem. Failed messages still require monitoring, investigation, remediation, replay, or deliberate closure.

Safe replay depends on idempotency. A retried or redriven message must retain the same business-operation identity so that duplicate delivery cannot create duplicate charges, shipments, rewards, or state transitions. External API calls require additional care because a timeout may leave the final result unknown.

A production-ready design treats DLQs as active operational workflows rather than forgotten storage. Queue depth, failure reasons, oldest-message age, replay success, and repeated failure rates should be visible and owned by a responsible team.

Key Takeaway

Retries should recover temporary failures, dead-letter queues should isolate unrecoverable messages, and idempotency should ensure that every retry or replay remains safe.

More Articles to Read

• Event-Driven Architecture in Distributed Systems
• Message Delivery Guarantees: At-Most-Once vs At-Least-Once vs Exactly-Once
• Background Workers Explained: Designing Reliable Asynchronous Processing
• Transactional Outbox Pattern for Reliable Messaging
• Message Queue Best Practices for Production Systems