Must-Known Message Broker Patterns

Message brokers are more than infrastructure for moving messages between services. Production systems depend on a set of recurring message broker patterns that determine how work is distributed, how events are broadcast, how failures are retried, how duplicates are handled, and how services remain consistent.

The most important patterns include Point-to-Point, Publish/Subscribe, Competing Consumers, Request-Reply, Dead Letter Queue, Retry with Backoff, Idempotent Consumer, Transactional Outbox, Saga, Message Deduplication, Priority Queue, delayed delivery, and message ordering. Understanding these patterns makes it much easier to design reliable asynchronous systems.

Table of Contents

• Message Broker Patterns at a Glance
• 1. Point-to-Point
• 2. Publish/Subscribe
• 3. Competing Consumers
• 4. Request-Reply
• 5. Dead Letter Queue
• 6. Retry with Backoff
• 7. Idempotent Consumer
• 8. Message Deduplication
• 9. Transactional Outbox
• 10. Saga Pattern
• 11. Priority Queue
• 12. Delayed Messages
• 13. Message Ordering
• 14. Consumer Groups and Partitioning
• 15. Fan-Out Processing
• 16. Poison Message Handling
• Combining Message Broker Patterns
• Production Design Example
• Common Message Broker Mistakes
• Frequently Asked Questions
  • What Is the Difference Between a Queue and Pub/Sub?
  • Do Message Brokers Guarantee Exactly-Once Processing?
  • When Should a Dead Letter Queue Be Used?
  • Should Every Consumer Be Idempotent?
• Conclusion
  • More Articles to Read

Message Broker Patterns at a Glance

Different patterns solve different messaging problems. Some define how messages are routed, while others address reliability, scalability, consistency, or failure recovery.

Production systems usually combine several of these patterns rather than choosing only one.

1. Point-to-Point

The Point-to-Point pattern sends a message to a queue where one consumer processes it.

Even when several consumers are listening to the queue, a particular message should normally be processed by only one of them.

Consider an image-processing system:

Upload Service
      ↓
image-processing
      ↓
Image Worker

The producer submits:

{
  "job_id": "job-9182",
  "image_id": "img-52",
  "operation": "generate_thumbnails"
}

Only one worker needs to generate the thumbnails.

Point-to-Point messaging is commonly used for:

• background jobs;
• email delivery;
• image processing;
• video transcoding;
• report generation;
• asynchronous commands.

The key idea is:

One message → One successful processor

2. Publish/Subscribe

The Publish/Subscribe pattern distributes one event to multiple independent subscribers.

The publisher does not need to know how many subscribers exist.

For example, an Order service can publish:

{
  "event_type": "OrderCreated",
  "order_id": "order-8472",
  "customer_id": "customer-91"
}

Different systems react independently.

The Inventory service reserves products. Analytics records the sale. Notification sends confirmation. Fraud Detection evaluates the transaction.

Adding another subscriber does not require modifying the Order service.

This loose coupling is one of the foundations of Event-Driven Architecture in Distributed Systems.

3. Competing Consumers

A single consumer may not process messages quickly enough. The Competing Consumers pattern allows several workers to consume from the same queue. Each message is assigned to one worker.

Suppose one worker handles 100 jobs per second but incoming traffic reaches 800 jobs per second.

Instead of redesigning the producer, more workers can be added:

1 worker   → ~100 jobs/sec
4 workers  → ~400 jobs/sec
8 workers  → ~800 jobs/sec

Actual scaling depends on database contention, external APIs, CPU, network bandwidth, broker throughput, and other shared dependencies, so throughput is not always perfectly linear.

This pattern is commonly used by background workers and job-processing systems. A deeper production treatment is available in Background Workers Explained: Designing Reliable Asynchronous Processing.

4. Request-Reply

Messaging is usually associated with asynchronous one-way communication, but sometimes a producer needs a response. The Request-Reply pattern uses messages in both directions.

The request includes a correlation ID:

{
  "message_id": "msg-123",
  "correlation_id": "req-991",
  "reply_to": "payment-replies",
  "command": "CheckPaymentStatus",
  "payment_id": "payment-52"
}

The response carries the same correlation ID:

{
  "correlation_id": "req-991",
  "payment_id": "payment-52",
  "status": "CAPTURED"
}

The caller matches the response with the original request.

Request-Reply is useful when messaging infrastructure is already central to the architecture, but it introduces timeouts, correlation state, reply routing, and cleanup.

For ordinary synchronous service calls, HTTP or gRPC may remain simpler.

5. Dead Letter Queue

A message may continue failing after repeated delivery attempts.

The Dead Letter Queue, or DLQ, isolates those messages instead of retrying them forever.

Main Queue
    ↓
 Consumer
    ↓
 Processing fails
    ↓
 Retry limit reached
    ↓
   DLQ

Examples include:

• invalid payloads;
• unsupported schema versions;
• deleted referenced entities;
• unexpected business-state conflicts;
• permanent downstream rejection.

A DLQ should not become a place where failed messages disappear unnoticed.

Production systems should monitor:

• DLQ message count;
• rate of new dead-lettered messages;
• oldest message age;
• failure reason;
• source queue or topic;
• number of processing attempts.

The complete failure-handling workflow is covered in Dead-Letter Queues, Retries, and Poison Messages.

6. Retry with Backoff

Many message-processing failures are temporary.

Examples include:

Database timeout
HTTP 503
Rate limit
Network interruption
Temporary broker failure

Immediately retrying can make the failure worse.

Failure
 ↓
Retry
 ↓
Failure
 ↓
Retry
 ↓
Failure
 ↓
Retry...

A better approach is exponential backoff.

Attempt 1 → wait 1 second
Attempt 2 → wait 2 seconds
Attempt 3 → wait 4 seconds
Attempt 4 → wait 8 seconds
Attempt 5 → wait 16 seconds

Jitter can randomize the delay so thousands of consumers do not retry simultaneously.

import random

def retry_delay(attempt: int) -> float:
    base = min(2 ** attempt, 60)
    return base + random.uniform(0, base * 0.25)

Retries should also be bounded.

Message
  ↓
Attempt
  ↓ failure
Retry with backoff
  ↓ failure
Retry with backoff
  ↓ failure
DLQ

Retry behavior should distinguish transient failures from permanent failures. Retrying malformed JSON for six hours will not make the JSON valid.

7. Idempotent Consumer

Distributed messaging frequently uses at-least-once delivery, which means the same message can arrive multiple times.

Broker
  ↓
Consumer processes message ✓
  ↓
Consumer crashes before ACK
  ↓
Broker redelivers message
  ↓
Consumer receives it again

If the operation is:

Charge credit card $100

processing the message twice can be disastrous.

An Idempotent Consumer makes repeated processing of the same logical message safe.

One common technique is storing processed message IDs:

BEGIN;

INSERT INTO processed_messages (message_id)
VALUES ('msg-8472')
ON CONFLICT DO NOTHING;

-- Apply business operation only when
-- the message was inserted successfully.

COMMIT;

Another approach is designing the operation itself to be idempotent.

Set shipment status = SHIPPED

is naturally easier to retry than:

Increment shipped_count by 1

Idempotency is one of the most important assumptions when designing reliable consumers.

8. Message Deduplication

Message Deduplication identifies repeated logical messages and prevents them from producing repeated effects.

Messages should carry a stable identifier:

{
  "message_id": "msg-551",
  "event_type": "PaymentCaptured",
  "payment_id": "payment-19"
}

The consumer can check whether that message has already been processed.

Message arrives
     ↓
Was msg-551 processed?
     │
   ┌─┴─┐
  yes  no
   ↓    ↓
Ignore Process

Deduplication and idempotency are related but not identical.

Deduplication detects repeated messages. Idempotency ensures repeated execution does not change the final result incorrectly.

Systems can use either or both depending on the operation.

9. Transactional Outbox

A service often needs to update its database and publish an event.

Create Order
   ↓
INSERT order
   ↓
Publish OrderCreated

The dangerous case is:

Database COMMIT ✓
       ↓
Application crashes
       ↓
Publish event ✗

The order exists, but other services never learn about it.

The Transactional Outbox stores the business change and outgoing event in the same database transaction.

BEGIN;

INSERT INTO orders (
    id,
    customer_id,
    status
)
VALUES (
    'order-8472',
    'customer-91',
    'PENDING'
);

INSERT INTO outbox (
    id,
    event_type,
    aggregate_id,
    payload
)
VALUES (
    'evt-991',
    'OrderCreated',
    'order-8472',
    '{"order_id":"order-8472"}'
);

COMMIT;

A separate publisher later delivers the outbox event.

Application
    ↓
Database Transaction
    ├── Order
    └── Outbox Event
           ↓
     Outbox Publisher
           ↓
         Broker

If the broker is unavailable, the event remains durable and can be retried.

The pattern is explained in detail in Transactional Outbox Pattern for Reliable Messaging.

10. Saga Pattern

The Saga Pattern coordinates a business transaction that spans multiple services without requiring one distributed database transaction.

Instead of locking resources across every service, a saga breaks the workflow into a sequence of local transactions. Each service commits its own change and then triggers the next step, often through messages or events.

Consider an order workflow:

Create Order
     ↓
Reserve Inventory
     ↓
Capture Payment
     ↓
Create Shipment
     ↓
Order Completed

Each step belongs to a different service and database. If a later step fails, the saga executes compensating actions for previously completed operations.

For example, if shipment creation fails after payment and inventory reservation:

Create Shipment ✗
       ↓
Refund Payment
       ↓
Release Inventory
       ↓
Cancel Order

Compensation is not the same as rolling back a database transaction. Each compensating action is another business operation that can fail, be retried, and require idempotent processing.

Sagas commonly use one of two coordination approaches:

• Choreography — services publish events and react to events from other services without a central coordinator.
• Orchestration — a dedicated saga orchestrator tracks workflow state and explicitly sends commands for the next step.

Message brokers are frequently used to transport saga commands and events, but reliable messaging is critical. Losing an event such as PaymentCaptured can leave the workflow permanently incomplete.

For this reason, Saga is commonly combined with the Transactional Outbox, retries, idempotent consumers, and dead-letter handling.

The key distinction is:

Saga
→ coordinates a distributed business workflow

Transactional Outbox
→ reliably publishes messages after local transactions

Idempotency
→ makes retries and duplicate delivery safe

DLQ
→ isolates messages that cannot be processed

Sagas are especially useful for workflows such as order processing, payments, travel booking, fulfillment, and other operations where one business transaction crosses multiple independently owned services.

11. Priority Queue

Not every message has equal business importance.

A Priority Queue allows urgent messages to be processed before lower-priority work.

For example:

Password reset email → High
Purchase receipt      → Normal
Weekly report         → Low

One implementation uses broker-native message priorities. Another uses separate queues:

jobs-high
jobs-normal
jobs-low

Separate queues can provide clearer capacity controls and isolation.

Priority requires care because constant high-priority traffic can starve low-priority messages.

Systems may need quotas, weighted processing, or dedicated worker pools to guarantee progress for every class of work.

12. Delayed Messages

Some messages should not be processed immediately.

Examples include:

• retrying a failed operation in five minutes;
• sending a reminder tomorrow;
• expiring an unpaid order after 30 minutes;
• checking a long-running job later;
• executing scheduled background work.

The pattern looks like:

Producer
   ↓
Delayed Message
   ↓
Wait
   ↓
Available Queue
   ↓
Consumer

A message might contain:

{
  "type": "ExpireOrder",
  "order_id": "order-8472",
  "execute_after": "2026-09-27T18:30:00Z"
}

Depending on the broker, delayed processing can use native scheduling, message TTLs, delay queues, or an external scheduler.

Long delays deserve special consideration. Keeping millions of messages inside a broker for months may be less practical than storing schedules durably in a database and publishing them when they become due.

13. Message Ordering

Some workflows require messages to be processed in a specific order.

Consider an order lifecycle:

OrderCreated
     ↓
PaymentCaptured
     ↓
OrderShipped
     ↓
OrderDelivered

Processing OrderShipped before OrderCreated may break the consumer.

Global ordering is expensive and often unnecessary.

A more scalable requirement is usually:

Preserve ordering for one order_id

Messages can use the aggregate ID as the partition key:

order-100 → Partition 1
order-101 → Partition 3
order-102 → Partition 2

All events for order-100 then follow the same ordered path.

Ordering must be designed end-to-end. A broker can preserve partition order while parallel application processing later reorders the actual side effects.

Kafka-specific ordering and deduplication trade-offs are covered in Kafka Ordering Guarantees and Message Deduplication.

14. Consumer Groups and Partitioning

Stream-oriented brokers commonly distribute work using partitions and consumer groups.

Topic
 ├── Partition 0 → Consumer A
 ├── Partition 1 → Consumer B
 ├── Partition 2 → Consumer C
 └── Partition 3 → Consumer A

Consumers inside the same group cooperate to process the stream.

A partition is normally assigned to only one consumer in the group at a time, which provides a useful combination of parallelism and per-partition ordering.

Suppose a topic has 12 partitions.

3 consumers  → ~4 partitions each
6 consumers  → ~2 partitions each
12 consumers → ~1 partition each
20 consumers → 8 consumers idle

Increasing consumer count beyond the available partitions does not increase parallel consumption for that group.

Partition-key selection is therefore an important scaling decision. A key with highly uneven traffic can create a hot partition even when other consumers have spare capacity.

15. Fan-Out Processing

The Fan-Out pattern sends one event into several independent processing pipelines.

Consider an uploaded video:

VideoUploaded
      │
      ├──→ Transcoding
      ├──→ Thumbnail Generation
      ├──→ Content Moderation
      └──→ Analytics

Each workflow has its own queue and scaling characteristics.

This differs from competing consumers.

Competing Consumers:

Message
   ↓
Worker A OR Worker B OR Worker C


Fan-Out:

Message
   ├──→ Pipeline A
   ├──→ Pipeline B
   └──→ Pipeline C

Fan-out is useful when several independent actions must react to the same event without coupling those actions to the producer.

Failures are isolated as well. A slow analytics pipeline does not necessarily need to delay thumbnail generation.

16. Poison Message Handling

A poison message is a message that repeatedly fails processing because something about the message or application state makes successful processing impossible.

For example:

{
  "event_type": "PaymentCaptured",
  "schema_version": 999
}

If the consumer supports only schema versions 1 and 2, retrying the message every second will not solve the problem.

Without poison-message handling:

Bad message
    ↓
Fail
    ↓
Retry
    ↓
Fail
    ↓
Retry forever

This wastes resources and can block healthy messages when strict ordering is required.

A better workflow is:

Message
   ↓
Process
   ↓ failure
Classify failure
   │
   ├── Transient → Retry with backoff
   │
   └── Permanent → DLQ

Poison-message handling therefore combines several patterns: bounded retries, failure classification, dead-lettering, observability, and controlled replay.

Combining Message Broker Patterns

Real production systems rarely use these patterns independently.

Consider order processing.

Create Order
     ↓
Transactional Outbox
     ↓
OrderCreated
     ↓
Saga
     ↓
Publish/Subscribe
     │
     ├──→ Inventory Queue
     │       ↓
     │   Competing Consumers
     │       ↓
     │   Retry + Backoff
     │       ↓
     │      DLQ
     │
     ├──→ Payment Queue
     │
     └──→ Notification Queue

The Transactional Outbox ensures the initial event is not lost after the order commits.

The Saga coordinates the business workflow across Order, Inventory, Payment, and other services.

Publish/Subscribe distributes events to independent domains.

Competing Consumers scale processing.

Idempotent Consumers make redelivery safe.

Retry with Backoff handles temporary failures.

The DLQ isolates messages that still cannot be processed.

Ordering can preserve the sequence of events for the same order.

Each pattern addresses a different failure, consistency, or scalability problem.

Production Design Example

Consider an e-commerce platform processing thousands of orders per second.

An Order service stores an order and its outgoing event atomically:

POST /orders
     ↓
Order Service
     ↓
Database Transaction
     ├── orders
     └── outbox

An outbox publisher sends OrderCreated to the broker.

The order then enters a distributed workflow:

OrderCreated
     ↓
Reserve Inventory
     ↓
Capture Payment
     ↓
Create Shipment
     ↓
Confirm Order

Each service commits its own local transaction. Messages coordinate progress between the steps.

If payment fails after inventory has already been reserved, the saga compensates:

Capture Payment ✗
       ↓
Release Inventory
       ↓
Cancel Order

Events from the workflow can also be distributed to independent subscribers:

                   OrderConfirmed
                         │
            ┌────────────┼────────────┐
            ↓            ↓            ↓
       Notification   Analytics   Fulfillment

Inventory traffic is high, so eight competing consumers process its queue.

Inventory Queue
       │
 ┌─────┼─────┬───── ... ─────┐
 ↓     ↓     ↓               ↓
W1    W2    W3              W8

Every message contains a stable message ID:

{
  "message_id": "msg-991",
  "event_type": "ReserveInventory",
  "order_id": "order-8472",
  "attempt": 1
}

A worker tries to reserve inventory.

If the inventory database experiences a temporary timeout:

Attempt 1
   ↓ timeout
Wait 2s

Attempt 2
   ↓ timeout
Wait 4s

Attempt 3
   ↓ success
ACK

If processing continues failing after the configured limit:

Inventory Queue
      ↓
Retries exhausted
      ↓
inventory-dlq

The DLQ record keeps diagnostic metadata:

{
  "message_id": "msg-991",
  "order_id": "order-8472",
  "attempts": 5,
  "failure_type": "InventoryValidationError",
  "source": "inventory-reservation"
}

Consumers store processed message IDs so broker redelivery does not reserve inventory, capture a payment, or execute a compensating action twice.

Events use order_id as the ordering key when sequence matters:

order-8472:

OrderCreated
     ↓
InventoryReserved
     ↓
PaymentCaptured
     ↓
OrderConfirmed

Notification traffic uses a different scaling policy because sending emails is constrained by an external provider.

The notification workers therefore apply rate limiting and delayed retries rather than simply increasing consumer count.

The architecture uses different patterns at different points because each boundary has different reliability, consistency, and scaling requirements.

Useful production metrics include:

• queue depth;
• oldest message age;
• consumer throughput;
• consumer lag;
• processing latency;
• retry rate;
• DLQ growth;
• duplicate detection rate;
• outbox backlog;
• saga completion latency;
• failed compensations;
• stuck saga count;
• message publication failures;
• per-partition traffic distribution.

Queue depth alone is not enough. A queue containing 50,000 messages may be healthy if consumers process 100,000 messages per second, while a queue containing only 500 messages can represent a serious outage if its oldest message has been waiting for an hour.

Common Message Broker Mistakes

• Assuming message delivery means successful processing. Delivery, acknowledgment, and business completion are different events.
• Assuming messages are delivered only once. Many reliable systems intentionally prefer redelivery over message loss.
• Using unlimited retries. Permanent failures can consume resources forever.
• Retrying immediately. Aggressive retries can overload an already failing dependency.
• Using a DLQ without monitoring it. Failed messages become invisible production failures.
• Using non-idempotent consumers. Redelivery can create duplicate payments, emails, reservations, or state transitions.
• Treating saga compensation like a database rollback. Compensations are separate business operations and can fail independently.
• Keeping saga state only in memory. A process restart can lose the workflow state needed to continue or compensate.
• Assuming global ordering is necessary. Per-entity or per-partition ordering is usually more scalable.
• Publishing after a database commit without an outbox or equivalent reliability mechanism. A crash between the two operations can permanently lose the event.
• Scaling consumers without checking downstream capacity. More workers can overload databases and external APIs.
• Using one queue for unrelated workloads. Slow or failing jobs can interfere with latency-sensitive work.
• Ignoring message schema evolution. Producers and consumers often deploy independently.
• Monitoring only broker health. A healthy broker does not mean consumers or distributed workflows are processing messages correctly.

Frequently Asked Questions

Message broker terminology often overlaps because routing, delivery guarantees, consumer behavior, distributed workflows, and failure recovery are separate concerns that work together.

What Is the Difference Between a Queue and Pub/Sub?

A queue normally distributes each message to one successful consumer, while Publish/Subscribe distributes an event to multiple independent subscribers.

Queue:

Message → Consumer A OR B OR C


Pub/Sub:

Event → Subscriber A
      → Subscriber B
      → Subscriber C

The right choice depends on whether the message represents a unit of work that should be performed once or an event that several systems may need to observe.

Do Message Brokers Guarantee Exactly-Once Processing?

Not automatically. Broker-level delivery guarantees and business-level processing guarantees are different things.

A consumer can complete a database transaction and crash before acknowledging the message. The broker may then deliver it again.

For that reason, production consumers frequently assume duplicate delivery is possible even when the messaging platform provides stronger guarantees in specific parts of the pipeline. The trade-offs are covered in Message Delivery Guarantees: At-Most-Once vs At-Least-Once vs Exactly-Once.

When Should a Dead Letter Queue Be Used?

A DLQ is useful when a message cannot be processed successfully after the permitted retry strategy or when the failure is known to be permanent.

Messages in the DLQ should retain enough context for investigation and controlled replay. DLQ growth should also trigger monitoring or alerts rather than relying on manual inspection.

Should Every Consumer Be Idempotent?

Consumers performing important state changes should generally be designed under the assumption that messages can be delivered more than once.

Some operations are naturally idempotent, while others require message IDs, unique constraints, idempotency keys, or transactional deduplication. The appropriate mechanism depends on the business effect being protected.

Conclusion

Reliable message-driven systems are built from multiple patterns rather than from a broker alone. Point-to-Point and Publish/Subscribe define how messages are distributed. Competing Consumers and partitioning provide scalability. Retries, DLQs, and poison-message handling provide failure recovery. Idempotency and deduplication make redelivery safe. Transactional Outbox connects database transactions reliably to asynchronous messaging. Saga coordinates business workflows that span multiple services.

The patterns should be selected according to concrete requirements around delivery, ordering, latency, scalability, consistency, and failure recovery.

The core principle is: assume failures, retries, duplicates, delays, partial processing, and compensation are normal parts of asynchronous communication, then design message flows so those conditions are safe and observable.

More Articles to Read

• Message Queues Explained: Producers, Consumers, and Brokers
• Message Queue Best Practices for Production Systems
• Kafka Reliability: Retries, Dead Letter Topics, and Failure Handling