Message Queues Explained: Producers, Consumers, and Brokers

Modern applications often need to perform work that should not block the main request. An order service may need to reserve inventory, send a confirmation email, update analytics, and notify a warehouse. Executing every operation synchronously increases response time and creates direct dependencies between services.

A message queue separates the component that creates work from the component that processes it. Producers publish messages, brokers store and route them, and consumers process them independently. This model improves scalability and resilience, but it also introduces important decisions around ordering, acknowledgements, retries, concurrency, and failure handling.

Table of Contents

• What Is a Message Queue?
• Why Message Queues Exist
• Core Components
  • Producer
  • Broker
  • Consumer
• Message Lifecycle
  • Publishing Messages
  • Storing and Routing
  • Consuming Messages
  • Acknowledgements
• Queues vs Topics
  • Queue Model
  • Topic Model
• Push vs Pull Consumers
  • Push Consumption
  • Pull Consumption
• Message Ordering
• Scaling Consumers
• Real-World Example: Order Processing
• Popular Message Queue Technologies
• When to Use Message Queues
• Common Mistakes
• Production Checklist
• Conclusion
• Key Takeaway
• More Articles to Read

What Is a Message Queue?

A message queue is an infrastructure component that accepts messages from one application component and makes them available to another component for later processing. The producer does not need to wait for the consumer to finish the work.

A message usually represents a command, task, notification, or event. Examples include:

• Process order ORD-10482.
• Send a password reset email.
• Generate a monthly invoice.
• Resize an uploaded image.
• Record that a payment was completed.

The broker acts as an intermediary. It receives the message, stores it temporarily or durably, and delivers it according to the configured routing and consumption model.

# Producer publishes work instead of processing it directly.
message = {
    "message_id": "msg-8b73c1",
    "type": "order.created",
    "order_id": "ORD-10482",
    "customer_id": "CUS-742",
    "created_at": "2026-07-22T18:10:00Z"
}

broker.publish(
    queue="order-processing",
    message=message
)

The application handling the customer request can return quickly after the broker accepts the message. A background consumer processes the order independently.

Why Message Queues Exist

Without a queue, services commonly communicate through synchronous network requests. A request may pass through several services before it can complete.

# Synchronous request chain.
def create_order(order):
    inventory.reserve(order.items)
    payment.authorize(order.payment)
    email.send_confirmation(order.customer_email)
    analytics.record_order(order)

    return {"status": "created"}

This design appears simple, but the order endpoint now depends on inventory, payment, email, and analytics. A slow email provider may delay the customer response. An analytics outage may cause order creation to fail even though analytics is not part of the critical business transaction.

A queue allows non-critical or independently scalable work to run asynchronously.

# The request performs the critical transaction and publishes follow-up work.
def create_order(order):
    saved_order = database.save_order(order)

    broker.publish(
        queue="order-events",
        message={
            "type": "order.created",
            "order_id": saved_order.id
        }
    )

    return {
        "status": "accepted",
        "order_id": saved_order.id
    }

Message queues provide several practical benefits:

• Decoupling: producers do not need direct knowledge of every consumer.
• Traffic buffering: temporary spikes can accumulate in the queue instead of overwhelming workers.
• Independent scaling: producers and consumers can scale separately.
• Failure isolation: a consumer outage does not always stop producers from accepting work.
• Asynchronous processing: long-running tasks can execute outside the request path.

Queues do not eliminate failures. They move failures into an asynchronous workflow that must be monitored and managed explicitly.

Core Components

Most messaging systems are built around three roles: producer, broker, and consumer. A single application may perform more than one role, but the responsibilities remain distinct.

Producer

A producer creates and publishes messages. It may be an API service, scheduled job, database change processor, command-line tool, or another consumer.

A producer is responsible for creating a message that contains enough information for downstream processing. The message should normally include a stable identifier, type, creation timestamp, and business reference.

from uuid import uuid4
from datetime import datetime, timezone


def build_invoice_message(invoice_id: str) -> dict:
    return {
        "message_id": str(uuid4()),
        "type": "invoice.generate",
        "invoice_id": invoice_id,
        "created_at": datetime.now(timezone.utc).isoformat(),
        "schema_version": 1
    }

Advantages

• Publishes work without waiting for processing to finish.
• Can remain independent of consumer implementation details.
• Can handle traffic bursts by relying on the broker as a buffer.

Disadvantages

• Successful publication does not necessarily mean successful processing.
• Producer code must handle broker unavailability and publication failures.
• Message schema changes require compatibility planning.

When to Use

Use a producer whenever an application needs to submit asynchronous work or announce a business event to other components.

Real-World Use Cases

• An order API publishing an order.created message.
• A file service publishing an image-processing task.
• A billing service publishing an invoice-generation command.

Example

def submit_video_transcoding(video_id: str) -> None:
    broker.publish(
        queue="video-transcoding",
        message={
            "message_id": str(uuid4()),
            "type": "video.transcode",
            "video_id": video_id,
            "output_format": "mp4"
        }
    )

Broker

The broker receives messages from producers and makes them available to consumers. Depending on the technology, the broker may provide durable storage, routing rules, partitions, acknowledgements, access control, retention policies, and replication.

The broker also absorbs differences between production and consumption rates. If producers publish 5,000 messages per second while consumers process 3,000, the broker temporarily stores the difference.

Advantages

• Separates producers from consumers.
• Buffers traffic spikes.
• Can provide durable storage and replicated delivery infrastructure.
• Supports routing, filtering, partitions, and consumer coordination.

Disadvantages

• Becomes critical infrastructure that requires monitoring and capacity planning.
• Can introduce operational complexity and additional latency.
• Incorrect retention or acknowledgement settings may cause message loss or backlog growth.

When to Use

Use a broker when multiple components need asynchronous communication, traffic buffering, independent scaling, or durable message delivery.

Real-World Use Cases

• Distributing jobs across background workers.
• Streaming domain events between services.
• Buffering telemetry before storage and analysis.

Example

A broker may contain several queues for different workloads:

queues = {
    "payments": {
        "workers": 20,
        "retention_hours": 24
    },
    "emails": {
        "workers": 8,
        "retention_hours": 72
    },
    "image-processing": {
        "workers": 12,
        "retention_hours": 48
    }
}

Consumer

A consumer retrieves or receives messages and performs the associated work. Consumers may update a database, call an external API, generate a file, send a notification, or publish another message.

Consumers should assume that processing can fail. Database connections can time out, external services can return errors, and worker processes can stop after completing work but before acknowledging the message.

def handle_message(message: dict) -> None:
    if message["type"] != "invoice.generate":
        raise ValueError("Unsupported message type")

    invoice_id = message["invoice_id"]

    # The handler should be safe when processing is retried.
    invoice = database.get_invoice(invoice_id)

    if invoice.status == "generated":
        return

    document = generate_invoice_pdf(invoice)
    storage.upload(invoice_id, document)
    database.mark_invoice_generated(invoice_id)

Advantages

• Can scale independently from producers.
• Can isolate expensive or slow workloads from API servers.
• Can process messages concurrently across multiple worker instances.

Disadvantages

• Requires careful handling of duplicate processing and partial failures.
• Backlogs can grow when consumers are slower than producers.
• Consumer deployments must support graceful shutdown and safe acknowledgement behavior.

When to Use

Use consumers for background jobs, external integrations, scheduled processing, data pipelines, notifications, and other asynchronous work.

Real-World Use Cases

• Email delivery workers.
• Payment settlement processors.
• Search indexing consumers.
• Video transcoding workers.

Example

while True:
    messages = broker.receive(
        queue="email-delivery",
        max_messages=10,
        wait_seconds=20
    )

    for message in messages:
        try:
            send_email(message["recipient"], message["template"])
            broker.acknowledge(message)
        except TemporaryEmailError:
            # Leave the message unacknowledged so it can be retried.
            continue

Message Lifecycle

A message typically passes through four stages: publication, broker storage and routing, consumption, and acknowledgement. Each stage can fail independently.

Publishing Messages

The producer serializes a message and sends it to the broker. The broker may confirm that it accepted the message.

A producer should not assume that calling a client library means the broker stored the message. Network failures can occur before or after the broker accepts it, leaving the producer uncertain about the result.

def publish_with_confirmation(message: dict) -> None:
    result = broker.publish(
        queue="order-processing",
        message=message,
        require_confirmation=True
    )

    if not result.confirmed:
        raise RuntimeError("Message publication was not confirmed")

Production systems usually apply timeouts, limited retries, structured logging, and stable message identifiers around publication.

Storing and Routing

After accepting a message, the broker determines where it should go. A simple system places every message into one queue. More advanced systems may route messages using message type, topic, routing key, tenant, priority, or partition key.

def route_message(message: dict) -> str:
    routes = {
        "payment.authorize": "payment-commands",
        "email.send": "email-delivery",
        "image.resize": "image-processing"
    }

    return routes[message["type"]]

The broker may store messages in memory, on disk, across replicated nodes, or in a distributed log. The durability and retention model depends on the selected technology and configuration.

Consuming Messages

The broker delivers a message to a consumer or allows a consumer to retrieve it. The message may become temporarily invisible to other consumers while one worker processes it.

For example, a worker may receive a message with a 60-second visibility timeout. If the worker does not acknowledge it within 60 seconds, the broker makes it available again.

message = broker.receive(
    queue="report-generation",
    visibility_timeout_seconds=60
)

generate_report(message["report_id"])
broker.acknowledge(message)

The visibility timeout must be longer than normal processing time or extended while long-running work continues. Otherwise, another worker may receive the same message before the first worker finishes.

Acknowledgements

An acknowledgement tells the broker that processing completed successfully. The broker can then delete the message, advance a consumer offset, or mark the record as completed.

The safest order is usually:

1. Receive the message.
2. Perform the business operation.
3. Persist the result.
4. Acknowledge the message.

def process_payment_message(message: dict) -> None:
    payment_id = message["payment_id"]

    process_payment(payment_id)
    database.mark_payment_processed(payment_id)

    # Acknowledge only after durable business state is saved.
    broker.acknowledge(message)

Acknowledging before processing creates a risk of message loss. If the worker crashes after acknowledging but before completing the operation, the broker considers the message finished.

Acknowledging after processing can result in duplicate delivery. If the worker completes the operation and crashes before acknowledging, the message may be delivered again. This is why reliable consumers often need idempotent processing.

Queues vs Topics

Queues and topics solve different distribution problems. A queue normally distributes work across competing consumers. A topic distributes the same event to multiple independent subscribers.

Queue Model

In a queue model, multiple workers compete for messages. Each message is normally processed by one worker from the consumer group.

Advantages

• Simple horizontal scaling across workers.
• Useful for distributing background jobs.
• Prevents every worker from processing the same task.

Disadvantages

• One queue does not automatically provide a copy to multiple independent systems.
• Global ordering becomes difficult when many workers process concurrently.

When to Use

Use a queue when one unit of work should be handled once by one worker from a worker pool.

Real-World Use Cases

• Generating invoices.
• Processing uploaded files.
• Sending individual emails.
• Running scheduled reports.

Example

# Three workers share the same queue.
workers = [
    Worker(queue="invoice-generation"),
    Worker(queue="invoice-generation"),
    Worker(queue="invoice-generation")
]

# Each invoice message is assigned to one available worker.

Topic Model

In a topic model, one published event may be delivered to multiple subscriptions. Each subscription represents an independent consumer or consumer group.

An order.created event may be consumed by inventory, analytics, notifications, and fraud-detection services.

Advantages

• Supports fan-out to multiple independent systems.
• Allows new subscribers to be added without changing the producer.
• Works well for domain events and integration events.

Disadvantages

• More subscribers create more operational dependencies and message traffic.
• Event schema evolution becomes increasingly important.
• Failures must be managed separately for every subscription.

When to Use

Use a topic when multiple systems need their own copy of the same event.

Real-World Use Cases

• Order events consumed by analytics and fulfillment.
• User registration events consumed by email and fraud systems.
• Shipment events consumed by tracking and customer-notification services.

Example

broker.publish(
    topic="orders",
    message={
        "type": "order.created",
        "order_id": "ORD-10482"
    }
)

# Independent subscriptions:
# - inventory-service
# - analytics-service
# - notification-service

Push vs Pull Consumers

Brokers deliver messages through either a push or pull model. Some systems support both.

Push Consumption

In the push model, the broker sends messages to a consumer endpoint as soon as they become available.

Advantages

• Low delivery latency.
• Consumers do not need to poll continuously.
• Works well with HTTP endpoints and serverless handlers.

Disadvantages

• The broker may overwhelm a slow consumer.
• Consumer endpoints must be reachable by the broker.
• Backpressure and retry behavior can be harder to control.

When to Use

Use push delivery for low-latency event notifications, webhooks, and managed serverless integrations where the broker controls invocation.

Real-World Use Cases

• Triggering a serverless function.
• Delivering webhook-style events.
• Sending notifications to an HTTP service.

Example

# Broker sends the message to this HTTP endpoint.
@app.post("/messages/orders")
def receive_order_event(message: dict):
    process_order_event(message)

    # A successful response confirms processing.
    return {"status": "ok"}

Pull Consumption

In the pull model, consumers request messages when they have available capacity.

Advantages

• Consumers control processing rate and batch size.
• Backpressure is easier to implement.
• Workers can use long polling to reduce empty requests.

Disadvantages

• Poor polling configuration may increase latency or broker requests.
• Consumers must manage polling loops and graceful shutdown.

When to Use

Use pull consumption for worker pools, batch processing, high-throughput pipelines, and workloads where consumers need direct control over capacity.

Real-World Use Cases

• Image-processing workers.
• Batch data transformation.
• Payment-processing services.

Example

while worker.is_running:
    messages = broker.receive(
        queue="image-processing",
        max_messages=20,
        wait_seconds=20
    )

    for message in messages:
        process_image(message)
        broker.acknowledge(message)

Message Ordering

Message ordering is often more limited than expected. A broker may preserve order within one queue or partition, but concurrency, retries, and multiple consumers can change the order in which processing completes.

Consider two account events:

1. account.created
2. account.suspended

If the events are processed by different workers, the suspension may complete before account creation. Systems that require ordering should route related messages using the same ordering or partition key.

broker.publish(
    topic="account-events",
    partition_key="ACC-9081",
    message={
        "type": "account.suspended",
        "account_id": "ACC-9081"
    }
)

Using the account ID as the partition key can preserve order for one account while allowing unrelated accounts to process concurrently.

Ordering has trade-offs. Sending every message through one partition provides stronger global ordering but limits throughput. Partitioning improves scalability but only guarantees ordering within each partition.

Consumers should also consider stale events. A message can be delayed and arrive after a newer update. Including an event version or sequence number allows the consumer to reject outdated state transitions.

def apply_account_event(event: dict) -> None:
    account = database.get_account(event["account_id"])

    if event["version"] <= account.last_event_version:
        # Ignore an old or duplicate event.
        return

    update_account(account, event)
    account.last_event_version = event["version"]
    database.save(account)

Scaling Consumers

Consumers usually scale horizontally. Multiple worker instances read from the same queue and divide the workload.

The required worker count depends on message arrival rate and average processing time. A simple estimate is:

# Estimated concurrency needed to keep up with traffic.
required_concurrency = messages_per_second * average_processing_seconds

# Example:
# 200 messages/second * 0.25 seconds = 50 concurrent workers.

This estimate does not include safety capacity, retry traffic, uneven processing time, or external service limits. Production systems normally include additional headroom.

Scaling should be based on useful signals such as:

• Number of visible messages.
• Age of the oldest unprocessed message.
• Message arrival rate.
• Consumer processing rate.
• Average and percentile processing duration.
• Error and retry rate.

Queue depth alone can be misleading. A backlog of 10,000 messages may be acceptable when workers process 20,000 messages per second. A backlog of 500 may be critical when the oldest message has waited for two hours.

Consumers must also respect downstream capacity. Increasing workers from 10 to 100 may overload a database or third-party API.

MAX_DATABASE_CONCURRENCY = 40

worker_pool = WorkerPool(
    queue="account-updates",
    concurrency=MAX_DATABASE_CONCURRENCY
)

Effective scaling balances queue latency with database connections, API rate limits, CPU, memory, and cost.

Real-World Example: Order Processing

Consider an e-commerce platform that receives an order through an HTTP API. The critical request path validates the request, stores the order, and publishes an order-processing message.

def create_order(request: dict) -> dict:
    order = database.insert_order(
        customer_id=request["customer_id"],
        items=request["items"],
        status="pending"
    )

    broker.publish(
        queue="order-processing",
        message={
            "message_id": str(uuid4()),
            "type": "order.process",
            "order_id": order.id
        }
    )

    return {
        "order_id": order.id,
        "status": "pending"
    }

A worker receives the message and coordinates the background work.

def process_order(message: dict) -> None:
    order = database.get_order(message["order_id"])

    if order.status == "completed":
        # Duplicate processing should not repeat the operation.
        return

    inventory.reserve(order.items)
    payment.authorize(order.payment_method, order.total)
    database.mark_order_completed(order.id)

    broker.publish(
        topic="order-events",
        message={
            "type": "order.completed",
            "order_id": order.id
        }
    )

Other services subscribe to order.completed:

• The notification service sends a confirmation email.
• The warehouse service begins fulfillment.
• The analytics service records the completed order.
• The loyalty service adds reward points.

This design removes several non-critical operations from the customer-facing request. It also allows each consumer to scale independently.

However, several failure scenarios still require explicit handling:

• The order is stored, but message publication fails.
• Inventory is reserved, but payment authorization fails.
• The worker completes processing but crashes before acknowledgement.
• The confirmation email provider is temporarily unavailable.
• A malformed message repeatedly fails.

Later articles in this series address these cases through delivery guarantees, idempotency, retries, dead-letter queues, and the transactional outbox pattern.

Popular Message Queue Technologies

Messaging technologies differ significantly in delivery model, ordering, throughput, operational complexity, and retention behavior. The correct choice depends on the workload rather than product popularity.

Examples of commonly used messaging systems include RabbitMQ, Apache Kafka, Amazon SQS, Amazon SNS, Google Cloud Pub/Sub, Azure Service Bus, NATS, and Redis Streams.

A traditional broker may be suitable for command-style work queues with acknowledgements and routing. A distributed log may be better for replayable event streams, high throughput, and long retention. A managed cloud queue may be preferable when reducing operational responsibility is more important than infrastructure portability.

When to Use Message Queues

Message queues are useful when work can be processed asynchronously or when production and consumption rates should be separated.

Strong use cases include:

• Long-running tasks that should not block an API request.
• Traffic spikes that need temporary buffering.
• Workloads that require independent consumer scaling.
• Communication between services that should not be tightly coupled.
• External integrations that may be slow or temporarily unavailable.
• Fan-out events consumed by multiple independent systems.

Queues are not automatically appropriate for every operation. A synchronous request may be simpler when the caller requires an immediate result, processing is fast, and failure must be returned directly.

For example, validating a password during login should usually remain synchronous. Generating a large export after login can run asynchronously.

Common Mistakes

Production Checklist

• Define message ownership: document which service publishes and which services consume each message type.
• Include stable identifiers: add message IDs and business entity IDs for tracing and deduplication.
• Version message schemas: allow producers and consumers to deploy independently.
• Set explicit timeouts: configure publication, polling, visibility, and processing timeouts.
• Acknowledge after durable completion: avoid deleting messages before business state is saved.
• Plan for duplicates: make important consumer operations idempotent.
• Separate workload types: avoid mixing slow jobs with latency-sensitive jobs in one queue.
• Configure bounded retries: distinguish temporary failures from permanent failures.
• Use dead-letter queues: isolate messages that cannot be processed successfully.
• Monitor message age: alert when the oldest message exceeds the expected processing delay.
• Track throughput: compare publishing rate with successful consumption rate.
• Protect downstream systems: limit concurrency according to database and API capacity.
• Support graceful shutdown: stop receiving new work and finish or release active messages safely.
• Secure broker access: restrict publishing and consumption permissions by queue or topic.
• Test failure scenarios: terminate workers during processing and verify recovery behavior.

Conclusion

Message queues provide a practical foundation for asynchronous processing in distributed systems. Producers create work, brokers buffer and route it, and consumers process it independently. This separation reduces direct service dependencies and allows each part of the system to scale according to its own workload.

The apparent simplicity of publishing and consuming messages hides several production concerns. Acknowledgements, visibility timeouts, ordering, duplicate delivery, backlog growth, and downstream capacity all affect reliability. A queue should therefore be treated as a core distributed-system component rather than a basic task list.

A well-designed messaging workflow defines clear message schemas, stable identifiers, safe acknowledgement behavior, bounded concurrency, monitoring, and failure handling from the beginning.

Key Takeaway

A message queue decouples the creation of work from its execution, but reliable messaging depends on how producers publish, brokers retain and route, and consumers process and acknowledge every message.

More Articles to Read

• Event-Driven Architecture in Distributed Systems
• Message Delivery Guarantees: At-Most-Once vs At-Least-Once vs Exactly-Once
• Background Workers Explained: Designing Reliable Asynchronous Processing
• Dead-Letter Queues, Retries, and Poison Messages
• Transactional Outbox Pattern for Reliable Messaging
• Message Queue Best Practices for Production Systems