What Is the Transactional Outbox Pattern?

The Transactional Outbox Pattern reliably coordinates a database change with publishing a message or event. Instead of updating the database and sending a message as two independent operations, the application writes both the business data and an outbox record in the same database transaction.

A separate publisher later reads the outbox and sends the message to a broker. This avoids a common distributed-systems failure where the database commits successfully but message publication fails, leaving other services unaware of the change.

Table of Contents

• The Dual-Write Problem
• How the Transactional Outbox Pattern Works
• The Outbox Table
• Publishing Outbox Events
  • Polling Publisher
  • Change Data Capture
• Delivery Semantics and Duplicate Messages
• Message Ordering
• Failure Scenarios and Recovery
• Production Design Example
• Common Transactional Outbox Mistakes
• Production Checklist
• Frequently Asked Questions
  • Why Not Publish Directly After the Database Commit?
  • Does Transactional Outbox Provide Exactly-Once Delivery?
  • Is Transactional Outbox the Same as the Saga Pattern?
  • Should Published Outbox Records Be Deleted?
  • When Should the Transactional Outbox Pattern Be Used?
• Conclusion
  • More Articles to Read

The Dual-Write Problem

Consider an order service that must create an order in PostgreSQL and publish an order.created event to a message broker.

A straightforward implementation performs two operations:

1. INSERT order into database
2. Publish order.created event

The problem is that these operations belong to different systems and therefore do not share a normal local database transaction.

Suppose the database succeeds but message publication fails:

Order Service
     |
     +---- INSERT order --------> Database
     |                           ✓ COMMIT
     |
     +---- Publish event -------> Message Broker
                                 ✗ FAILURE

The order exists, but downstream services never receive the event.

Reversing the operations does not solve the problem:

1. Publish order.created
2. INSERT order

Now the message can succeed while the database transaction fails:

Publish event     ✓
Database commit   ✗

Downstream services believe
an order exists when it does not.

This is the dual-write problem: an application needs two state changes in separate systems to behave consistently, but there is no atomic transaction spanning them.

Distributed transactions can sometimes coordinate multiple resources, but they introduce significant coupling and operational trade-offs. Designing Distributed Transactions with Sagas and Two-Phase Commit covers those alternatives in more detail.

How the Transactional Outbox Pattern Works

The Transactional Outbox Pattern changes the architecture so the critical synchronous operation touches only one transactional resource: the application database.

Instead of publishing directly to the broker, the application writes an event into an outbox table.

Application
     |
     | Database Transaction
     |
     +---- INSERT order
     |
     +---- INSERT outbox event
     |
     v
   COMMIT
     |
     v
Both changes become durable together

A separate publisher processes the outbox asynchronously:

Application
     |
     v
Database
+-------------------+
| Business Data     |
| Outbox            |
+-------------------+
          |
          v
   Outbox Publisher
          |
          v
    Message Broker
          |
          v
      Consumers

The database transaction now provides the atomicity:

Business change + Outbox event

          |
     +----+----+
     |         |
   COMMIT    ROLLBACK
     |         |
   both       neither

If order creation fails, the outbox record also disappears with the rollback. If the transaction commits, the event is durably available for later publication.

The message broker no longer has to be available during the business transaction.

The Outbox Table

An outbox table stores messages that need to be published.

A simple PostgreSQL schema might look like:

CREATE TABLE outbox_events (
    id UUID PRIMARY KEY,
    aggregate_type VARCHAR(100) NOT NULL,
    aggregate_id VARCHAR(100) NOT NULL,
    event_type VARCHAR(100) NOT NULL,
    payload JSONB NOT NULL,
    created_at TIMESTAMP NOT NULL DEFAULT CURRENT_TIMESTAMP,
    published_at TIMESTAMP NULL
);

An order transaction can then write both records:

import json
import uuid


def create_order(db, order):
    event_id = str(uuid.uuid4())

    with db.transaction():
        db.execute(
            """
            INSERT INTO orders (
                id,
                customer_id,
                status,
                total
            )
            VALUES (%s, %s, %s, %s)
            """,
            (
                order.id,
                order.customer_id,
                "CREATED",
                order.total,
            ),
        )

        db.execute(
            """
            INSERT INTO outbox_events (
                id,
                aggregate_type,
                aggregate_id,
                event_type,
                payload
            )
            VALUES (%s, %s, %s, %s, %s)
            """,
            (
                event_id,
                "order",
                order.id,
                "order.created",
                json.dumps({
                    "event_id": event_id,
                    "order_id": order.id,
                    "customer_id": order.customer_id,
                    "total": str(order.total),
                }),
            ),
        )

The important part is not the exact schema. The important property is that the business write and outbox write participate in the same database transaction.

The resulting state is always one of two possibilities:

The inconsistent combinations that existed with direct dual writes are removed from the synchronous transaction.

Publishing Outbox Events

Writing the outbox record is only the first half of the pattern. Another component must reliably move pending events from the database to the message broker.

Two common approaches are polling and Change Data Capture.

Polling Publisher

A polling publisher periodically queries unpublished records:

SELECT
    id,
    event_type,
    payload
FROM outbox_events
WHERE published_at IS NULL
ORDER BY created_at
LIMIT 100;

It publishes each event and marks successfully published records:

Outbox Table
     |
     | SELECT pending
     v
Publisher
     |
     | publish
     v
Message Broker
     |
     | success
     v
Mark event published

A simplified publisher might look like:

def publish_batch(db, broker):
    events = db.fetch_pending_outbox(limit=100)

    for event in events:
        broker.publish(
            event_type=event.event_type,
            payload=event.payload,
        )

        db.mark_outbox_published(event.id)

Polling is simple and works without specialized database infrastructure. The main trade-offs are polling latency, database load, batch sizing, and coordination between multiple publisher instances.

At higher throughput, publishers can claim rows in batches using database locking or another ownership mechanism so multiple workers do not unnecessarily process the same records concurrently.

Change Data Capture

Another approach is to capture committed outbox changes from the database transaction log.

Application
     |
     v
Database Transaction
     |
     +---- Business Row
     |
     +---- Outbox Row
     |
     v
Transaction Log
     |
     v
CDC Pipeline
     |
     v
Message Broker

The application still writes the outbox record transactionally, but a CDC system observes committed database changes and publishes them downstream.

This can avoid frequent polling and support high event volumes, but it introduces CDC infrastructure and additional operational complexity.

What Is Change Data Capture? explains the underlying CDC mechanism and its trade-offs.

Delivery Semantics and Duplicate Messages

The Transactional Outbox Pattern prevents lost messages caused by the database-to-broker dual write, but it does not automatically guarantee that each message reaches the broker exactly once.

Consider this failure:

Publisher
    |
    | publish event
    v
Broker
    |
    | ✓ stored
    |
    X publisher crashes
      before marking outbox row
      as published

After restarting, the publisher sees the same outbox record as pending and publishes it again.

Attempt 1 → Broker ✓
             |
             X crash

Restart

Attempt 2 → Broker ✓

Possible duplicate message

This is normally handled using at-least-once publication plus idempotent consumers.

The outbox event should therefore have a stable unique identifier:

{
  "event_id": "c749b4d2-67f0-4f80-a442-125e9d1db09e",
  "event_type": "order.created",
  "order_id": "ORD-9182"
}

A consumer can remember processed event IDs and ignore redelivery:

event_id
   |
   v
Already processed?
   |
+--+--+
|     |
no   yes
|     |
v     v
work  skip

This is why transactional outbox and message deduplication are commonly used together. What Is Message Deduplication? covers consumer-side duplicate protection in detail.

Message Ordering

Ordering becomes important when multiple events describe changes to the same entity.

For example:

Order ORD-100

1. order.created
2. order.paid
3. order.shipped

If independent publishers process outbox records concurrently, events can potentially reach the broker in a different order unless ordering is deliberately preserved.

Common approaches include:

• ordering outbox rows by a monotonic sequence;
• partitioning events by aggregate ID;
• processing one aggregate's events sequentially;
• using broker partition keys such as order_id;
• including aggregate versions so consumers can detect unexpected order.

For example:

{
  "event_id": "evt-812",
  "event_type": "order.paid",
  "order_id": "ORD-100",
  "version": 2
}

The correct strategy depends on whether strict ordering is actually required. Global ordering is expensive and usually unnecessary; ordering per aggregate, customer, account, or another business key is more common.

Failure Scenarios and Recovery

The value of the Transactional Outbox Pattern becomes clearer when examining failures at each stage.

The pattern converts an ambiguous cross-system write into durable local state that can be retried.

This changes:

Database succeeded.
Did message publication succeed?

into:

Database transaction committed.
Outbox event exists.
Keep trying until publication succeeds.

That is a much easier failure mode to operate.

Production Design Example

Consider an e-commerce system where an Order Service owns the order database and several downstream services react to order events.

Client
  |
  v
Order API
  |
  | single transaction
  v
PostgreSQL
+----------------------+
| orders               |
| outbox_events        |
+----------------------+
          |
          v
   Outbox Publisher
          |
          v
     Message Broker
      /     |      \
     v      v       v
Payment  Inventory  Notification
Service   Service      Service

A request creates an order:

POST /orders
      |
      v
BEGIN
      |
      +---- INSERT orders
      |
      +---- INSERT outbox_events
      |
      v
COMMIT
      |
      v
201 Created

The API does not need to wait for the message broker before completing the transaction.

The outbox contains:

{
  "id": "evt-9281",
  "aggregate_type": "order",
  "aggregate_id": "ORD-481",
  "event_type": "order.created",
  "payload": {
    "event_id": "evt-9281",
    "order_id": "ORD-481",
    "customer_id": "CUS-91",
    "total": "149.95"
  }
}

The publisher processes records in batches:

Pending Outbox Rows
        |
        v
   Fetch Batch
        |
        v
Publish to Broker
        |
   +----+----+
   |         |
success    failure
   |         |
   v         v
mark       leave pending
published  and retry

Suppose the broker is unavailable for five minutes.

Orders continue to be created because their database transactions do not depend on broker availability. Outbox rows accumulate:

Broker unavailable

12:00  10 pending
12:01  80 pending
12:02  170 pending
12:03  260 pending
12:04  350 pending

When the broker recovers, publishers drain the backlog.

This creates an important operational requirement: the publisher must have enough throughput to catch up faster than new outbox events arrive.

Useful metrics include:

Oldest unpublished event age is especially useful. A backlog of 10,000 events may be acceptable during a traffic spike if they are only seconds old, while 20 events that have been stuck for an hour indicate a serious problem.

Outbox records also need a lifecycle. Published rows can be deleted, archived, or partitioned and expired according to operational and audit requirements.

For a broader production treatment, see Transactional Outbox Pattern for Reliable Messaging.

Common Transactional Outbox Mistakes

• Writing the outbox record outside the business transaction. This recreates the original dual-write problem.
• Publishing directly to the broker inside the database transaction. The broker operation is still not part of the local database transaction and can make transactions unnecessarily long.
• Assuming publication is exactly once. A publisher can crash after broker acknowledgement but before recording publication.
• Using consumers that are not idempotent. Duplicate publication can create duplicate business effects.
• Generating a new event ID for every retry. Consumers can no longer recognize the same logical event.
• Ignoring event ordering. Concurrent publishers can reorder related events.
• Polling too aggressively. Frequent empty queries can create unnecessary database load.
• Polling too slowly. Events gain avoidable publication latency.
• Processing unbounded batches. Large batches can create long transactions, memory pressure, or broker bursts.
• Never cleaning published records. The outbox table grows indefinitely and can degrade database performance.
• Ignoring publisher backlog. The API may appear healthy while downstream systems are hours behind.
• Sharing one outbox without clear ownership. Event schemas, publishers, and cleanup responsibilities become difficult to manage.

Production Checklist

• Write business data and the outbox event in the same database transaction.
• Give every event a stable unique identifier.
• Include enough event metadata for routing and diagnostics.
• Define the event schema explicitly.
• Choose polling or CDC based on throughput and operational requirements.
• Use bounded publisher batches.
• Coordinate multiple publisher instances safely.
• Retry transient publication failures.
• Use backoff when the broker is unavailable.
• Design consumers for duplicate delivery.
• Use idempotency or message deduplication for non-idempotent side effects.
• Define ordering requirements explicitly.
• Partition related messages consistently when ordering matters.
• Monitor pending outbox count.
• Monitor oldest unpublished event age.
• Monitor publisher throughput and failures.
• Alert when publication lag exceeds acceptable limits.
• Plan for broker outages and backlog recovery.
• Ensure publisher capacity can drain accumulated backlog.
• Define retention and cleanup for published rows.
• Test crashes before publication.
• Test crashes after successful publication.
• Test database rollback behavior.
• Test broker outages.
• Test duplicate delivery.
• Test multiple publisher instances concurrently.

Frequently Asked Questions

The Transactional Outbox Pattern solves a specific consistency problem: reliably connecting a local database transaction to asynchronous messaging without requiring a distributed transaction across the database and broker.

Why Not Publish Directly After the Database Commit?

Because the process can fail between the two operations.

Database COMMIT ✓
       |
       X process crashes
       |
Publish message never happens

The outbox makes the intention to publish part of the same durable transaction as the business change.

Does Transactional Outbox Provide Exactly-Once Delivery?

Not automatically. The pattern prevents the event from being lost when the business transaction commits, but the publisher may deliver the same event more than once.

At-least-once publication combined with idempotent consumers is a common design. The broader delivery trade-offs are explained in Message Delivery Guarantees: At-Most-Once vs At-Least-Once vs Exactly-Once.

Is Transactional Outbox the Same as the Saga Pattern?

No. They solve different problems.

A transactional outbox reliably publishes messages that correspond to a local database transaction. A Saga coordinates a business workflow consisting of multiple distributed operations and typically uses compensating actions when later steps fail.

The two patterns are frequently used together. What Is a Saga Pattern? covers Saga coordination separately.

Should Published Outbox Records Be Deleted?

Eventually, usually yes, unless there is a specific audit or replay requirement.

Common approaches include deleting published rows after a retention period, moving them to archive storage, or partitioning the table by time and dropping old partitions efficiently.

Immediate deletion is not always necessary. Keeping records briefly can help diagnostics, but unlimited retention turns the outbox into an ever-growing operational table.

When Should the Transactional Outbox Pattern Be Used?

It is most useful when a service must commit local database state and reliably publish an event or command representing that change.

Typical examples include order creation, payment state changes, shipment updates, account changes, inventory updates, and other workflows where losing the corresponding message would leave downstream systems inconsistent.

Conclusion

The Transactional Outbox Pattern solves the dual-write problem by storing business changes and messages in the same local database transaction. Once committed, a separate publisher can reliably deliver the outbox events to a message broker without making the business transaction depend on broker availability.

The pattern does not eliminate every messaging problem. Publishers can create duplicates, related events may require ordering, outbox tables need cleanup, and publication backlogs need monitoring. Reliable implementations therefore combine the outbox with idempotent consumers, stable event IDs, explicit ordering rules, retry policies, and operational observability.

Key Takeaway: do not try to make a database write and message publication reliable by hoping both independent operations succeed. Commit the business change and the intention to publish atomically, then deliver the message asynchronously and retry safely.

More Articles to Read

• What Is a Transactional Outbox?
• Event-Driven Architecture in Distributed Systems
• Message Queue Best Practices for Production Systems