What Is a Transactional Outbox?

A Transactional Outbox is a reliability pattern that keeps a database change and the event or message describing that change consistent without requiring a distributed transaction between the database and a message broker.

Instead of updating the database and publishing directly to Kafka, RabbitMQ, Amazon SQS, or another messaging system, the application stores the outgoing message in an outbox table inside the same database transaction. A separate publisher later reads the outbox and delivers the message to the broker.

Table of Contents

• Why the Transactional Outbox Exists
• The Dual-Write Problem
• How the Transactional Outbox Works
  • Writing the Business Data and Outbox Record
  • Publishing Outbox Records
  • Marking Messages as Published
• Outbox Table Design
• Polling Publisher vs Change Data Capture
• Delivery Guarantees and Duplicate Messages
• Message Ordering
• Transactional Outbox and Idempotency
• Transactional Outbox and Sagas
• Failure Scenarios
• When to Use the Transactional Outbox
• Production Design Example
• Production Checklist
• Common Transactional Outbox Mistakes
• Frequently Asked Questions
  • Does the Transactional Outbox Guarantee Exactly-Once Delivery?
  • Can the Outbox Be a Separate Database?
  • Should Outbox Records Be Deleted?
  • Does Every Service Need an Outbox?
• Conclusion
  • More Articles to Read

Why the Transactional Outbox Exists

Event-driven applications frequently need to perform two operations together:

1. Change application data
2. Publish an event describing the change

For example, an Order service creates an order in PostgreSQL and publishes an OrderCreated event.

Order Service
     │
     ├──→ PostgreSQL
     │    INSERT order
     │
     └──→ Message Broker
          Publish OrderCreated

Both operations belong to one logical business action. If the order exists, downstream systems should eventually receive the event.

The problem is that the database and message broker are two independent systems. A normal database transaction cannot atomically commit both operations.

The Dual-Write Problem

Consider this application code:

order = create_order(data)
database.commit()

broker.publish({
    "type": "OrderCreated",
    "order_id": order.id,
})

The database commit succeeds first.

Then the application crashes:

INSERT order
     ↓
Database COMMIT ✓
     ↓
Application crashes
     ↓
Publish event ✗

The resulting state is inconsistent:

Database:
Order exists ✓

Message Broker:
OrderCreated missing ✗

A downstream Inventory service may never reserve inventory. A Notification service may never send confirmation. Analytics may never observe the order.

Changing the order of operations does not solve the problem.

broker.publish(event)
database.commit()

Now the message can succeed while the database commit fails:

Publish OrderCreated ✓
        ↓
Database COMMIT ✗

Consumers receive an event describing an order that does not exist.

This is the dual-write problem: one logical operation requires writes to two independent systems, but there is no atomic transaction covering both

How the Transactional Outbox Works

The Transactional Outbox removes the message broker from the application's critical database transaction.

Instead of trying to perform:

Database + Broker

atomically, the application performs:

Business Data + Outbox Record

inside one local database transaction.

The architecture becomes:

Application
    │
    ↓
Database Transaction
    │
    ├── Business Data
    │
    └── Outbox Record
             │
             ↓
      Outbox Publisher
             │
             ↓
       Message Broker
             │
             ↓
          Consumers

Writing the Business Data and Outbox Record

Suppose an Order service creates a new order.

Instead of publishing immediately, it performs both database writes in one transaction:

BEGIN;

INSERT INTO orders (
    id,
    customer_id,
    total,
    status
)
VALUES (
    'order-8472',
    'customer-91',
    149.00,
    'PENDING'
);

INSERT INTO outbox (
    id,
    aggregate_type,
    aggregate_id,
    event_type,
    payload,
    created_at
)
VALUES (
    'event-5521',
    'order',
    'order-8472',
    'OrderCreated',
    '{"order_id":"order-8472","customer_id":"customer-91","total":149.00}',
    CURRENT_TIMESTAMP
);

COMMIT;

The database guarantees that either both records commit or neither does.

Order INSERT       ✓
Outbox INSERT      ✓
--------------------
Transaction COMMIT ✓

If the transaction fails:

Order INSERT       ✗
Outbox INSERT      ✗
--------------------
Transaction ROLLBACK

There is no state where the order commits successfully but the corresponding outbox record disappears because of an application crash between two independent systems.

Publishing Outbox Records

A separate publisher reads unpublished records from the outbox.

SELECT *
FROM outbox
WHERE published_at IS NULL
ORDER BY created_at
LIMIT 100;

It publishes each event to the message broker:

for event in load_pending_outbox_events():
    broker.publish(
        topic=event.event_type,
        payload=event.payload,
    )

    mark_as_published(event.id)

The publisher can run as:

• a background worker;
• a separate service;
• a scheduled process;
• a database change-data-capture pipeline.

The critical property is that publication is now retryable.

If the broker is unavailable, the event remains in the database.

Outbox record exists
       ↓
Broker unavailable
       ↓
Publish fails
       ↓
Record remains pending
       ↓
Retry later

Marking Messages as Published

With a polling implementation, successfully delivered messages can be marked as published:

UPDATE outbox
SET published_at = CURRENT_TIMESTAMP
WHERE id = 'event-5521';

The row might now contain:

id           event-5521
event_type   OrderCreated
aggregate_id order-8472
created_at   12:00:01
published_at 12:00:02

Published records can later be archived or deleted according to a retention policy.

However, marking a message as published introduces another important failure window that affects delivery semantics.

Outbox Table Design

An outbox record usually contains enough information to publish an event without rereading mutable business data.

A practical schema might look like:

CREATE TABLE outbox (
    id UUID PRIMARY KEY,
    aggregate_type VARCHAR(100) NOT NULL,
    aggregate_id VARCHAR(100) NOT NULL,
    event_type VARCHAR(150) NOT NULL,
    payload JSONB NOT NULL,
    created_at TIMESTAMP NOT NULL,
    published_at TIMESTAMP NULL
);

Common fields include:

Additional production fields can include:

• event schema version;
• correlation ID;
• trace ID;
• partition or ordering key;
• retry metadata;
• destination topic;
• tenant ID when required.

The payload should normally represent the event as it existed when the business transaction committed.

Publishing only an entity ID and rebuilding the entire event from the current database state later can accidentally publish information from a newer version of the entity.

Polling Publisher vs Change Data Capture

There are two common approaches for moving outbox records to the message broker.

Polling Publisher

A worker periodically queries the outbox table.

Database
   ↑
Poll every N milliseconds
   │
Publisher
   │
   ↓
Broker

The approach is straightforward and keeps the implementation under application control.

For example:

def publish_batch():
    events = load_pending_events(limit=100)

    for event in events:
        try:
            publish(event)
            mark_published(event.id)
        except Exception:
            continue

The polling interval creates a latency trade-off.

Short interval
→ lower publication latency
→ more database queries

Long interval
→ fewer database queries
→ higher publication latency

High-volume implementations also need efficient indexing, batching, and safe coordination between multiple publishers.

Change Data Capture

Another option is to stream committed outbox changes from the database transaction log using Change Data Capture, commonly called CDC.

Application
     ↓
Database Transaction
     ↓
Outbox Table
     ↓
Database Transaction Log
     ↓
CDC Connector
     ↓
Message Broker

Instead of repeatedly querying for unpublished rows, the CDC infrastructure observes committed changes and forwards them downstream.

This can reduce polling overhead and support high-throughput pipelines, but introduces additional infrastructure and operational complexity.

Neither approach changes the fundamental pattern: the application atomically stores the business state and an outbox representation of the event.

Delivery Guarantees and Duplicate Messages

The Transactional Outbox solves lost messages caused by the database/broker dual-write problem, but it does not automatically provide exactly-once delivery.

Consider the publisher:

1. Read outbox record
2. Publish message
3. Mark record as published

The process can crash between steps 2 and 3:

Publish to broker ✓
        ↓
Publisher crashes
        ↓
Mark published ✗

After restarting, the outbox record still appears unpublished.

The publisher retries:

First publication  → OrderCreated
Retry publication  → OrderCreated

The consumer can therefore receive the same logical event more than once.

In practice, the Transactional Outbox commonly produces at-least-once delivery behavior.

This is why delivery guarantees should be considered together with the broader concepts in Message Delivery Guarantees: At-Most-Once vs At-Least-Once vs Exactly-Once.

Message Ordering

Outbox rows are committed in database transactions, but that does not automatically guarantee that every consumer will observe events in the desired business order.

Suppose an order produces:

OrderCreated
     ↓
OrderPaid
     ↓
OrderShipped

The application may need these events to remain ordered for the same order.

A useful approach is to associate events with the aggregate:

aggregate_id = order-8472

and use that identifier as the partition or ordering key when publishing.

order-8472:

OrderCreated
     ↓
OrderPaid
     ↓
OrderShipped

Global ordering across every event in a large system is usually unnecessary and expensive. Ordering requirements should normally be defined around the business entity or partition that actually requires them.

Multiple publisher workers also need careful coordination. If two workers process rows for the same aggregate simultaneously, they can publish events out of order even when the rows were originally created correctly.

Transactional Outbox and Idempotency

Because duplicate delivery is possible, consumers should normally process events idempotently.

Every outbox event should have a stable unique identifier:

{
  "event_id": "event-5521",
  "event_type": "OrderCreated",
  "order_id": "order-8472"
}

A consumer can record processed event IDs:

BEGIN;

INSERT INTO processed_events (event_id)
VALUES ('event-5521')
ON CONFLICT DO NOTHING;

-- Apply business change only when
-- this event was not previously processed.

COMMIT;

Another strategy is to design the business operation itself to be naturally idempotent.

For example:

Set order status = CONFIRMED

is often easier to retry safely than:

Increment confirmed_order_count by 1

Idempotency is therefore a complementary pattern rather than something the outbox eliminates. The relationship is covered in more detail in Idempotency and Deduplication in Distributed Systems.

Transactional Outbox and Sagas

The Transactional Outbox is frequently used inside distributed workflows such as sagas.

Suppose an Inventory service handles a ReserveInventory command.

It needs to:

1. Reserve inventory
2. Publish InventoryReserved

Without an outbox:

Reserve inventory ✓
       ↓
Process crashes
       ↓
InventoryReserved ✗

The saga can become permanently stuck because the next participant never receives the event.

With an outbox:

BEGIN;

UPDATE inventory
SET available = available - 1,
    reserved = reserved + 1
WHERE product_id = 'product-52';

INSERT INTO outbox (
    id,
    aggregate_id,
    event_type,
    payload,
    created_at
)
VALUES (
    'event-9921',
    'order-8472',
    'InventoryReserved',
    '{"order_id":"order-8472","product_id":"product-52"}',
    CURRENT_TIMESTAMP
);

COMMIT;

The inventory reservation and the intent to publish its event now succeed or fail together.

The patterns solve different problems:

Saga
  ↓
Coordinates a multi-step
business transaction

Transactional Outbox
  ↓
Reliably connects a committed
local transaction to messaging

They are often combined because reliable event publication is necessary for reliable event-driven workflows.

Failure Scenarios

A useful way to understand the Transactional Outbox is to examine what happens when each component fails.

The important property is that the dangerous failure:

Business transaction committed
+
Event permanently forgotten

is removed from the normal application flow.

The message becomes durable before the database transaction is considered complete.

When to Use the Transactional Outbox

The pattern is useful when a committed database change must reliably cause a message or event to be published.

Typical examples include:

• publishing domain events from microservices;
• triggering asynchronous workflows after database changes;
• starting saga steps;
• sending notifications after committed operations;
• updating search indexes asynchronously;
• propagating changes to other services;
• feeding analytics or audit pipelines.

The pattern is particularly valuable when losing an event would leave other systems permanently inconsistent.

It may be unnecessary when no reliable relationship exists between the database transaction and messaging.

For example, best-effort telemetry may not justify an application outbox:

Request completed
     ↓
Best-effort debug metric

The reliability requirement should determine whether the additional storage, publishing, cleanup, and monitoring complexity is worthwhile.

Production Design Example

Consider an e-commerce Order service using PostgreSQL and a message broker.

Creating an order must eventually trigger:

• inventory reservation;
• payment processing;
• customer notifications;
• analytics updates.

The request arrives:

POST /orders
     ↓
Order Service
     ↓
PostgreSQL Transaction

The transaction writes both the order and its event:

BEGIN;

INSERT INTO orders (
    id,
    customer_id,
    status,
    total
)
VALUES (
    'order-8472',
    'customer-91',
    'PENDING',
    149.00
);

INSERT INTO outbox (
    id,
    aggregate_type,
    aggregate_id,
    event_type,
    payload,
    created_at
)
VALUES (
    'evt-8472-1',
    'order',
    'order-8472',
    'OrderCreated',
    '{
        "event_id": "evt-8472-1",
        "order_id": "order-8472",
        "customer_id": "customer-91",
        "total": 149.00
    }',
    CURRENT_TIMESTAMP
);

COMMIT;

The API can now return success without waiting for every downstream service.

A pool of outbox workers processes pending messages in batches.

SELECT *
FROM outbox
WHERE published_at IS NULL
ORDER BY created_at
LIMIT 100
FOR UPDATE SKIP LOCKED;

SKIP LOCKED allows multiple workers to process different rows without waiting for each other.

A worker publishes:

OrderCreated
event_id = evt-8472-1
partition_key = order-8472

and records successful publication.

Downstream consumers receive the event:

                    OrderCreated
                         │
            ┌────────────┼────────────┐
            ↓            ↓            ↓
       Inventory      Payment    Notification
        Service       Service       Service

Each consumer handles duplicate events safely using the stable event_id.

Suppose the message broker becomes unavailable for five minutes.

Order creation can still succeed as long as PostgreSQL is healthy:

New orders
    ↓
PostgreSQL
    ↓
Outbox backlog grows
    ↓
Broker recovers
    ↓
Workers drain backlog

The outbox therefore acts as a durable boundary between synchronous business transactions and asynchronous messaging.

However, backlog growth must be observable.

Useful production metrics include:

• number of unpublished outbox records;
• age of the oldest unpublished event;
• publication throughput;
• publication latency;
• broker failure rate;
• retry count;
• duplicate publication count when measurable;
• outbox table size;
• cleanup or archival failures.

The most useful alert is often not simply:

Publisher failed once

but:

Oldest unpublished event
has been pending for 5 minutes

That directly measures whether the system is failing to deliver committed business events within the expected time.

Production Checklist

• Write business data and the outbox record in the same local database transaction.
• Give every event a stable unique ID.
• Make publishers safe to retry.
• Design consumers to tolerate duplicate delivery.
• Define ordering requirements explicitly.
• Use an aggregate or partition key when per-entity ordering matters.
• Store enough event data to publish the committed version reliably.
• Version event schemas when payloads evolve.
• Process outbox rows in batches rather than one query per event at high volume.
• Define retention, archival, or deletion rules.
• Monitor backlog size and oldest-event age.
• Alert on events that remain unpublished beyond the expected delivery window.

Common Transactional Outbox Mistakes

• Writing the outbox record in a separate transaction. This recreates the original dual-write problem.
• Assuming the pattern guarantees exactly-once delivery. A publisher can send the same message again after a crash.
• Using non-idempotent consumers. Duplicate delivery can then create duplicate business effects.
• Deleting the outbox row before publication is confirmed. A failed publication can become a permanently lost event.
• Ignoring event ordering. Parallel publishers can reorder related events unless the design preserves the required sequence.
• Polling without appropriate indexes. A large outbox table can create unnecessary database load.
• Never cleaning old rows. The outbox can grow indefinitely and degrade database performance.
• Publishing mutable current state instead of the committed event payload. A later update can change what the original event should have represented.
• Missing schema versions. Long-lived consumers may fail when event structures evolve.
• Monitoring worker health but not backlog age. A running worker can still be too slow to keep up with production traffic.

Frequently Asked Questions

The Transactional Outbox is conceptually simple, but several implementation details determine whether it actually provides the intended reliability.

Does the Transactional Outbox Guarantee Exactly-Once Delivery?

No. It prevents a committed database change from depending on an unreliable immediate broker publication, but the publisher can still deliver an event more than once.

Consumers should therefore be designed for duplicate delivery, usually through idempotent processing or explicit deduplication.

Can the Outbox Be a Separate Database?

Not if the separate database cannot participate atomically in the same transaction as the business change.

The fundamental guarantee comes from committing the business state and outbox record together:

One transactional boundary:

Business Data
     +
Outbox Record

Putting the outbox in an unrelated database turns the operation back into two independent writes.

Should Outbox Records Be Deleted?

Usually, yes, eventually. Published rows should have a defined retention strategy so the table does not grow forever.

Depending on auditing and operational requirements, records can be deleted after a retention period, archived elsewhere, or partitioned so old data can be removed efficiently.

Does Every Service Need an Outbox?

No. The pattern is useful when a service must reliably publish information as a consequence of a committed local transaction.

A service that does not publish transactional events, or where occasional message loss is acceptable, may not need the additional complexity.

Conclusion

The Transactional Outbox solves one of the most common reliability problems in event-driven systems: atomically coordinating a database change with the intent to publish a message.

The application commits its business data and an outbox record in the same local transaction. A separate publisher then delivers the durable event to the message broker and retries when publication fails.

The pattern removes the dangerous database/broker dual write, but it does not eliminate every messaging problem. Duplicate delivery, consumer idempotency, ordering, backlog management, event schema evolution, and outbox cleanup still require explicit design.

The core principle is: do not try to make the database and message broker commit together; atomically persist the business change and the intent to publish, then deliver that intent asynchronously.

More Articles to Read

• Transactional Outbox Pattern for Reliable Messaging
• Event-Driven Architecture in Distributed Systems
• Message Queue Best Practices for Production Systems