Transactional Outbox Pattern for Reliable Messaging

Distributed applications often need to update a database and publish a message as part of one business operation. An order service may save a confirmed order and publish an order.confirmed event. A payment service may record a successful charge and notify fulfilment. If one action succeeds while the other fails, services can observe inconsistent state.

The transactional outbox pattern solves this dual-write problem by storing the business change and an outgoing message in the same database transaction. A separate publisher later reads the outbox record and delivers it to a message broker. This design does not eliminate duplicate delivery, but it prevents committed business changes from silently losing their corresponding messages.

Table of Contents

• The Dual-Write Problem
• What Is the Transactional Outbox Pattern?
  • Core Components
  • Message Flow
• Why a Database Transaction Is Not Enough
• Outbox Table Design
  • Minimum Outbox Fields
  • Message Payload vs Entity Reference
  • Outbox Indexes
• Writing Business Data and Outbox Records
  • Order Confirmation Example
  • Database-Generated Outbox Records
• Publishing Outbox Messages
  • Polling Publisher
  • Change Data Capture
  • Polling vs Change Data Capture
• Safe Message Claiming
• Delivery Guarantees
• Handling Duplicate Publication
• Consumer Idempotency
• Message Ordering
• Outbox Retries and Failures
• Poison Outbox Records
• Outbox Retention and Cleanup
• Outbox Partitioning
• Monitoring and Alerting
• Schema Evolution
• Security and Sensitive Data
• Deployment Architectures
  • Publisher Inside the Application
  • Dedicated Publisher Service
  • Serverless Outbox Publisher
• CloudFormation Example
• Real-World Payment Workflow
• Transactional Outbox vs Alternatives
• Testing the Outbox Pattern
• Common Mistakes
• Production Checklist
• Conclusion
• Key Takeaway
• More Articles to Read

The Dual-Write Problem

A dual write occurs when one business operation must update two independent systems. A common example is updating a relational database and publishing a message to a broker.

Consider an order service that performs these steps:

1. Update the order status to confirmed.
2. Publish an order.confirmed event.

If the database update happens first, the process can crash before publishing the event.

def confirm_order(order_id: str) -> None:
    orders.mark_confirmed(order_id)

    # The process may crash here.
    message_broker.publish(
        topic="order-events",
        message={
            "type": "order.confirmed",
            "order_id": order_id
        }
    )

The order is committed, but fulfilment, billing, analytics, and notification services never receive the event.

Reversing the operations creates a different failure.

def confirm_order(order_id: str) -> None:
    message_broker.publish(
        topic="order-events",
        message={
            "type": "order.confirmed",
            "order_id": order_id
        }
    )

    # The database update may fail after publication.
    orders.mark_confirmed(order_id)

Downstream services receive an event for an order that remains unconfirmed in the source database.

A local database transaction cannot automatically include an external broker. The database and broker commit independently, leaving a failure window between them.

What Is the Transactional Outbox Pattern?

The transactional outbox pattern replaces the direct broker publication with a database insert. The application stores the business update and the outgoing message in the same local database transaction.

Because both records use the same transaction, either both commit or neither commits.

Core Components

A typical transactional outbox implementation contains four components:

• Business service: updates domain data.
• Outbox table: stores messages waiting for publication.
• Outbox publisher: reads unpublished records and sends them to the broker.
• Message broker: delivers messages to downstream consumers.

The business service does not need to wait for the broker during the original database transaction. Broker downtime therefore does not prevent a valid business update from being committed.

Message Flow

The complete flow is:

1. The application begins a database transaction.
2. Business data is inserted or updated.
3. An outbox record is inserted in the same transaction.
4. The transaction commits.
5. A publisher reads the outbox record.
6. The publisher sends the message to the broker.
7. The broker confirms publication.
8. The publisher marks the outbox record as published or removes it.

The source-of-truth database now contains durable evidence that a message must eventually be published.

Why a Database Transaction Is Not Enough

A database transaction guarantees atomicity only for operations managed by that database. Publishing to a message broker, calling an external API, or writing to another database remains outside the local transaction.

Distributed transactions can coordinate multiple systems through protocols such as two-phase commit, but many modern brokers, cloud services, and databases do not participate in one shared transaction. Even when supported, distributed transactions add operational complexity, lock duration, coordination overhead, and failure modes.

The transactional outbox avoids a distributed commit. It first makes the publication request durable in the local database and then completes broker delivery asynchronously.

Outbox Table Design

The outbox table is the durable bridge between the database transaction and the message broker. Its structure should support efficient publishing, retries, monitoring, ordering, and cleanup.

Minimum Outbox Fields

A practical relational outbox table may contain:

CREATE TABLE outbox_messages (
    outbox_id UUID PRIMARY KEY,
    aggregate_type VARCHAR(100) NOT NULL,
    aggregate_id VARCHAR(100) NOT NULL,
    event_type VARCHAR(150) NOT NULL,
    event_version INTEGER NOT NULL,
    payload_json JSONB NOT NULL,
    headers_json JSONB NOT NULL DEFAULT '{}'::JSONB,
    status VARCHAR(30) NOT NULL DEFAULT 'pending',
    attempt_count INTEGER NOT NULL DEFAULT 0,
    available_at TIMESTAMP NOT NULL DEFAULT NOW(),
    created_at TIMESTAMP NOT NULL DEFAULT NOW(),
    published_at TIMESTAMP,
    last_error TEXT
);

Important fields include:

• outbox_id: unique message identifier.
• aggregate_type: business entity category such as order or payment.
• aggregate_id: identifier used for tracing and ordering.
• event_type: logical message type.
• event_version: schema version.
• payload_json: immutable event data.
• headers_json: correlation, causation, tenant, and tracing metadata.
• status: publication state.
• attempt_count: number of publication attempts.
• available_at: next eligible retry time.
• published_at: successful publication time.

Message Payload vs Entity Reference

An outbox record can store either the complete event payload or only a reference to business data.

Complete payload

The outbox stores the exact message that will be published.

Advantages

• The message is an immutable snapshot.
• Later business-data changes do not modify event meaning.
• The publisher does not need additional queries.
• Replay produces the original event.

Disadvantages

• More database storage is required.
• Sensitive fields may be duplicated.
• Large payloads can increase table and index pressure.

When to Use

Use complete payloads for domain events, integration events, audit-sensitive workflows, and cases where the event must represent state at transaction time.

Real-World Use Cases

• Order-confirmed events.
• Payment-authorised events.
• Shipment-created notifications.

Example

event_payload = {
    "order_id": order.order_id,
    "customer_id": order.customer_id,
    "confirmed_at": order.confirmed_at.isoformat(),
    "total_amount": str(order.total_amount),
    "currency": order.currency
}

Entity reference

The outbox stores an identifier, and the publisher loads current business data before publication.

Advantages

• Smaller outbox records.
• Less duplicated data.
• Payload creation can be deferred.

Disadvantages

• The referenced record may change or disappear.
• The published event may not represent the original transaction.
• Additional database queries increase load.
• Replay may produce different content.

When to Use

Use references for command-like jobs where current state is intentionally loaded at processing time, not for immutable historical events.

Real-World Use Cases

• Generate the latest account report.
• Rebuild a search document from current state.
• Recalculate a derived projection.

Example

job_payload = {
    "job_type": "rebuild_customer_search_document",
    "customer_id": customer.customer_id
}

Outbox Indexes

The publisher must find eligible messages without scanning the whole table.

CREATE INDEX idx_outbox_pending_available
ON outbox_messages (available_at, created_at)
WHERE status = 'pending';

A partial index keeps published rows out of the active lookup structure.

If ordering is required per aggregate, an additional index can help:

CREATE INDEX idx_outbox_aggregate_order
ON outbox_messages (
    aggregate_type,
    aggregate_id,
    created_at
);

Indexes should remain narrow. Adding the entire JSON payload to active indexes increases write amplification and storage use.

Writing Business Data and Outbox Records

The critical rule is that the business update and outbox insert must share the same database transaction and connection.

Order Confirmation Example

The following service confirms an order and stores an integration event atomically.

import json
import uuid
from datetime import datetime, timezone


def confirm_order(database, order_id: str) -> None:
    now = datetime.now(timezone.utc)

    with database.transaction() as transaction:
        order = transaction.fetch_one(
            """
            SELECT
                order_id,
                customer_id,
                status,
                total_amount,
                currency
            FROM orders
            WHERE order_id = %s
            FOR UPDATE
            """,
            [order_id]
        )

        if order is None:
            raise ValueError("Order does not exist")

        if order["status"] == "confirmed":
            return

        transaction.execute(
            """
            UPDATE orders
            SET
                status = 'confirmed',
                confirmed_at = %s
            WHERE order_id = %s
            """,
            [now, order_id]
        )

        event_id = str(uuid.uuid4())

        payload = {
            "event_id": event_id,
            "event_type": "order.confirmed",
            "event_version": 1,
            "occurred_at": now.isoformat(),
            "order_id": order["order_id"],
            "customer_id": order["customer_id"],
            "total_amount": str(order["total_amount"]),
            "currency": order["currency"]
        }

        transaction.execute(
            """
            INSERT INTO outbox_messages (
                outbox_id,
                aggregate_type,
                aggregate_id,
                event_type,
                event_version,
                payload_json,
                headers_json,
                status,
                available_at,
                created_at
            )
            VALUES (
                %s,
                'order',
                %s,
                'order.confirmed',
                1,
                %s,
                %s,
                'pending',
                %s,
                %s
            )
            """,
            [
                event_id,
                order_id,
                json.dumps(payload),
                json.dumps({
                    "correlation_id": event_id
                }),
                now,
                now
            ]
        )

If the outbox insert fails, the order update rolls back. If the order update fails, no event is stored. The transaction preserves consistency between domain state and publication intent.

Database-Generated Outbox Records

Triggers can generate outbox rows automatically when business tables change.

Advantages

• Applications cannot accidentally forget the outbox insert.
• Legacy applications can gain event publication without major code changes.
• Database-level enforcement can cover multiple writers.

Disadvantages

• Business intent becomes hidden in database logic.
• Complex event payloads are harder to construct.
• Testing and versioning can become less visible.
• Low-level row changes do not always map cleanly to domain events.

When to Use

Use triggers when several applications update the same tables, when existing code cannot be modified safely, or when database-level capture is an intentional architectural decision.

Real-World Use Cases

• Publishing changes from a legacy monolith.
• Capturing updates from multiple administrative tools.
• Creating low-level change events for data integration.

Example

CREATE OR REPLACE FUNCTION create_order_outbox_event()
RETURNS TRIGGER AS $$
BEGIN
    IF OLD.status IS DISTINCT FROM NEW.status
       AND NEW.status = 'confirmed' THEN

        INSERT INTO outbox_messages (
            outbox_id,
            aggregate_type,
            aggregate_id,
            event_type,
            event_version,
            payload_json,
            status,
            available_at,
            created_at
        )
        VALUES (
            gen_random_uuid(),
            'order',
            NEW.order_id,
            'order.confirmed',
            1,
            jsonb_build_object(
                'order_id', NEW.order_id,
                'customer_id', NEW.customer_id,
                'confirmed_at', NEW.confirmed_at
            ),
            'pending',
            NOW(),
            NOW()
        );
    END IF;

    RETURN NEW;
END;
$$ LANGUAGE plpgsql;

Publishing Outbox Messages

After the transaction commits, a publisher must transfer pending outbox records to the broker. Two common approaches are polling and change data capture.

Polling Publisher

A polling publisher queries the outbox table at regular intervals.

Advantages

• Simple architecture.
• Works with most relational databases.
• Easy to implement inside an existing service.
• Retry logic remains under application control.

Disadvantages

• Polling adds database queries.
• Publication latency depends on polling frequency.
• Multiple publisher instances require safe row claiming.

When to Use

Use polling for low-to-moderate throughput, straightforward operations, and systems where adding change-data-capture infrastructure is unnecessary.

Real-World Use Cases

• Order and payment events from a SaaS application.
• Background publication from a monolith.
• Internal systems processing thousands of events per minute.

Example

def publish_outbox_batch(database, broker) -> int:
    messages = claim_outbox_messages(
        database=database,
        batch_size=100
    )

    published_count = 0

    for message in messages:
        try:
            broker.publish(
                topic=topic_for_event(message["event_type"]),
                key=message["aggregate_id"],
                message=message["payload_json"],
                headers=message["headers_json"]
            )

            mark_outbox_published(
                database=database,
                outbox_id=message["outbox_id"]
            )

            published_count += 1

        except Exception as error:
            schedule_outbox_retry(
                database=database,
                outbox_id=message["outbox_id"],
                error=error
            )

    return published_count

Change Data Capture

Change data capture reads committed database-log changes and forwards matching outbox inserts to a broker or integration pipeline.

Advantages

• Low publication latency.
• No continuous polling queries.
• Scales well for high write volumes.
• Can preserve database commit order.

Disadvantages

• Requires additional infrastructure.
• Database-log retention and connector health become operational concerns.
• Schema changes can affect connectors.
• Failure recovery requires understanding offsets and log positions.

When to Use

Use change data capture for high-throughput systems, low-latency integration, or environments already operating a database-change pipeline.

Real-World Use Cases

• Large commerce platforms publishing millions of events.
• Database-to-stream replication.
• Multi-service event platforms using database logs as the source.

Example

def handle_database_change(change: dict) -> None:
    if change["table"] != "outbox_messages":
        return

    if change["operation"] != "insert":
        return

    row = change["after"]

    message_broker.publish(
        topic=topic_for_event(row["event_type"]),
        key=row["aggregate_id"],
        message=row["payload_json"],
        headers=row["headers_json"]
    )

Polling vs Change Data Capture

The better option depends on scale, latency requirements, database capabilities, and operational maturity.

Safe Message Claiming

Multiple publisher instances may read the same pending records concurrently. Row claiming prevents every instance from publishing the same batch at the same time.

PostgreSQL can claim rows using FOR UPDATE SKIP LOCKED.

WITH claimed_messages AS (
    SELECT outbox_id
    FROM outbox_messages
    WHERE status = 'pending'
      AND available_at <= NOW()
    ORDER BY created_at
    FOR UPDATE SKIP LOCKED
    LIMIT 100
)
UPDATE outbox_messages AS outbox
SET status = 'processing'
FROM claimed_messages
WHERE outbox.outbox_id = claimed_messages.outbox_id
RETURNING outbox.*;

Each publisher receives a different unlocked batch. The transaction used for claiming should be short so locks do not remain open while network publication occurs.

A processing lease can recover records after a publisher crashes.

ALTER TABLE outbox_messages
ADD COLUMN locked_at TIMESTAMP,
ADD COLUMN locked_by VARCHAR(100);

UPDATE outbox_messages
SET
    status = 'pending',
    locked_at = NULL,
    locked_by = NULL
WHERE status = 'processing'
  AND locked_at < NOW() - INTERVAL '5 minutes';

The lease duration must exceed normal publication time while remaining short enough to recover abandoned records.

Delivery Guarantees

The transactional outbox normally provides at-least-once publication.

Consider this sequence:

1. The publisher sends the message.
2. The broker accepts it.
3. The publisher crashes before marking the outbox row as published.
4. The row remains pending or processing.
5. The publisher sends it again after recovery.

The business event is not lost, but the broker may receive it more than once.

The pattern therefore provides:

• Atomic business update and publication intent.
• Durable recovery after broker or process failure.
• At-least-once delivery when retry is enabled.
• No automatic exactly-once business processing.

Exactly-once claims should be treated carefully. A broker may deduplicate publication, but downstream databases and external APIs still need idempotent business operations.

Handling Duplicate Publication

Every outbox message needs a stable unique identifier. The same identifier must remain unchanged during publication retries.

broker.publish(
    topic="order-events",
    key=message["aggregate_id"],
    message=message["payload_json"],
    headers={
        "message_id": str(message["outbox_id"]),
        "event_type": message["event_type"],
        "event_version": str(message["event_version"])
    }
)

A broker with deduplication support may suppress repeated messages within a defined window. However, broker-side deduplication should not replace consumer idempotency because:

• Deduplication windows may expire.
• Messages may be replayed intentionally.
• Different topics or routes may bypass deduplication.
• Consumer processing can still fail after side effects occur.

Consumer Idempotency

Consumers should treat duplicate delivery as normal rather than exceptional.

A common technique stores processed message IDs.

CREATE TABLE processed_messages (
    consumer_name VARCHAR(150) NOT NULL,
    message_id UUID NOT NULL,
    processed_at TIMESTAMP NOT NULL DEFAULT NOW(),
    PRIMARY KEY (consumer_name, message_id)
);

def handle_order_confirmed(database, message: dict) -> None:
    message_id = message["event_id"]

    with database.transaction() as transaction:
        inserted = transaction.execute(
            """
            INSERT INTO processed_messages (
                consumer_name,
                message_id
            )
            VALUES (%s, %s)
            ON CONFLICT DO NOTHING
            """,
            [
                "fulfilment-order-consumer",
                message_id
            ]
        )

        if inserted.row_count == 0:
            return

        transaction.execute(
            """
            INSERT INTO fulfilment_orders (
                order_id,
                customer_id,
                status
            )
            VALUES (%s, %s, 'pending')
            ON CONFLICT (order_id) DO NOTHING
            """,
            [
                message["order_id"],
                message["customer_id"]
            ]
        )

The deduplication record and business update should share one transaction. Otherwise, the consumer may mark a message processed without applying its business effect.

Message Ordering

Outbox records may need to preserve order for events related to the same aggregate.

For example:

1. order.created
2. order.confirmed
3. order.cancelled

If events are published out of order, a consumer may observe cancellation before creation.

Useful ordering strategies include:

• Use the aggregate ID as the broker partition key.
• Store an aggregate sequence number.
• Claim rows in sequence order.
• Process one active event per aggregate.
• Make consumers reject or defer stale sequence numbers.

ALTER TABLE outbox_messages
ADD COLUMN aggregate_version BIGINT;

broker.publish(
    topic="order-events",
    key=message["aggregate_id"],
    message={
        **message["payload_json"],
        "aggregate_version": message["aggregate_version"]
    }
)

Global ordering across every aggregate is rarely necessary and severely limits throughput. Ordering should usually be scoped to one business entity or partition.

Outbox Retries and Failures

Broker publication can fail because of network errors, authentication problems, rate limits, broker outages, invalid messages, or oversized payloads.

Transient failures should use delayed retries with exponential backoff and jitter.

import random
from datetime import datetime, timedelta, timezone


def schedule_outbox_retry(
    database,
    outbox_id: str,
    error: Exception
) -> None:
    message = database.fetch_one(
        """
        SELECT attempt_count
        FROM outbox_messages
        WHERE outbox_id = %s
        """,
        [outbox_id]
    )

    attempt = message["attempt_count"] + 1
    base_delay = min(5 * (2 ** attempt), 900)
    jitter = random.randint(0, max(1, base_delay // 4))
    available_at = (
        datetime.now(timezone.utc)
        + timedelta(seconds=base_delay + jitter)
    )

    database.execute(
        """
        UPDATE outbox_messages
        SET
            status = 'pending',
            attempt_count = %s,
            available_at = %s,
            last_error = %s,
            locked_at = NULL,
            locked_by = NULL
        WHERE outbox_id = %s
        """,
        [
            attempt,
            available_at,
            str(error)[:2000],
            outbox_id
        ]
    )

Retrying immediately without backoff can overload a failing broker and consume database capacity.

Poison Outbox Records

A poison outbox record repeatedly fails because the message itself cannot be published successfully.

Examples include:

• Payload exceeds the broker size limit.
• Required routing information is missing.
• Payload serialization always fails.
• The event type has no valid destination.
• Headers contain unsupported values.

After a bounded number of attempts, the record should move to a failed state.

UPDATE outbox_messages
SET
    status = 'failed',
    last_error = :last_error
WHERE outbox_id = :outbox_id
  AND attempt_count >= 10;

A failed outbox record should remain searchable and alertable. It can be corrected, republished manually, or closed after investigation.

Repeatedly failing records should not block later messages unless strict aggregate ordering requires it. In ordered workflows, operations must decide whether to pause that aggregate, repair the message, or skip it explicitly.

Outbox Retention and Cleanup

Published records accumulate quickly. A retention policy prevents the active table from growing without limits.

Common cleanup strategies include:

• Delete published records after several days.
• Archive them to a history table.
• Partition the table by creation date.
• Keep failed records longer than successful records.

DELETE FROM outbox_messages
WHERE status = 'published'
  AND published_at < NOW() - INTERVAL '7 days';

Large deletes can create locks, transaction-log growth, and table bloat. Cleanup should run in bounded batches.

DELETE FROM outbox_messages
WHERE outbox_id IN (
    SELECT outbox_id
    FROM outbox_messages
    WHERE status = 'published'
      AND published_at < NOW() - INTERVAL '7 days'
    ORDER BY published_at
    LIMIT 5000
);

Retention must leave enough history for incident investigation and replay requirements.

Outbox Partitioning

High-volume systems may partition the outbox table by time, tenant, hash, or business domain.

Time-based partitioning simplifies retention because an old partition can be detached or dropped without deleting rows individually.

CREATE TABLE outbox_messages (
    outbox_id UUID NOT NULL,
    aggregate_id VARCHAR(100) NOT NULL,
    event_type VARCHAR(150) NOT NULL,
    payload_json JSONB NOT NULL,
    status VARCHAR(30) NOT NULL,
    created_at TIMESTAMP NOT NULL,
    published_at TIMESTAMP,
    PRIMARY KEY (outbox_id, created_at)
) PARTITION BY RANGE (created_at);

CREATE TABLE outbox_messages_2026_07
PARTITION OF outbox_messages
FOR VALUES FROM ('2026-07-01') TO ('2026-08-01');

Partitioning adds maintenance complexity and should solve a measured scaling problem rather than being introduced automatically.

Monitoring and Alerting

An outbox can appear healthy while messages quietly remain unpublished. Monitoring should focus on backlog age and publication progress, not only process availability.

Useful structured log fields include:

• Outbox ID.
• Event type.
• Aggregate ID.
• Attempt count.
• Publisher instance.
• Broker destination.
• Correlation ID.
• Publication duration.
• Error classification.

The most important alert is often the age of the oldest pending record. A stable backlog count can still hide one permanently stuck business event.

Schema Evolution

Outbox records may remain unpublished during outages or be replayed months later. Consumers therefore need explicit schema versions.

event = {
    "event_id": event_id,
    "event_type": "order.confirmed",
    "event_version": 2,
    "occurred_at": occurred_at,
    "data": {
        "order_id": order_id,
        "customer_id": customer_id,
        "total": {
            "amount": str(total_amount),
            "currency": currency
        }
    }
}

Safe evolution strategies include:

• Add optional fields instead of renaming required fields immediately.
• Keep old consumers compatible during migration.
• Version events when meaning changes.
• Support multiple versions during rolling deployments.
• Test old outbox records against new publisher code.

The publisher should usually treat the stored payload as opaque. Reconstructing it with the newest application code can accidentally change old event meaning.

Security and Sensitive Data

Outbox payloads may duplicate customer, payment, healthcare, or operational data. Database access, backups, logs, and replay tools must follow the same security controls as the source business tables.

Useful practices include:

• Store only fields required by consumers.
• Avoid credentials and access tokens.
• Encrypt sensitive values where required.
• Restrict publisher and operator access.
• Redact payloads from application logs.
• Apply retention limits.
• Track manual replay activity.

Large binary documents should normally remain in object storage. The outbox message can contain a stable object identifier rather than embedding the file.

Deployment Architectures

The publisher can run inside the business application, as a dedicated service, or as a serverless scheduled process. Each option has different scaling and operational characteristics.

Publisher Inside the Application

The same deployment runs API handlers and a background publisher process.

Advantages

• Simple repository and deployment model.
• Shared database models and configuration.
• Suitable for small systems.

Disadvantages

• API and publisher scaling become coupled.
• Application deployments interrupt publication.
• Resource contention may affect request processing.

When to Use

Use this model for monoliths, low-to-moderate traffic, and systems where one deployment unit is operationally preferable.

Real-World Use Cases

• A web application with one background worker process.
• A container service running API and worker task types.
• A small internal platform.

Example

def run_application_process(process_type: str) -> None:
    if process_type == "api":
        start_api_server()
        return

    if process_type == "outbox-publisher":
        run_outbox_publisher()
        return

    raise ValueError(f"Unsupported process type: {process_type}")

Dedicated Publisher Service

A separate service owns outbox publication.

Advantages

• Independent scaling.
• Clear operational ownership.
• Publisher deployments do not restart the API.
• Broker libraries and retry logic remain isolated.

Disadvantages

• Another service must be deployed and monitored.
• Database access must be managed across service boundaries.
• Schema compatibility requires coordination.

When to Use

Use a dedicated service when publication throughput differs significantly from API traffic or when messaging infrastructure has separate ownership.

Real-World Use Cases

• High-volume commerce events.
• Shared publishing infrastructure.
• Services with strict broker-security controls.

Example

def run_outbox_publisher() -> None:
    while True:
        published_count = publish_outbox_batch(
            database=database,
            broker=message_broker
        )

        if published_count == 0:
            sleep(1)

Serverless Outbox Publisher

A scheduled function polls the outbox and publishes a bounded batch.

Advantages

• No continuously running worker is required.
• Simple scheduling for low-volume systems.
• Automatic execution isolation.

Disadvantages

• Cold starts add latency.
• Database connection management becomes important.
• Execution-time limits constrain large backlogs.
• Frequent schedules can increase invocation cost.

When to Use

Use serverless polling for low-volume workloads where seconds or minutes of publication latency are acceptable.

Real-World Use Cases

• Administrative notifications.
• Small SaaS applications.
• Periodic integration exports.

Example

def handler(event, context) -> dict:
    total_published = 0

    for _ in range(10):
        published = publish_outbox_batch(
            database=database,
            broker=message_broker
        )

        total_published += published

        if published == 0:
            break

    return {
        "published_messages": total_published
    }

CloudFormation Example

The following simplified CloudFormation configuration creates a scheduled Lambda publisher, an EventBridge schedule, and permissions for publishing to an SNS topic. The relational database and networking configuration are omitted to keep the example focused.

AWSTemplateFormatVersion: "2010-09-09"
Transform: AWS::Serverless-2016-10-31

Parameters:
  DatabaseSecretArn:
    Type: String

Resources:
  IntegrationEventsTopic:
    Type: AWS::SNS::Topic
    Properties:
      TopicName: integration-events

  OutboxPublisherFunction:
    Type: AWS::Serverless::Function
    Properties:
      FunctionName: outbox-publisher
      Runtime: python3.12
      Handler: app.handler
      Timeout: 60
      MemorySize: 512
      Environment:
        Variables:
          DATABASE_SECRET_ARN: !Ref DatabaseSecretArn
          EVENTS_TOPIC_ARN: !Ref IntegrationEventsTopic
          BATCH_SIZE: "100"
      Policies:
        - SNSPublishMessagePolicy:
            TopicName: !GetAtt IntegrationEventsTopic.TopicName
        - Statement:
            - Effect: Allow
              Action:
                - secretsmanager:GetSecretValue
              Resource: !Ref DatabaseSecretArn
      Events:
        PublishSchedule:
          Type: ScheduleV2
          Properties:
            ScheduleExpression: rate(1 minute)
            FlexibleTimeWindow:
              Mode: "OFF"

For higher throughput, a continuously running container publisher or change-data-capture pipeline is usually more appropriate than a once-per-minute function.

Real-World Payment Workflow

Consider a payment service that records a successful capture and publishes a payment.captured event.

The event triggers:

• Order fulfilment.
• Customer receipt generation.
• Accounting updates.
• Fraud analytics.
• Revenue reporting.

The capture result and outbox event are stored atomically.

def record_payment_capture(
    database,
    payment_id: str,
    provider_reference: str,
    amount: str,
    currency: str
) -> None:
    event_id = generate_uuid()
    occurred_at = current_timestamp()

    with database.transaction() as transaction:
        updated = transaction.execute(
            """
            UPDATE payments
            SET
                status = 'captured',
                provider_reference = %s,
                captured_at = %s
            WHERE payment_id = %s
              AND status != 'captured'
            """,
            [
                provider_reference,
                occurred_at,
                payment_id
            ]
        )

        if updated.row_count == 0:
            return

        transaction.execute(
            """
            INSERT INTO outbox_messages (
                outbox_id,
                aggregate_type,
                aggregate_id,
                event_type,
                event_version,
                payload_json,
                headers_json,
                status,
                available_at,
                created_at
            )
            VALUES (
                %s,
                'payment',
                %s,
                'payment.captured',
                1,
                %s,
                %s,
                'pending',
                %s,
                %s
            )
            """,
            [
                event_id,
                payment_id,
                serialize_json({
                    "event_id": event_id,
                    "event_type": "payment.captured",
                    "event_version": 1,
                    "occurred_at": occurred_at,
                    "payment_id": payment_id,
                    "provider_reference": provider_reference,
                    "amount": amount,
                    "currency": currency
                }),
                serialize_json({
                    "correlation_id": payment_id
                }),
                occurred_at,
                occurred_at
            ]
        )

If the broker is unavailable, the payment remains captured and the outbox record waits. Once the broker recovers, the publisher sends the event.

The publisher may send the event twice if it crashes after broker acknowledgement. Fulfilment therefore uses the event ID as an idempotency key.

def create_fulfilment_request(
    database,
    event: dict
) -> None:
    with database.transaction() as transaction:
        inserted = transaction.execute(
            """
            INSERT INTO processed_messages (
                consumer_name,
                message_id
            )
            VALUES (
                'fulfilment-payment-consumer',
                %s
            )
            ON CONFLICT DO NOTHING
            """,
            [event["event_id"]]
        )

        if inserted.row_count == 0:
            return

        transaction.execute(
            """
            INSERT INTO fulfilment_requests (
                payment_id,
                status,
                created_at
            )
            VALUES (%s, 'pending', NOW())
            ON CONFLICT (payment_id) DO NOTHING
            """,
            [event["payment_id"]]
        )

The workflow remains safe across database failures, broker outages, publisher restarts, and duplicate message delivery.

Transactional Outbox vs Alternatives

The transactional outbox is most valuable when a service owns a database and must reliably notify other systems about committed business changes.

Testing the Outbox Pattern

Happy-path tests are not enough. The pattern exists specifically to handle failures between systems.

Important test scenarios include:

• Business update succeeds and outbox insert succeeds.
• Outbox insert fails and the business update rolls back.
• Broker is unavailable for several minutes.
• Publisher crashes before publication.
• Publisher crashes after broker acceptance.
• Two publisher instances claim work concurrently.
• A processing lease expires.
• A poison payload exceeds the retry limit.
• Published rows are cleaned up safely.
• An old event version is replayed after deployment.
• Consumers receive the same event multiple times.
• Events for one aggregate preserve required ordering.

def test_order_and_outbox_are_atomic(database):
    database.fail_next_insert_into("outbox_messages")

    try:
        confirm_order(
            database=database,
            order_id="order-100"
        )
    except DatabaseError:
        pass

    order = database.fetch_one(
        """
        SELECT status
        FROM orders
        WHERE order_id = 'order-100'
        """
    )

    outbox_count = database.fetch_value(
        """
        SELECT COUNT(*)
        FROM outbox_messages
        WHERE aggregate_id = 'order-100'
        """
    )

    assert order["status"] == "pending"
    assert outbox_count == 0

Failure injection is especially useful. Tests should terminate the publisher at specific points to confirm that no event is lost and duplicates remain harmless.

Common Mistakes

Production Checklist

• Use one local transaction: commit business data and the outbox record together.
• Use stable message IDs: preserve the same identifier across every retry.
• Store immutable event payloads: keep the original transaction meaning.
• Version event contracts: support old records during deployments and replay.
• Include correlation metadata: connect API requests, outbox records, and consumer logs.
• Index pending records: avoid full-table scans.
• Keep claim transactions short: do not call the broker while holding database locks.
• Use safe concurrent claiming: prevent multiple publishers from taking the same batch unnecessarily.
• Add processing leases: recover records abandoned after crashes.
• Publish in bounded batches: limit memory use and transaction size.
• Use broker acknowledgement: mark published only after acceptance.
• Expect duplicate publication: design for at-least-once delivery.
• Make consumers idempotent: deduplicate messages with business updates atomically.
• Use aggregate keys: preserve ordering where business rules require it.
• Avoid global ordering: partition by business entity where possible.
• Classify publication failures: separate transient and permanent problems.
• Use exponential backoff: prevent retry storms during broker outages.
• Add jitter: spread recovery traffic across time.
• Limit retry attempts: move poison records to a failed state.
• Create replay tooling: support controlled correction and republishing.
• Preserve original IDs during replay: keep duplicate protection effective.
• Monitor pending count: detect growing publication backlog.
• Monitor oldest pending age: detect messages that are not progressing.
• Monitor retry and failure rates: identify broker or deployment problems.
• Alert on critical failed events: do not allow important operations to remain hidden.
• Define retention: clean published records after an appropriate recovery window.
• Delete in batches: avoid large cleanup transactions.
• Consider partitioning at scale: simplify retention and reduce active-table size.
• Protect sensitive payloads: minimise, encrypt, redact, and restrict access.
• Test broker outages: verify durable backlog and automatic recovery.
• Test publisher crashes: confirm no message loss around acknowledgement.
• Test duplicate delivery: verify downstream side effects occur once.
• Test rolling deployments: confirm old event versions remain publishable and consumable.

Conclusion

The transactional outbox pattern provides a practical solution to the dual-write problem. Instead of trying to update a database and message broker atomically, the application records the business change and publication intent in one local transaction. A separate publisher then delivers the stored message asynchronously.

This design prevents committed business changes from silently losing their corresponding events. Broker downtime, network failures, and publisher crashes become recoverable because unpublished records remain durable in the source database.

The pattern does not provide automatic exactly-once processing. A publisher can send the same message more than once if it crashes after broker acceptance but before updating the outbox row. Stable message identifiers and idempotent consumers are therefore essential parts of the design.

A production implementation also needs efficient indexing, safe concurrent claiming, processing leases, bounded retries, failure isolation, retention, schema versioning, controlled replay, and monitoring based on backlog age. The outbox table is not only a persistence detail; it is an operational queue that requires ownership and visibility.

Key Takeaway

The transactional outbox pattern makes business updates and message-publication intent atomic, while asynchronous publishing, retries, and idempotent consumers provide reliable end-to-end delivery.

More Articles to Read

• Message Queues Explained: Producers, Consumers, and Brokers
• Event-Driven Architecture in Distributed Systems
• Message Delivery Guarantees: At-Most-Once vs At-Least-Once vs Exactly-Once
• Background Workers Explained: Designing Reliable Asynchronous Processing
• Dead-Letter Queues, Retries, and Poison Messages
• Message Queue Best Practices for Production Systems