Designing Distributed Transactions with Sagas and Two-Phase Commit

Distributed transactions coordinate one business operation across multiple services, databases, queues, or external systems. Unlike a local database transaction, no single transaction manager automatically guarantees that every participating component commits or rolls back together.

Sagas and Two-Phase Commit solve this problem using different consistency models. Sagas accept temporary inconsistency and recover through compensating actions, while Two-Phase Commit coordinates participants so they either commit or abort as one distributed unit.

Table of Contents

• The Distributed Transaction Problem
• Local vs Distributed Transactions
  • Local Atomicity
  • Partial Failure
  • Consistency Boundaries
• Saga Pattern
  • Advantages
  • Disadvantages
  • When to Use / Real-World Use Cases
  • Example
• Saga Choreography
  • Advantages
  • Disadvantages
  • When to Use / Real-World Use Cases
  • Example
• Saga Orchestration
  • Advantages
  • Disadvantages
  • When to Use / Real-World Use Cases
  • Example
• Two-Phase Commit
  • Advantages
  • Disadvantages
  • When to Use / Real-World Use Cases
  • Example
• Saga vs Two-Phase Commit
• Compensating Transactions
  • Compensation Is Not Rollback
  • Designing Compensations
  • Non-Compensatable Actions
• Saga State Machine
  • Saga States
  • Persistent Progress
  • Recovery After a Crash
• Message Delivery and Consistency
  • Transactional Outbox
  • Inbox and Deduplication
  • At-Least-Once Delivery
• Idempotency
  • Idempotent Commands
  • Idempotency Keys
  • Deduplication Storage
• Timeouts, Retries, and Backoff
  • Retryable Failures
  • Retry Budgets
  • Stuck Sagas
• Concurrency and Isolation
  • Semantic Locking
  • Optimistic Concurrency
  • Ordering and Race Conditions
• Production Design Example
  • Order Workflow
  • Failure Scenarios
  • Observability Model
• Ready-to-Use Example
  • PostgreSQL Schema
  • Python Saga Orchestrator
  • Outbox Publisher
  • Two-Phase Commit SQL Example
• Common Mistakes
• Production Checklist
• Conclusion

The Distributed Transaction Problem

A business operation often spans several independently deployed systems. Creating an order may require reserving inventory, authorizing payment, creating a shipment, and updating customer-facing status.

Create Order
  -> Order Service
  -> Inventory Service
  -> Payment Service
  -> Shipping Service
  -> Notification Service

Each participant can succeed or fail independently. The network can time out after an operation succeeds, a service can restart between steps, or a message can be delivered more than once.

Key Point: distributed transaction design is primarily failure design. The normal success path is usually the simplest part.

Local vs Distributed Transactions

Local and distributed transactions operate across different consistency boundaries. A relational database can atomically commit changes under one transaction manager, while independently operated services cannot assume one shared commit operation.

Local Atomicity

A local database transaction groups several statements into one atomic unit.

BEGIN;

-- Reserve inventory inside the same database transaction.
UPDATE inventory
SET available_quantity = available_quantity - 1
WHERE product_id = 'prd_123'
  AND available_quantity > 0;

-- Create the order only if the reservation succeeded.
INSERT INTO orders (
    id,
    product_id,
    status
)
VALUES (
    'ord_123',
    'prd_123',
    'created'
);

COMMIT;

If a statement fails, the database can restore the previous state because it controls all participating data.

Partial Failure

Distributed systems cannot assume every participant is available at the same moment. A timeout does not prove that an operation failed; it proves only that the result was not received in time.

Order Service
  -> Payment Service: authorize payment

Payment Service
  -> payment succeeds
  -> response is lost

Order Service
  -> sees timeout
  -> outcome is unknown

This uncertainty makes retries, idempotency, reconciliation, and persistent workflow state essential.

Consistency Boundaries

A system should keep strongly consistent invariants inside the smallest practical local transaction and coordinate broader workflows asynchronously.

Rule of Thumb: use local ACID transactions inside a service and explicit coordination between services.

Saga Pattern

A Saga represents one distributed business transaction as a sequence of local transactions. Each participant commits independently, and failures are handled through retries or compensating actions.

Advantages

• No long-lived distributed locks: services commit local work independently.
• Good service autonomy: each service owns its database and transaction boundary.
• Scales across heterogeneous systems: participants may use different databases or external APIs.
• Works with asynchronous messaging: workflows can continue through temporary outages.
• Supports long-running operations: a saga may take seconds, hours, or days.

Disadvantages

• Temporary inconsistency: intermediate states are visible before completion.
• Compensation complexity: every reversible side effect needs a business-level undo action.
• Harder reasoning: failure paths multiply as the number of steps grows.
• No automatic isolation: concurrent workflows may observe or modify intermediate state.
• Operational state required: progress, attempts, errors, and compensation status must be durable.

When to Use / Real-World Use Cases

• E-commerce checkout across orders, inventory, payments, and shipping.
• Travel booking across flights, hotels, and rental cars.
• Account provisioning across identity, billing, and infrastructure systems.
• Marketplace transactions involving buyers, sellers, payouts, and fulfillment.
• Long-running business workflows that cannot hold database locks.

Example

Forward steps:
  1. Create order
  2. Reserve inventory
  3. Authorize payment
  4. Create shipment

Compensations:
  4. Cancel shipment
  3. Void or refund payment
  2. Release inventory
  1. Mark order canceled

Compensation usually runs in reverse order because later actions may depend on earlier ones.

Saga Choreography

In a choreographed saga, services publish domain events and react to events from other services. No central component explicitly controls the entire workflow.

Advantages

• Loose runtime coupling: participants react to events rather than call one central controller.
• No orchestration service: fewer central workflow components.
• Natural event-driven architecture: domain events drive state transitions.
• Independent extensibility: new consumers can subscribe without changing the producer.

Disadvantages

• Workflow becomes implicit: the end-to-end process is distributed across event handlers.
• Difficult debugging: understanding one transaction requires tracing many services and topics.
• Event cycles: poorly designed reactions can create loops or unexpected cascades.
• Harder global policy: timeouts, retries, and compensation rules are spread across participants.
• Testing complexity: end-to-end behavior depends on many independently deployed consumers.

When to Use / Real-World Use Cases

• Small workflows with a limited number of participants.
• Event-driven domains where services already communicate through domain events.
• Independent reactions such as analytics, email, and search indexing.
• Workflows without complex branching or centralized deadlines.

Example

Order Service
  -> publishes OrderCreated

Inventory Service
  -> consumes OrderCreated
  -> reserves inventory
  -> publishes InventoryReserved

Payment Service
  -> consumes InventoryReserved
  -> authorizes payment
  -> publishes PaymentAuthorized

Shipping Service
  -> consumes PaymentAuthorized
  -> creates shipment
  -> publishes ShipmentCreated

If payment fails, the Payment Service may publish PaymentAuthorizationFailed, which causes the Inventory Service to release its reservation and the Order Service to cancel the order.

Saga Orchestration

In an orchestrated saga, one component stores workflow state and commands each participant to perform or compensate a step.

Advantages

• Explicit workflow: sequence, branches, timeouts, and compensations are visible in one place.
• Central observability: the orchestrator can expose end-to-end saga status.
• Easier complex logic: branching, parallel steps, deadlines, and retries are easier to model.
• Controlled compensation: rollback order and policies are centrally defined.
• Clear ownership: one component owns workflow progression.

Disadvantages

• Central dependency: the orchestrator must be highly available and durable.
• Potential business-logic concentration: too much domain knowledge may move into one service.
• Command coupling: the orchestrator depends on participant command contracts.
• State-machine complexity: every transition and failure path must be modeled.
• Risk of a distributed monolith: one orchestrator may coordinate unrelated domains.

When to Use / Real-World Use Cases

• Checkout workflows with sequential dependencies and compensations.
• Long-running provisioning with deadlines and human approval steps.
• Complex financial workflows requiring explicit audit history.
• Processes with branching logic or parallel execution.
• Systems requiring a clear operational dashboard for workflow status.

Example

Order Saga Orchestrator
  -> ReserveInventory command
  <- InventoryReserved event

  -> AuthorizePayment command
  <- PaymentAuthorized event

  -> CreateShipment command
  <- ShipmentCreated event

  -> MarkOrderCompleted command

When payment authorization fails, the orchestrator sends ReleaseInventory and then marks the order canceled.

Two-Phase Commit

Two-Phase Commit, commonly abbreviated as 2PC, coordinates several transactional participants through a prepare phase and a commit phase.

Advantages

• Strong atomic outcome: all participants commit or all abort.
• No business compensation required: uncommitted changes are rolled back transactionally.
• Simple application semantics: the operation appears as one transaction.
• Useful for tightly controlled infrastructure: participants share a compatible transaction protocol.

Disadvantages

• Blocking behavior: prepared participants may hold locks while waiting for the final decision.
• Coordinator dependency: failure after prepare can leave participants uncertain.
• Reduced availability: all participants generally need to be reachable for progress.
• Operational complexity: in-doubt prepared transactions require monitoring and recovery.
• Limited interoperability: many queues, NoSQL stores, SaaS APIs, and cloud services do not participate.
• Poor long-running fit: locks and reserved resources should not remain open for minutes or hours.

When to Use / Real-World Use Cases

• Short transactions across compatible relational databases.
• Legacy enterprise systems using XA-compatible resource managers.
• Tightly controlled environments where participants and coordinator are operated together.
• High-value invariants that require atomic commit and cannot tolerate compensation.
• Database federation scenarios where lock duration remains very short.

Example

Phase 1: Prepare

Coordinator
  -> Database A: prepare
  -> Database B: prepare

Database A:
  records changes
  keeps locks
  votes YES

Database B:
  records changes
  keeps locks
  votes YES

Phase 2: Commit

Coordinator
  -> Database A: commit
  -> Database B: commit

If any participant votes no during prepare, the coordinator tells all participants to abort.

Saga vs Two-Phase Commit

Sagas and Two-Phase Commit optimize for different system properties. A Saga favors availability, service autonomy, and long-running workflows. Two-Phase Commit favors atomic consistency within a tightly coordinated environment.

Rule of Thumb: use Sagas across autonomous services. Consider Two-Phase Commit only when participants support it, transactions are short, and atomicity is worth the availability and operational cost.

Compensating Transactions

Compensation restores an acceptable business state after already committed steps cannot be completed as planned.

Compensation Is Not Rollback

A database rollback erases uncommitted changes. A compensation creates new committed changes that semantically reverse or offset earlier ones.

A refunded payment is not identical to a payment that never happened. Fees, audit history, exchange rates, and customer-visible entries may remain.

Designing Compensations

Compensations should be idempotent, retryable, observable, and safe when the original action did not complete.

• Use stable business identifiers such as reservation or payment IDs.
• Record original outcomes before choosing compensation.
• Make cancellation commands repeatable.
• Separate compensation status from forward-step status.
• Define manual recovery when automatic compensation fails repeatedly.

Non-Compensatable Actions

Some actions cannot be fully reversed. Emails may already be read, external transfers may have settled, and physical goods may have shipped.

Irreversible steps should be delayed until the workflow has passed its highest-risk failure points.

Safer order:

1. Reserve inventory
2. Authorize payment
3. Validate shipment
4. Capture payment
5. Create shipment
6. Send confirmation email

Irreversible or costly actions happen later.

Production Note: compensation design often determines step order.

Saga State Machine

An orchestrated saga should be represented as a durable state machine rather than a chain of in-memory method calls.

Saga States

Explicit states make transitions, retries, and recovery auditable.

PENDING
  -> INVENTORY_RESERVING
  -> PAYMENT_AUTHORIZING
  -> SHIPMENT_CREATING
  -> COMPLETED

Failure path:
  PAYMENT_FAILED
  -> INVENTORY_RELEASING
  -> COMPENSATED

Every incoming command result should be valid only for specific current states.

Persistent Progress

The orchestrator must persist progress before or together with publishing the next command.

Recovery After a Crash

After restart, the orchestrator should find incomplete sagas and continue from persisted state.

Orchestrator crashes after publishing AuthorizePayment

After restart:
  - load saga state
  - receive duplicate or delayed payment result
  - validate current state
  - continue exactly once logically
  - tolerate at-least-once message delivery

Recovery should not depend on reconstructing state from application logs.

Message Delivery and Consistency

A saga often requires a service to update its database and publish an event. Performing those operations independently creates a dual-write problem.

Transactional Outbox

The transactional outbox stores the business change and outgoing message in one local database transaction.

Local database transaction:

1. Update order
2. Insert outbox message
3. Commit

Separate publisher:
4. Read unpublished outbox messages
5. Publish to broker
6. Mark as published

This guarantees that an event is not lost after the business change commits. It does not guarantee one-time publication, so consumers must remain idempotent.

Inbox and Deduplication

The inbox pattern records processed message IDs in the consumer’s database.

BEGIN;

-- Ignore a duplicate delivery that was already processed.
INSERT INTO message_inbox (
    consumer_name,
    message_id,
    processed_at
)
VALUES (
    'inventory-service',
    :message_id,
    now()
)
ON CONFLICT DO NOTHING;

-- Apply the business change only when the inbox record was newly inserted.

COMMIT;

The inbox row and business update should commit in the same transaction.

At-Least-Once Delivery

Most practical message systems provide at-least-once delivery. A message may be redelivered after consumer failure or acknowledgement loss.

Idempotency

Every saga command, event handler, and compensation should tolerate duplicate execution.

Idempotent Commands

A participant should return the existing result when it receives the same logical command again.

ReserveInventory
  saga_id: saga_123
  command_id: cmd_456
  order_id: ord_789

First delivery:
  creates reservation res_100

Duplicate delivery:
  returns existing reservation res_100

Idempotency Keys

Stable keys connect repeated attempts to one logical operation.

Deduplication Storage

Deduplication state should usually live in the same database transaction as the side effect.

BEGIN;

-- Reserve this command ID. Duplicate attempts insert nothing.
INSERT INTO processed_commands (
    command_id,
    processed_at
)
VALUES (
    :command_id,
    now()
)
ON CONFLICT DO NOTHING;

-- Continue only when the insert affected one row.
-- Apply the inventory reservation in the same transaction.

COMMIT;

Cache-only deduplication can fail if the cache loses data while the business database retains the side effect.

Timeouts, Retries, and Backoff

Distributed workflows must distinguish temporary failures, permanent business failures, and unknown outcomes.

Retryable Failures

Retry Budgets

Infinite retries hide broken workflows and consume capacity. Each step should define maximum attempts, maximum elapsed time, and escalation behavior.

Payment authorization policy:

initial delay: 1 second
maximum delay: 30 seconds
backoff: exponential
jitter: enabled
maximum attempts: 5
final action: start compensation and alert

Stuck Sagas

A watchdog should identify sagas that remain in one state longer than expected.

• Retry the current command when safe.
• Query participant status when the outcome is unknown.
• Start compensation after a business deadline.
• Move to manual review when automated resolution is unsafe.
• Alert with saga ID and current state.

Concurrency and Isolation

Sagas do not automatically isolate intermediate changes from other transactions. Concurrent workflows may compete for the same resources.

Semantic Locking

Semantic locking represents temporary ownership through business state rather than a long database lock.

Inventory item states:

AVAILABLE
  -> RESERVED by saga_123
  -> SOLD

or

RESERVED
  -> RELEASED after compensation

Other workflows should respect the reservation state.

Optimistic Concurrency

Version columns prevent two workers from applying conflicting transitions.

UPDATE sagas
SET
    state = :next_state,
    version = version + 1,
    updated_at = now()
WHERE id = :saga_id
  AND version = :expected_version;

If no row is updated, another worker changed the saga and the transition should be reloaded.

Ordering and Race Conditions

A compensation result may arrive after a retry succeeds, or an old event may arrive after the saga advances.

Current saga state: PAYMENT_AUTHORIZED

Late event arrives:
  InventoryReservationFailed

Handler:
  validate whether event is valid for current state
  ignore or record stale event
  never blindly move state backward

Every handler should verify the current state and expected command or correlation ID.

Production Design Example

Consider an order workflow that reserves inventory, authorizes payment, and creates a shipment.

Order Workflow

1. Order Service creates order with status PENDING
2. Orchestrator sends ReserveInventory
3. Inventory Service creates reservation
4. Orchestrator sends AuthorizePayment
5. Payment Service authorizes payment
6. Orchestrator sends CreateShipment
7. Shipping Service creates shipment
8. Orchestrator marks order COMPLETED

Each participant persists its local state and outgoing result through one local transaction and an outbox record.

Failure Scenarios

Observability Model

Every log, metric, command, and event should carry the saga ID and business entity ID.

{
  "event": "payment_authorized",
  "saga_id": "saga_123",
  "order_id": "ord_789",
  "command_id": "cmd_456",
  "payment_id": "pay_100",
  "saga_state": "PAYMENT_AUTHORIZING",
  "attempt": 2
}

Ready-to-Use Example

The following example implements a durable orchestrated saga with PostgreSQL, a transactional outbox, optimistic concurrency, idempotent message processing, and explicit compensation states.

PostgreSQL Schema

CREATE TABLE order_sagas (
    id text PRIMARY KEY,
    order_id text NOT NULL UNIQUE,
    state text NOT NULL,
    version integer NOT NULL DEFAULT 1,
    payload jsonb NOT NULL DEFAULT '{}'::jsonb,
    attempt_count integer NOT NULL DEFAULT 0,
    last_error text,
    created_at timestamptz NOT NULL DEFAULT now(),
    updated_at timestamptz NOT NULL DEFAULT now()
);

-- Finds incomplete workflows for watchdog and recovery processing.
CREATE INDEX order_sagas_incomplete_idx
ON order_sagas (
    updated_at
)
WHERE state NOT IN ('COMPLETED', 'COMPENSATED', 'MANUAL_REVIEW');

CREATE TABLE outbox_messages (
    id text PRIMARY KEY,
    aggregate_type text NOT NULL,
    aggregate_id text NOT NULL,
    message_type text NOT NULL,
    payload jsonb NOT NULL,
    created_at timestamptz NOT NULL DEFAULT now(),
    published_at timestamptz,
    publish_attempts integer NOT NULL DEFAULT 0
);

-- Allows publishers to scan unpublished messages efficiently.
CREATE INDEX outbox_messages_unpublished_idx
ON outbox_messages (
    created_at
)
WHERE published_at IS NULL;

CREATE TABLE inbox_messages (
    consumer_name text NOT NULL,
    message_id text NOT NULL,
    processed_at timestamptz NOT NULL DEFAULT now(),
    PRIMARY KEY (
        consumer_name,
        message_id
    )
);

CREATE TABLE inventory_reservations (
    id text PRIMARY KEY,
    saga_id text NOT NULL UNIQUE,
    order_id text NOT NULL,
    product_id text NOT NULL,
    quantity integer NOT NULL CHECK (quantity > 0),
    status text NOT NULL,
    created_at timestamptz NOT NULL DEFAULT now(),
    updated_at timestamptz NOT NULL DEFAULT now()
);

Python Saga Orchestrator

from __future__ import annotations

import json
from dataclasses import dataclass
from enum import StrEnum
from typing import Any
from uuid import uuid4


class SagaState(StrEnum):
    PENDING = "PENDING"
    INVENTORY_RESERVING = "INVENTORY_RESERVING"
    PAYMENT_AUTHORIZING = "PAYMENT_AUTHORIZING"
    SHIPMENT_CREATING = "SHIPMENT_CREATING"
    INVENTORY_RELEASING = "INVENTORY_RELEASING"
    PAYMENT_VOIDING = "PAYMENT_VOIDING"
    COMPLETED = "COMPLETED"
    COMPENSATED = "COMPENSATED"
    MANUAL_REVIEW = "MANUAL_REVIEW"


@dataclass(frozen=True)
class Saga:
    id: str
    order_id: str
    state: SagaState
    version: int
    payload: dict[str, Any]


class ConcurrentSagaUpdate(Exception):
    pass


class InvalidSagaTransition(Exception):
    pass


class SagaRepository:
    def __init__(self, connection: Any):
        self.connection = connection

    def load(self, saga_id: str) -> Saga:
        row = self.connection.execute(
            """
            SELECT id, order_id, state, version, payload
            FROM order_sagas
            WHERE id = %s
            """,
            (saga_id,),
        ).fetchone()

        if row is None:
            raise KeyError(f"Saga not found: {saga_id}")

        return Saga(
            id=row["id"],
            order_id=row["order_id"],
            state=SagaState(row["state"]),
            version=row["version"],
            payload=row["payload"],
        )

    def transition(
        self,
        saga: Saga,
        next_state: SagaState,
        payload: dict[str, Any],
        command_type: str | None = None,
        command_payload: dict[str, Any] | None = None,
    ) -> None:
        # Persist the saga transition and outgoing command in one transaction.
        with self.connection.transaction():
            result = self.connection.execute(
                """
                UPDATE order_sagas
                SET
                    state = %s,
                    version = version + 1,
                    payload = %s::jsonb,
                    updated_at = now()
                WHERE id = %s
                  AND version = %s
                """,
                (
                    next_state.value,
                    json.dumps(payload),
                    saga.id,
                    saga.version,
                ),
            )

            # Optimistic concurrency prevents two workers from progressing
            # the same saga from the same old state.
            if result.rowcount != 1:
                raise ConcurrentSagaUpdate(saga.id)

            if command_type is not None:
                self.connection.execute(
                    """
                    INSERT INTO outbox_messages (
                        id,
                        aggregate_type,
                        aggregate_id,
                        message_type,
                        payload
                    )
                    VALUES (%s, 'order_saga', %s, %s, %s::jsonb)
                    """,
                    (
                        f"msg_{uuid4().hex}",
                        saga.id,
                        command_type,
                        json.dumps(command_payload or {}),
                    ),
                )


class OrderSagaOrchestrator:
    def __init__(self, repository: SagaRepository):
        self.repository = repository

    def start(self, saga_id: str) -> None:
        saga = self.repository.load(saga_id)

        if saga.state != SagaState.PENDING:
            raise InvalidSagaTransition(
                f"Cannot start saga from state {saga.state}"
            )

        command_id = f"cmd_{uuid4().hex}"

        self.repository.transition(
            saga=saga,
            next_state=SagaState.INVENTORY_RESERVING,
            payload={
                **saga.payload,
                "inventory_command_id": command_id,
            },
            command_type="ReserveInventory",
            command_payload={
                "command_id": command_id,
                "saga_id": saga.id,
                "order_id": saga.order_id,
                "product_id": saga.payload["product_id"],
                "quantity": saga.payload["quantity"],
            },
        )

    def on_inventory_reserved(
        self,
        saga_id: str,
        reservation_id: str,
    ) -> None:
        saga = self.repository.load(saga_id)

        # Duplicate messages are harmless after the saga already advanced.
        if saga.state != SagaState.INVENTORY_RESERVING:
            return

        command_id = f"cmd_{uuid4().hex}"

        self.repository.transition(
            saga=saga,
            next_state=SagaState.PAYMENT_AUTHORIZING,
            payload={
                **saga.payload,
                "reservation_id": reservation_id,
                "payment_command_id": command_id,
            },
            command_type="AuthorizePayment",
            command_payload={
                "command_id": command_id,
                "saga_id": saga.id,
                "order_id": saga.order_id,
                "amount_cents": saga.payload["amount_cents"],
                "payment_method_id": saga.payload["payment_method_id"],
            },
        )

    def on_payment_authorized(
        self,
        saga_id: str,
        payment_id: str,
    ) -> None:
        saga = self.repository.load(saga_id)

        if saga.state != SagaState.PAYMENT_AUTHORIZING:
            return

        command_id = f"cmd_{uuid4().hex}"

        self.repository.transition(
            saga=saga,
            next_state=SagaState.SHIPMENT_CREATING,
            payload={
                **saga.payload,
                "payment_id": payment_id,
                "shipment_command_id": command_id,
            },
            command_type="CreateShipment",
            command_payload={
                "command_id": command_id,
                "saga_id": saga.id,
                "order_id": saga.order_id,
                "address_id": saga.payload["address_id"],
            },
        )

    def on_shipment_created(
        self,
        saga_id: str,
        shipment_id: str,
    ) -> None:
        saga = self.repository.load(saga_id)

        if saga.state != SagaState.SHIPMENT_CREATING:
            return

        self.repository.transition(
            saga=saga,
            next_state=SagaState.COMPLETED,
            payload={
                **saga.payload,
                "shipment_id": shipment_id,
            },
            command_type="MarkOrderCompleted",
            command_payload={
                "command_id": f"cmd_{uuid4().hex}",
                "saga_id": saga.id,
                "order_id": saga.order_id,
            },
        )

    def on_payment_failed(
        self,
        saga_id: str,
        error_code: str,
    ) -> None:
        saga = self.repository.load(saga_id)

        if saga.state != SagaState.PAYMENT_AUTHORIZING:
            return

        # Payment did not complete, so only inventory requires compensation.
        self.repository.transition(
            saga=saga,
            next_state=SagaState.INVENTORY_RELEASING,
            payload={
                **saga.payload,
                "failure_code": error_code,
            },
            command_type="ReleaseInventory",
            command_payload={
                "command_id": f"cmd_{uuid4().hex}",
                "saga_id": saga.id,
                "reservation_id": saga.payload["reservation_id"],
            },
        )

    def on_shipment_failed(
        self,
        saga_id: str,
        error_code: str,
    ) -> None:
        saga = self.repository.load(saga_id)

        if saga.state != SagaState.SHIPMENT_CREATING:
            return

        # Shipment failed after payment authorization.
        # Void payment first, then release inventory.
        self.repository.transition(
            saga=saga,
            next_state=SagaState.PAYMENT_VOIDING,
            payload={
                **saga.payload,
                "failure_code": error_code,
            },
            command_type="VoidPayment",
            command_payload={
                "command_id": f"cmd_{uuid4().hex}",
                "saga_id": saga.id,
                "payment_id": saga.payload["payment_id"],
            },
        )

    def on_payment_voided(self, saga_id: str) -> None:
        saga = self.repository.load(saga_id)

        if saga.state != SagaState.PAYMENT_VOIDING:
            return

        self.repository.transition(
            saga=saga,
            next_state=SagaState.INVENTORY_RELEASING,
            payload=saga.payload,
            command_type="ReleaseInventory",
            command_payload={
                "command_id": f"cmd_{uuid4().hex}",
                "saga_id": saga.id,
                "reservation_id": saga.payload["reservation_id"],
            },
        )

    def on_inventory_released(self, saga_id: str) -> None:
        saga = self.repository.load(saga_id)

        if saga.state != SagaState.INVENTORY_RELEASING:
            return

        self.repository.transition(
            saga=saga,
            next_state=SagaState.COMPENSATED,
            payload=saga.payload,
            command_type="MarkOrderCanceled",
            command_payload={
                "command_id": f"cmd_{uuid4().hex}",
                "saga_id": saga.id,
                "order_id": saga.order_id,
                "reason": saga.payload.get("failure_code"),
            },
        )

Production Note: the example keeps transaction handling inside the repository so every state transition and outgoing command commit together.

Outbox Publisher

from __future__ import annotations

import asyncio
import json
from typing import Any


class OutboxPublisher:
    def __init__(
        self,
        connection: Any,
        message_broker: Any,
    ):
        self.connection = connection
        self.message_broker = message_broker

    async def publish_batch(self, batch_size: int = 100) -> int:
        # Lock rows so concurrent publisher processes do not publish
        # the same outbox record at the same time.
        with self.connection.transaction():
            rows = self.connection.execute(
                """
                SELECT
                    id,
                    aggregate_id,
                    message_type,
                    payload
                FROM outbox_messages
                WHERE published_at IS NULL
                ORDER BY created_at
                FOR UPDATE SKIP LOCKED
                LIMIT %s
                """,
                (batch_size,),
            ).fetchall()

            published_ids: list[str] = []

            for row in rows:
                await self.message_broker.publish(
                    topic=row["message_type"],
                    key=row["aggregate_id"],
                    message={
                        "message_id": row["id"],
                        "message_type": row["message_type"],
                        "aggregate_id": row["aggregate_id"],
                        "payload": row["payload"],
                    },
                )

                published_ids.append(row["id"])

            if published_ids:
                self.connection.execute(
                    """
                    UPDATE outbox_messages
                    SET
                        published_at = now(),
                        publish_attempts = publish_attempts + 1
                    WHERE id = ANY(%s)
                    """,
                    (published_ids,),
                )

        # A crash after broker publication but before published_at commits
        # can cause duplicate publication. Consumers must remain idempotent.
        return len(rows)


async def run_publisher(publisher: OutboxPublisher) -> None:
    while True:
        published_count = await publisher.publish_batch()

        # Poll less aggressively when no work exists.
        if published_count == 0:
            await asyncio.sleep(0.5)

Two-Phase Commit SQL Example

PostgreSQL prepared transactions can demonstrate the database side of Two-Phase Commit. Production use also requires a coordinator that durably records the final decision.

-- Participant A: reserve money in the billing database.
BEGIN;

UPDATE accounts
SET balance_cents = balance_cents - 5000
WHERE account_id = 'acct_source'
  AND balance_cents >= 5000;

-- Prepare instead of committing.
-- PostgreSQL keeps the transaction durable and retains required locks.
PREPARE TRANSACTION 'transfer_123_billing';


-- Participant B: credit money in another database connection.
BEGIN;

UPDATE accounts
SET balance_cents = balance_cents + 5000
WHERE account_id = 'acct_destination';

PREPARE TRANSACTION 'transfer_123_ledger';


-- Coordinator records the COMMIT decision durably before notifying participants.

-- Final phase on participant A.
COMMIT PREPARED 'transfer_123_billing';

-- Final phase on participant B.
COMMIT PREPARED 'transfer_123_ledger';

If prepare fails on either participant, the coordinator should abort every participant that prepared successfully:

ROLLBACK PREPARED 'transfer_123_billing';
ROLLBACK PREPARED 'transfer_123_ledger';

Important: abandoned prepared transactions can retain locks and transaction metadata. Monitor them and define an explicit recovery procedure.

Common Mistakes

Distributed transaction failures are usually caused by incomplete failure modeling rather than the absence of a particular framework.

• Treating a Saga as an automatic rollback mechanism.
• Publishing events separately from database commits without an outbox.
• Assuming exactly-once message delivery.
• Executing non-idempotent participant commands.
• Retriying unknown payment outcomes with a new idempotency key.
• Keeping Saga state only in application memory.
• Skipping optimistic concurrency when several workers can process the same Saga.
• Compensating steps in the wrong order.
• Performing irreversible actions too early.
• Ignoring compensation failures.
• Allowing infinite retries without deadlines or manual escalation.
• Using choreography for a workflow too complex to understand operationally.
• Using one orchestrator for unrelated business domains.
• Choosing Two-Phase Commit across unreliable or unsupported participants.
• Leaving prepared transactions unresolved.
• Assuming Saga-based systems provide transaction isolation automatically.

Production Checklist

A production distributed transaction design should make progress, duplication, compensation, and manual recovery explicit.

• Keep strong invariants inside local ACID transactions.
• Choose Saga or Two-Phase Commit from consistency and availability requirements.
• Use orchestration for complex workflows with branching and deadlines.
• Use choreography only when the event flow remains understandable.
• Persist Saga state and version every transition.
• Store outgoing messages through a transactional outbox.
• Deduplicate incoming commands and events.
• Make every forward and compensating action idempotent.
• Use stable operation and idempotency identifiers.
• Classify retryable, permanent, and unknown failures.
• Use exponential backoff with jitter.
• Define retry limits and business deadlines.
• Detect stuck Sagas with a watchdog.
• Define manual-review states and operator procedures.
• Delay irreversible operations until late in the workflow.
• Protect state transitions with optimistic concurrency.
• Validate incoming events against current Saga state.
• Include Saga IDs in logs, metrics, commands, and events.
• Monitor completion, compensation, duration, and stuck-state metrics.
• For 2PC, monitor prepared transactions and coordinator recovery.
• Load-test participant failures, duplicate delivery, and orchestrator restarts.

Conclusion

Sagas and Two-Phase Commit address distributed consistency through different trade-offs. Sagas commit local work independently and recover through compensation, making them suitable for autonomous services, heterogeneous infrastructure, and long-running workflows.

Two-Phase Commit provides an atomic distributed outcome but introduces blocking, coordinator dependency, retained locks, and stricter participant requirements. It is best reserved for short transactions across compatible systems where atomic commit is more important than availability and autonomy.

Key Takeaway: use local transactions for local invariants, Sagas for most cross-service business workflows, and Two-Phase Commit only when a tightly controlled environment truly requires atomic distributed commit.

More Articles to Read

• Consistency Models in Distributed Systems
• CAP Theorem: Practical Trade-Offs and Real-World Examples
• Idempotency and Deduplication in Distributed Systems
• Leader Election and Distributed Coordination
• Handling Partial Failures in Production Systems