Idempotency and Deduplication in Distributed Systems

Idempotency allows the same logical operation to be executed multiple times without producing additional side effects. Deduplication identifies repeated requests, commands, or events and prevents them from being processed as new work.

These patterns are essential because distributed systems cannot safely assume that requests are delivered once, responses are never lost, or a timeout means an operation failed. Reliable systems are designed around retries, duplicate delivery, uncertain outcomes, and workers that may crash at any point.

Table of Contents

• Why Duplicates Happen
• Idempotency vs Deduplication
  • Idempotent Operation
  • Deduplicated Processing
  • Why Both Are Needed
• Naturally Idempotent Operations
  • Set vs Increment
  • Replace vs Append
  • Conditional State Transitions
• Idempotency Keys
  • Key Generation
  • Key Scope
  • Binding Keys to Request Payloads
  • Storing Results
• HTTP Request Idempotency
  • Request Lifecycle
  • Concurrent Retries
  • Failed Requests
  • Response Replay
• Message Deduplication
  • At-Least-Once Delivery
  • Inbox Pattern
  • Consumer Idempotency
  • Ordering and Duplicates
• Deduplication Storage Strategies
  • Relational Database
  • Redis
  • Business Table Constraints
  • Broker-Level Deduplication
• Storage Strategy Comparison
• Transactional Inbox and Outbox
  • The Dual-Write Problem
  • Outbox Publication
  • Inbox Processing
• Exactly-Once Semantics
  • Exactly-Once Delivery
  • Exactly-Once Processing
  • Effectively-Once Outcomes
• Retention and Expiration
  • Deduplication Window
  • Cleanup Strategies
  • Late Duplicates
• Failure Scenarios
  • Response Lost After Success
  • Worker Crash After Side Effect
  • Duplicate In-Flight Requests
  • Deduplication Store Failure
• Production Design Example
  • Payment Request Flow
  • Event Consumer Flow
  • Observability
• Ready-to-Use Example
  • PostgreSQL Schema
  • FastAPI Idempotency Example
  • Message Consumer Example
  • Redis Deduplication Example
• Common Mistakes
• Production Checklist
• Conclusion
• More Articles to Read

Why Duplicates Happen

Duplicate execution is a normal consequence of reliable retry behavior. Clients, load balancers, message brokers, schedulers, and workers may all repeat an operation when they cannot confirm its outcome.

Client
  -> sends CreatePayment
  -> payment succeeds
  -> response is lost
  -> client times out
  -> client retries CreatePayment

From the client’s perspective, the first request has an unknown result. Retrying is correct, but processing the retry as a new payment is not.

Key Point: retries improve availability only when repeated execution is safe.

Idempotency vs Deduplication

Idempotency and deduplication are related but not identical. One controls the observable result of repeated execution; the other detects whether work has already been processed.

Idempotent Operation

An operation is idempotent when executing it several times produces the same final state as executing it once.

Set account status to ACTIVE

First execution:
  PENDING -> ACTIVE

Second execution:
  ACTIVE -> ACTIVE

Final state remains ACTIVE.

The operation may still perform internal reads or writes multiple times, but its externally visible business result does not multiply.

Deduplicated Processing

Deduplication checks a stable identifier and skips repeated work after the identifier has already been processed.

Message ID: msg_123

First delivery:
  inbox lookup: not found
  process message
  store msg_123

Second delivery:
  inbox lookup: found
  skip processing

Why Both Are Needed

Deduplication records can expire, stores can become temporarily unavailable, and two workers can race before either records completion. The underlying business operation should therefore be as idempotent as possible even when a deduplication layer exists.

Naturally Idempotent Operations

The safest idempotency mechanism is to design commands whose meaning naturally converges toward one state.

Set vs Increment

Setting a value is often idempotent. Incrementing a value is not.

Idempotent:
  Set inventory quantity to 10

Not idempotent:
  Increase inventory quantity by 10

-- Repeating this statement preserves the same final state.
UPDATE user_preferences
SET marketing_emails_enabled = false
WHERE user_id = :user_id;

-- Repeating this statement applies the side effect again.
UPDATE accounts
SET balance_cents = balance_cents + 1000
WHERE account_id = :account_id;

Commands that represent deltas usually require an operation identifier or ledger entry to prevent repeated application.

Replace vs Append

Replacing one resource representation can be idempotent. Appending a new item usually is not.

Conditional State Transitions

State transitions should validate the current state so stale or duplicate commands cannot reapply earlier work.

UPDATE orders
SET
    status = 'paid',
    paid_at = now()
WHERE id = :order_id
  AND status = 'payment_pending';

If the row is already paid, a duplicate command changes nothing. If it has moved to canceled, the transition is rejected instead of silently reversing business state.

Idempotency Keys

An idempotency key identifies one logical operation across repeated attempts. The server stores the outcome associated with the key and reuses it when the same operation is retried.

Key Generation

Keys should be unique, stable across retries, and generated before the first attempt.

Good:
  7ccf419e-f129-4ac8-9a31-6f522f3ae712

Also useful:
  order_123:payment_attempt_1

Weak:
  current timestamp only
  random key regenerated on every retry

A client-generated UUID is common for public APIs. Internally, a deterministic business key may work better when the logical operation already has a stable identity.

Key Scope

Keys should be scoped so unrelated clients or operations cannot collide.

Scoped identity:

tenant_id
+ endpoint or operation type
+ idempotency_key

Binding Keys to Request Payloads

Reusing one key with a different request body should be rejected. Otherwise, the server cannot determine which operation the key represents.

{
  "idempotency_key": "7ccf419e-f129-4ac8-9a31-6f522f3ae712",
  "request_hash": "sha256:8b7740...",
  "status": "completed",
  "resource_id": "pay_123"
}

The request hash should be calculated from a canonical representation so insignificant JSON formatting differences do not create different values.

Storing Results

The server can store either the created resource identifier or the complete response needed for deterministic replay.

HTTP Request Idempotency

HTTP request idempotency is most important for operations that create financial, inventory, messaging, or provisioning side effects.

Request Lifecycle

1. Client generates idempotency key
2. Server validates key and request payload
3. Server atomically reserves the key
4. Server performs business operation
5. Server stores final result
6. Server returns response
7. Retry receives stored result

The key reservation and business operation should share a transaction when both are stored in the same database.

Concurrent Retries

Two identical requests may arrive simultaneously. A read-before-write check is not sufficient because both requests may observe that the key does not exist.

Request A:
  lookup key -> missing

Request B:
  lookup key -> missing

Both perform the side effect.

Use a unique database constraint or atomic create-if-absent operation to ensure only one request owns the key.

INSERT INTO idempotency_records (
    tenant_id,
    operation,
    idempotency_key,
    request_hash,
    status
)
VALUES (
    :tenant_id,
    :operation,
    :idempotency_key,
    :request_hash,
    'processing'
)
ON CONFLICT DO NOTHING;

Failed Requests

Failure handling depends on whether the business operation was attempted and whether its outcome is known.

Response Replay

Repeated requests should return the same logical result, even when the original response was lost.

{
  "id": "pay_123",
  "status": "authorized",
  "amount_cents": 5000
}

The response may include a header indicating that the result came from an idempotency record, but client behavior should not depend on the distinction.

Message Deduplication

Message consumers should expect duplicate delivery even when the broker offers ordering or transactional publishing features.

At-Least-Once Delivery

At-least-once delivery favors durability: a broker redelivers a message when it cannot confirm successful processing.

Consumer receives message
  -> database update commits
  -> consumer crashes
  -> acknowledgement is never sent
  -> broker redelivers message

Redelivery is correct because the broker cannot see the database commit.

Inbox Pattern

The inbox pattern stores each processed message ID in the consumer’s local database.

BEGIN;

-- Reserve the message ID inside the same transaction as the business update.
INSERT INTO inbox_messages (
    consumer_name,
    message_id,
    processed_at
)
VALUES (
    'order-service',
    :message_id,
    now()
)
ON CONFLICT DO NOTHING;

-- Continue only when one row was inserted.
-- Apply the business state transition here.

COMMIT;

If the message ID already exists, the consumer acknowledges the duplicate without applying the business change again.

Consumer Idempotency

The inbox prevents repeated processing of the same message ID, but business constraints should also prevent equivalent operations under different message IDs.

Duplicate by message identity:
  msg_123 delivered twice

Duplicate by business identity:
  msg_123 and msg_456 both request
  shipment creation for order ord_789

A unique constraint on order_id in the shipments table can protect the business invariant even when upstream systems publish semantically duplicate messages.

Ordering and Duplicates

Deduplication does not solve out-of-order delivery. Consumers should validate aggregate version, sequence number, or current business state.

Expected:
  OrderCreated version 1
  OrderPaid version 2
  OrderShipped version 3

Actual delivery:
  version 1
  version 3
  version 2

Depending on the domain, the consumer may buffer later events, reject stale events, or retrieve the current aggregate state.

Deduplication Storage Strategies

Deduplication data can be stored in a relational database, Redis, the business table itself, or a broker-specific mechanism. Each option provides different consistency and retention behavior.

Relational Database

A relational database provides durable records, unique constraints, and the ability to commit deduplication state with the business update.

Advantages

• Transactional consistency with business data.
• Durable deduplication across process restarts.
• Strong uniqueness constraints during concurrent races.
• Useful audit history for financial and operational workflows.

Disadvantages

• Additional write load for every request or message.
• Storage growth requires retention and cleanup.
• Potential hot indexes under very high throughput.
• Cross-database workflows cannot share one local transaction.

When to Use / Real-World Use Cases

• Payments and billing.
• Order and inventory processing.
• Message consumers modifying relational data.
• Operations requiring durable audit records.

Example

CREATE TABLE processed_operations (
    scope text NOT NULL,
    operation_key text NOT NULL,
    result jsonb,
    processed_at timestamptz NOT NULL DEFAULT now(),
    PRIMARY KEY (
        scope,
        operation_key
    )
);

Redis

Redis can reserve keys with atomic commands and expiration, making it useful for high-throughput, bounded deduplication windows.

Advantages

• Low latency.
• Atomic create-if-absent through SET NX.
• Built-in expiration for bounded retention.
• Good fit for short-lived duplicate suppression.

Disadvantages

• Not transactionally coupled to another database.
• Data loss is possible depending on persistence configuration.
• Expired keys allow late duplicates.
• Redis success does not prove the business side effect succeeded.

When to Use / Real-World Use Cases

• Webhook duplicate suppression.
• Short API retry windows.
• High-volume non-financial events.
• Performance optimization in front of a durable idempotency layer.

Example

SET dedupe:webhook:evt_123 processing NX EX 86400

Only the first caller creates the key. Later callers observe that processing already started.

Business Table Constraints

A unique business constraint can make the side effect itself deduplicated.

Advantages

• Protects the real business invariant.
• No separate deduplication lookup is always required.
• Works during concurrent races.
• Remains correct even when message IDs differ.

Disadvantages

• Not every operation has a natural unique key.
• May not preserve the original response.
• Does not record every duplicate attempt.
• Complex operations may affect several tables.

When to Use / Real-World Use Cases

• One shipment per order.
• One reservation per Saga step.
• One ledger entry per operation ID.
• One user account per external identity.

Example

CREATE UNIQUE INDEX shipments_one_per_order_idx
ON shipments (order_id);

Broker-Level Deduplication

Some messaging systems provide duplicate detection or transactional consumption within limited boundaries.

Advantages

• Reduces duplicate delivery before messages reach consumers.
• Can simplify consumers within the supported broker model.
• May preserve ordering for one partition or message group.

Disadvantages

• Usually time-bounded.
• Does not protect external side effects.
• May only apply within one topic, queue, or producer session.
• Creates infrastructure-specific assumptions.

When to Use / Real-World Use Cases

• Additional defense alongside consumer idempotency.
• High-volume pipelines where reducing duplicate traffic is valuable.
• Broker-native stream processing within one transactional boundary.

Example

Producer assigns stable message ID:
  order-created:ord_123:v1

Broker suppresses repeated publication
inside its configured deduplication window.

Storage Strategy Comparison

The best design often combines several layers rather than relying on one mechanism.

Rule of Thumb: protect critical business invariants with database constraints and transactions, then use Redis or broker features as additional optimization layers.

Transactional Inbox and Outbox

Inbox and outbox patterns address the gap between local database transactions and asynchronous message delivery.

The Dual-Write Problem

Updating a database and publishing a message as separate operations can produce inconsistent outcomes.

Database commit succeeds
Message publication fails
  -> business state changed
  -> downstream systems never learn about it

Message publication succeeds
Database commit fails
  -> downstream systems react
  -> source business state does not exist

Outbox Publication

The outbox stores the outgoing event in the same local transaction as the business update.

BEGIN;

UPDATE orders
SET status = 'paid'
WHERE id = :order_id
  AND status = 'payment_pending';

INSERT INTO outbox_messages (
    id,
    aggregate_id,
    message_type,
    payload
)
VALUES (
    :message_id,
    :order_id,
    'OrderPaid',
    :payload
);

COMMIT;

A publisher later sends the outbox record to the broker. Duplicate publication remains possible, so consumers still need inbox deduplication.

Inbox Processing

The inbox and business change should commit together.

Receive event
  -> begin database transaction
  -> reserve message ID in inbox
  -> apply business state change
  -> create outgoing outbox event
  -> commit
  -> acknowledge message

This produces an effectively-once business outcome over at-least-once delivery.

Exactly-Once Semantics

“Exactly once” is often used imprecisely. Delivery, processing, and business effects are different guarantees.

Exactly-Once Delivery

Exactly-once delivery would mean that a message reaches a consumer once and only once. This is difficult to guarantee across failures because acknowledgements and network responses can be lost.

Exactly-Once Processing

Some stream-processing platforms can provide transactional processing within their own managed boundaries. The guarantee may not extend to an external email provider, payment gateway, or unrelated database.

Effectively-Once Outcomes

Most production systems aim for effectively-once business outcomes:

• Messages may be delivered repeatedly.
• Handlers may start repeatedly.
• Deduplication identifies completed work.
• Business constraints prevent repeated effects.
• Retries return the original result.

The transport can remain at least once while the business result behaves as if it happened once.

Retention and Expiration

Deduplication records cannot always be retained forever. The retention period determines how long a repeated operation remains protected.

Deduplication Window

The deduplication window should exceed the longest realistic retry or replay period.

Cleanup Strategies

• Delete by expiration timestamp in small batches.
• Partition tables by creation date and drop expired partitions.
• Archive critical records before removing them from the active table.
• Use Redis TTLs for non-critical short-lived suppression.
• Retain permanent business constraints even after response data expires.

Late Duplicates

After a deduplication record expires, an old request may be treated as new. Critical operations should therefore use stable business identifiers or ledger constraints that remain valid beyond the cache window.

Short-lived:
  idempotency response cache expires after 24 hours

Long-lived:
  unique payment operation ID remains in ledger permanently

Failure Scenarios

Idempotency design should be evaluated against specific crash points rather than only the normal execution path.

Response Lost After Success

1. Payment succeeds
2. Result is stored under idempotency key
3. HTTP response is lost
4. Client retries
5. Stored result is replayed

This is the primary scenario idempotency keys are designed to solve.

Worker Crash After Side Effect

1. Consumer receives message
2. External side effect succeeds
3. Consumer crashes before storing completion
4. Message is redelivered

The external system must support its own idempotency key or status lookup. A local inbox cannot atomically commit with an external payment or email provider.

Duplicate In-Flight Requests

A second request may arrive while the first is still processing.

Long-running operations should usually return an operation ID rather than hold duplicate HTTP connections open.

Deduplication Store Failure

When the deduplication store is unavailable, the system must choose between availability and duplicate risk.

• Fail closed: reject the operation until protection is restored.
• Fail open: continue and accept possible duplicates.
• Fallback to a durable business constraint.
• Queue the operation for later processing.

Payments, payouts, and inventory reservations generally require fail-closed behavior or a durable fallback.

Production Design Example

Consider a Payment API that accepts an HTTP request, calls an external provider, stores payment state, and publishes a PaymentAuthorized event.

Payment Request Flow

1. Client sends POST /payments with Idempotency-Key
2. API hashes canonical request payload
3. API reserves key with status PROCESSING
4. API creates local payment attempt
5. API calls provider using the same stable operation key
6. API stores provider result
7. API inserts outbox event
8. Local transaction commits
9. API marks idempotency record COMPLETED
10. Retry receives stored result

The external provider should receive the same key because the API cannot commit its database atomically with the provider’s system.

Event Consumer Flow

1. Order Service receives PaymentAuthorized
2. Start local database transaction
3. Insert message ID into inbox
4. Change order from PAYMENT_PENDING to PAID
5. Insert OrderPaid event into outbox
6. Commit
7. Acknowledge broker message

Observability

Logs and metrics should make duplicate behavior visible.

{
  "event": "idempotency_replay",
  "tenant_id": "org_123",
  "operation": "create_payment",
  "idempotency_key": "7ccf419e-f129-4ac8-9a31-6f522f3ae712",
  "record_status": "completed",
  "resource_id": "pay_123"
}

Ready-to-Use Example

The following implementation uses PostgreSQL for durable request idempotency and inbox processing, plus Redis for optional short-lived duplicate suppression.

PostgreSQL Schema

CREATE TABLE idempotency_records (
    tenant_id text NOT NULL,
    operation text NOT NULL,
    idempotency_key text NOT NULL,
    request_hash text NOT NULL,
    status text NOT NULL,
    response_status integer,
    response_body jsonb,
    resource_id text,
    error_code text,
    created_at timestamptz NOT NULL DEFAULT now(),
    updated_at timestamptz NOT NULL DEFAULT now(),
    expires_at timestamptz NOT NULL,
    PRIMARY KEY (
        tenant_id,
        operation,
        idempotency_key
    )
);

-- Finds old completed records for batched cleanup.
CREATE INDEX idempotency_records_expiration_idx
ON idempotency_records (
    expires_at
)
WHERE status IN ('completed', 'failed');

-- Detects records that may require recovery after a worker crash.
CREATE INDEX idempotency_records_processing_idx
ON idempotency_records (
    updated_at
)
WHERE status = 'processing';


CREATE TABLE inbox_messages (
    consumer_name text NOT NULL,
    message_id text NOT NULL,
    aggregate_id text,
    processed_at timestamptz NOT NULL DEFAULT now(),
    PRIMARY KEY (
        consumer_name,
        message_id
    )
);


CREATE TABLE payments (
    id text PRIMARY KEY,
    tenant_id text NOT NULL,
    operation_id text NOT NULL,
    amount_cents integer NOT NULL CHECK (amount_cents > 0),
    currency text NOT NULL,
    status text NOT NULL,
    provider_payment_id text,
    created_at timestamptz NOT NULL DEFAULT now(),
    updated_at timestamptz NOT NULL DEFAULT now(),
    UNIQUE (
        tenant_id,
        operation_id
    )
);


CREATE TABLE outbox_messages (
    id text PRIMARY KEY,
    aggregate_type text NOT NULL,
    aggregate_id text NOT NULL,
    message_type text NOT NULL,
    payload jsonb NOT NULL,
    created_at timestamptz NOT NULL DEFAULT now(),
    published_at timestamptz
);

CREATE INDEX outbox_messages_unpublished_idx
ON outbox_messages (
    created_at
)
WHERE published_at IS NULL;

FastAPI Idempotency Example

from __future__ import annotations

import hashlib
import json
from dataclasses import dataclass
from datetime import datetime, timedelta, timezone
from typing import Any
from uuid import uuid4

from fastapi import Depends, FastAPI, Header, HTTPException, status
from pydantic import BaseModel, Field

app = FastAPI(title="Idempotent Payments API")


class PaymentRequest(BaseModel):
    amount_cents: int = Field(gt=0)
    currency: str = Field(min_length=3, max_length=3)
    payment_method_id: str


@dataclass(frozen=True)
class Identity:
    tenant_id: str


def get_identity() -> Identity:
    # The tenant must come from validated authentication context,
    # not from a request body or query parameter.
    return Identity(tenant_id="org_123")


def canonical_request_hash(payload: dict[str, Any]) -> str:
    # Sorting keys and removing formatting differences creates a stable hash.
    canonical = json.dumps(
        payload,
        sort_keys=True,
        separators=(",", ":"),
    )

    return hashlib.sha256(canonical.encode("utf-8")).hexdigest()


class PaymentService:
    def __init__(
        self,
        connection: Any,
        provider: Any,
    ):
        self.connection = connection
        self.provider = provider

    async def create_payment(
        self,
        identity: Identity,
        idempotency_key: str,
        request: PaymentRequest,
    ) -> tuple[int, dict[str, Any]]:
        operation = "create_payment"
        request_hash = canonical_request_hash(request.model_dump())

        # Reserve the idempotency key before calling any external system.
        inserted = self.connection.execute(
            """
            INSERT INTO idempotency_records (
                tenant_id,
                operation,
                idempotency_key,
                request_hash,
                status,
                expires_at
            )
            VALUES (
                %s,
                %s,
                %s,
                %s,
                'processing',
                now() + interval '7 days'
            )
            ON CONFLICT DO NOTHING
            """,
            (
                identity.tenant_id,
                operation,
                idempotency_key,
                request_hash,
            ),
        )

        if inserted.rowcount == 0:
            existing = self.connection.execute(
                """
                SELECT
                    request_hash,
                    status,
                    response_status,
                    response_body
                FROM idempotency_records
                WHERE tenant_id = %s
                  AND operation = %s
                  AND idempotency_key = %s
                """,
                (
                    identity.tenant_id,
                    operation,
                    idempotency_key,
                ),
            ).fetchone()

            if existing["request_hash"] != request_hash:
                # One key must always represent one logical request.
                raise HTTPException(
                    status_code=status.HTTP_409_CONFLICT,
                    detail="Idempotency key was reused with a different payload.",
                )

            if existing["status"] == "completed":
                # Replay the original successful response.
                return (
                    existing["response_status"],
                    existing["response_body"],
                )

            if existing["status"] == "failed":
                # Replay a deterministic business failure.
                return (
                    existing["response_status"],
                    existing["response_body"],
                )

            # Another request may still be working or recovery may be required.
            raise HTTPException(
                status_code=status.HTTP_409_CONFLICT,
                detail="The operation is already processing.",
                headers={"Retry-After": "2"},
            )

        payment_id = f"pay_{uuid4().hex}"
        operation_id = f"{identity.tenant_id}:{idempotency_key}"

        try:
            # Send the same stable operation ID to the provider.
            # Provider retries must not create another charge.
            provider_result = await self.provider.authorize(
                operation_id=operation_id,
                amount_cents=request.amount_cents,
                currency=request.currency,
                payment_method_id=request.payment_method_id,
            )

            response_body = {
                "id": payment_id,
                "status": "authorized",
                "amount_cents": request.amount_cents,
                "currency": request.currency.upper(),
                "provider_payment_id": provider_result["id"],
            }

            with self.connection.transaction():
                self.connection.execute(
                    """
                    INSERT INTO payments (
                        id,
                        tenant_id,
                        operation_id,
                        amount_cents,
                        currency,
                        status,
                        provider_payment_id
                    )
                    VALUES (
                        %s,
                        %s,
                        %s,
                        %s,
                        %s,
                        'authorized',
                        %s
                    )
                    ON CONFLICT (tenant_id, operation_id) DO NOTHING
                    """,
                    (
                        payment_id,
                        identity.tenant_id,
                        operation_id,
                        request.amount_cents,
                        request.currency.upper(),
                        provider_result["id"],
                    ),
                )

                self.connection.execute(
                    """
                    INSERT INTO outbox_messages (
                        id,
                        aggregate_type,
                        aggregate_id,
                        message_type,
                        payload
                    )
                    VALUES (
                        %s,
                        'payment',
                        %s,
                        'PaymentAuthorized',
                        %s::jsonb
                    )
                    """,
                    (
                        f"msg_{uuid4().hex}",
                        payment_id,
                        json.dumps(response_body),
                    ),
                )

                # Persist the replayable response before returning it.
                self.connection.execute(
                    """
                    UPDATE idempotency_records
                    SET
                        status = 'completed',
                        response_status = 201,
                        response_body = %s::jsonb,
                        resource_id = %s,
                        updated_at = now()
                    WHERE tenant_id = %s
                      AND operation = %s
                      AND idempotency_key = %s
                    """,
                    (
                        json.dumps(response_body),
                        payment_id,
                        identity.tenant_id,
                        operation,
                        idempotency_key,
                    ),
                )

            return 201, response_body

        except KnownPaymentDecline as exc:
            response_body = {
                "error": "payment_declined",
                "message": str(exc),
            }

            # Store deterministic business failures so retries return
            # the same result instead of starting another provider attempt.
            self.connection.execute(
                """
                UPDATE idempotency_records
                SET
                    status = 'failed',
                    response_status = 422,
                    response_body = %s::jsonb,
                    error_code = 'payment_declined',
                    updated_at = now()
                WHERE tenant_id = %s
                  AND operation = %s
                  AND idempotency_key = %s
                """,
                (
                    json.dumps(response_body),
                    identity.tenant_id,
                    operation,
                    idempotency_key,
                ),
            )

            return 422, response_body

        except TimeoutError:
            # Do not delete the idempotency record because the provider
            # may have completed the operation before the timeout.
            self.connection.execute(
                """
                UPDATE idempotency_records
                SET
                    error_code = 'provider_outcome_unknown',
                    updated_at = now()
                WHERE tenant_id = %s
                  AND operation = %s
                  AND idempotency_key = %s
                """,
                (
                    identity.tenant_id,
                    operation,
                    idempotency_key,
                ),
            )

            raise HTTPException(
                status_code=status.HTTP_503_SERVICE_UNAVAILABLE,
                detail="Payment outcome is being verified.",
                headers={"Retry-After": "5"},
            )


@app.post("/v1/payments")
async def create_payment(
    request: PaymentRequest,
    idempotency_key: str = Header(
        alias="Idempotency-Key",
        min_length=16,
        max_length=200,
    ),
    identity: Identity = Depends(get_identity),
) -> dict[str, Any]:
    response_status, response_body = await payment_service.create_payment(
        identity=identity,
        idempotency_key=idempotency_key,
        request=request,
    )

    # A real implementation should return Response or JSONResponse
    # so the stored HTTP status can be replayed exactly.
    return {
        "status_code": response_status,
        "data": response_body,
    }

Production Note: provider timeouts require a reconciliation worker that queries the provider by the same stable operation ID and completes the local idempotency record.

Message Consumer Example

from __future__ import annotations

import json
from typing import Any
from uuid import uuid4


class OrderEventConsumer:
    CONSUMER_NAME = "order-service-payment-authorized"

    def __init__(self, connection: Any):
        self.connection = connection

    def handle_payment_authorized(
        self,
        message_id: str,
        payload: dict[str, Any],
    ) -> bool:
        order_id = payload["order_id"]
        payment_id = payload["payment_id"]

        with self.connection.transaction():
            inserted = self.connection.execute(
                """
                INSERT INTO inbox_messages (
                    consumer_name,
                    message_id,
                    aggregate_id
                )
                VALUES (%s, %s, %s)
                ON CONFLICT DO NOTHING
                """,
                (
                    self.CONSUMER_NAME,
                    message_id,
                    order_id,
                ),
            )

            if inserted.rowcount == 0:
                # The same message was already committed successfully.
                return False

            updated = self.connection.execute(
                """
                UPDATE orders
                SET
                    status = 'paid',
                    payment_id = %s,
                    updated_at = now()
                WHERE id = %s
                  AND status = 'payment_pending'
                """,
                (
                    payment_id,
                    order_id,
                ),
            )

            if updated.rowcount == 0:
                current = self.connection.execute(
                    """
                    SELECT status, payment_id
                    FROM orders
                    WHERE id = %s
                    """,
                    (order_id,),
                ).fetchone()

                # Treat the same completed business transition as idempotent.
                if (
                    current
                    and current["status"] == "paid"
                    and current["payment_id"] == payment_id
                ):
                    return False

                # Reject an invalid or conflicting state transition.
                raise RuntimeError(
                    f"Order {order_id} cannot accept payment {payment_id}"
                )

            event_payload = {
                "order_id": order_id,
                "payment_id": payment_id,
            }

            # Store the next event in the same transaction.
            self.connection.execute(
                """
                INSERT INTO outbox_messages (
                    id,
                    aggregate_type,
                    aggregate_id,
                    message_type,
                    payload
                )
                VALUES (
                    %s,
                    'order',
                    %s,
                    'OrderPaid',
                    %s::jsonb
                )
                """,
                (
                    f"msg_{uuid4().hex}",
                    order_id,
                    json.dumps(event_payload),
                ),
            )

        # Acknowledge the broker message only after the transaction commits.
        return True

Redis Deduplication Example

from __future__ import annotations

from typing import Any


class RedisDeduplicator:
    def __init__(self, redis_client: Any):
        self.redis = redis_client

    async def reserve(
        self,
        namespace: str,
        operation_id: str,
        ttl_seconds: int,
    ) -> bool:
        key = f"dedupe:{namespace}:{operation_id}"

        # SET NX is atomic: only the first worker reserves the operation.
        reserved = await self.redis.set(
            key,
            "processing",
            nx=True,
            ex=ttl_seconds,
        )

        return bool(reserved)

    async def mark_completed(
        self,
        namespace: str,
        operation_id: str,
        ttl_seconds: int,
    ) -> None:
        key = f"dedupe:{namespace}:{operation_id}"

        # Preserve the key after completion so later retries are suppressed.
        await self.redis.set(
            key,
            "completed",
            ex=ttl_seconds,
        )


async def process_webhook(
    event_id: str,
    payload: dict[str, Any],
    deduplicator: RedisDeduplicator,
) -> None:
    reserved = await deduplicator.reserve(
        namespace="payment-webhook",
        operation_id=event_id,
        ttl_seconds=7 * 24 * 60 * 60,
    )

    if not reserved:
        # Another worker is processing or already processed this event.
        return

    try:
        await apply_webhook_business_logic(payload)

        await deduplicator.mark_completed(
            namespace="payment-webhook",
            operation_id=event_id,
            ttl_seconds=7 * 24 * 60 * 60,
        )
    except Exception:
        # Deleting the key allows retry, but this is safe only when
        # the business operation is independently idempotent.
        await deduplicator.redis.delete(
            f"dedupe:payment-webhook:{event_id}"
        )
        raise

Redis alone is not sufficient for critical side effects unless the business operation has its own durable idempotency protection.

Common Mistakes

Idempotency failures usually appear only under retries, crashes, or concurrent execution, which makes them easy to miss during normal testing.

• Generating a new idempotency key for every retry.
• Checking for an existing key without a unique constraint.
• Reusing one key with a different request payload.
• Storing the key only after the side effect completes.
• Deleting processing records after an unknown timeout outcome.
• Assuming HTTP timeouts mean the server did nothing.
• Assuming message brokers deliver each event once.
• Recording an inbox message separately from the business update.
• Using Redis as the only protection for financial operations.
• Relying only on message IDs without protecting business-level uniqueness.
• Making increment, append, or charge operations retryable without operation IDs.
• Ignoring concurrent duplicate requests.
• Using deduplication without validating event ordering or aggregate version.
• Expiring records before the longest possible replay period.
• Promising exactly-once behavior beyond the actual transaction boundary.
• Failing open during deduplication-store outages for high-risk operations.
• Not monitoring records stuck in processing state.

Production Checklist

A production idempotency design should protect both request identity and the underlying business invariant.

• Assume every retryable operation may execute more than once.
• Design naturally idempotent commands where possible.
• Use stable idempotency keys across retries.
• Scope keys by tenant and operation.
• Bind each key to a canonical request hash.
• Reserve keys atomically before side effects.
• Use unique constraints to protect concurrent races.
• Store replayable results for completed operations.
• Preserve processing state when an external outcome is unknown.
• Send the same operation ID to external providers.
• Use transactional inbox processing for message consumers.
• Use a transactional outbox for outgoing messages.
• Protect business uniqueness with domain constraints.
• Validate aggregate state and event version.
• Choose retention longer than the maximum retry and replay period.
• Clean up expired records in controlled batches.
• Define fail-open or fail-closed behavior explicitly.
• Monitor duplicate rates, payload mismatches, and stuck records.
• Test crashes before and after every external side effect.
• Document the exact boundary of any exactly-once guarantee.

Conclusion

Idempotency and deduplication allow distributed systems to retry safely when responses are lost, messages are redelivered, workers crash, or several clients submit the same operation concurrently.

Idempotency should begin with business semantics: prefer state-setting commands, validate state transitions, and protect natural uniqueness with database constraints. Deduplication then adds stable request keys, inbox records, stored results, and bounded replay protection.

Key Takeaway: assume delivery is at least once, make business effects idempotent, deduplicate with atomic durable records, and use inbox and outbox transactions to produce effectively-once outcomes.

More Articles to Read

• Consistency Models in Distributed Systems
• CAP Theorem: Practical Trade-Offs and Real-World Examples
• Designing Distributed Transactions with Sagas and Two-Phase Commit
• Leader Election and Distributed Coordination
• Handling Partial Failures in Production Systems