What Is the Idempotent Consumer Pattern? Handling Duplicate Messages Reliably
Message brokers such as Amazon SQS, RabbitMQ, and Apache Kafka can deliver the same logical message more than once. A consumer may successfully update its database but crash before acknowledging the message. When the broker redelivers it, processing the same operation again can create duplicate orders, reserve inventory twice, or apply the same financial adjustment repeatedly.
The Idempotent Consumer Pattern prevents duplicate delivery from producing duplicate business effects. Instead of assuming every message arrives exactly once, the consumer uses a stable message identifier, database constraints, and transactional processing to recognize previously completed operations. This article explains the pattern, compares deduplication strategies, and implements a practical Python, FastAPI, and PostgreSQL example with concurrency-safe processing and integration tests.
Table of Contents
- What Is the Idempotent Consumer Pattern?
- Message Deduplication Strategies
- Transactions and Concurrency
- Production Example: Inventory Reservations
- Ready-to-Use Python and PostgreSQL Implementation
- Production Design Trade-offs
- Common Mistakes and Production Checklist
- Frequently Asked Questions
- Conclusion
What Is the Idempotent Consumer Pattern?
An idempotent consumer can process the same logical message repeatedly without applying its intended business effect more than once. It assumes message delivery may be repeated and makes processing safe through durable state or naturally idempotent business operations.
The pattern is especially important for consumers using at-least-once delivery, where reliability is achieved partly through redelivery after errors or missing acknowledgements.
Why Duplicate Messages Happen
Consider an inventory consumer processing an OrderCreated event. It reserves two keyboards and commits the transaction, but the worker crashes before acknowledging the message.
1. Consumer receives OrderCreated
2. Inventory reservation commits
3. Worker crashes before ACK
4. Broker redelivers OrderCreated
5. Another consumer receives the same event
If the second attempt reserves two more keyboards, one order incorrectly consumes four units. This is a correctness bug, not merely inefficient processing.
Duplicates can arise from consumer restarts, acknowledgement loss, visibility-timeout expiration, publisher retries, message replay, and concurrent processing.
What Idempotency Guarantees
Idempotency concerns the effect of an operation, not the number of times a handler executes. A consumer may receive a message five times but apply its business update only once.
| Operation | Repeated Execution |
|---|---|
| Set order status to SHIPPED | Often naturally idempotent |
| Increment shipped count by one | Not idempotent |
| Insert reservation with unique order ID | Can be idempotent |
| Charge a payment method | Requires provider-supported idempotency or reconciliation |
The intended guarantee is one logical business effect per idempotency identity. It does not mean a function executes exactly once or that a broker delivers only one message.
Message Deduplication Strategies
The appropriate strategy depends on whether duplicate detection belongs to the consumer infrastructure or can be enforced directly through the business data model.
For database-backed consumers, a durable uniqueness constraint is usually more reliable than checking a transient in-memory cache.
Processed Messages Table
A common implementation records successfully handled message IDs in a table with a composite key identifying the consumer and message.
CREATE TABLE processed_messages (
consumer_name TEXT NOT NULL,
message_id TEXT NOT NULL,
processed_at TIMESTAMPTZ NOT NULL DEFAULT now(),
PRIMARY KEY (consumer_name, message_id)
);
When processing starts, the consumer attempts to insert the marker. A successful insert authorizes processing; a uniqueness conflict indicates the message was already processed by that consumer.
The marker and business update must commit in the same transaction. Inserting the marker first and committing it separately can cause permanent message loss if the subsequent business operation fails.
Business-Level Idempotency
Some operations already have a natural business identity. An inventory reservation for an order and SKU should normally exist only once.
CREATE TABLE inventory_reservations (
order_id TEXT NOT NULL,
sku TEXT NOT NULL,
quantity INTEGER NOT NULL CHECK (quantity > 0),
PRIMARY KEY (order_id, sku)
);
The unique business key prevents duplicate reservation records even if the same operation is published under different message IDs.
This is a useful additional defense. Message deduplication protects individual deliveries, while business uniqueness protects the operation's meaning.
Inbox Pattern
The Inbox Pattern extends basic deduplication by persisting incoming messages for controlled processing, recovery, and observability.
An inbox record may include the message ID, payload, processing status, attempt count, timestamps, and error information.
Unlike a minimal processed-message table, an inbox can represent pending, processing, completed, and failed states. It is useful when processing is asynchronous or cannot finish safely within the message-receiving request.
However, an inbox introduces state management: stale claims, worker leases, retries, and recovery must be implemented correctly. A processed-message table is usually simpler when one local database transaction can handle the complete business update.
Transactions and Concurrency
Durable deduplication must handle concurrent consumers, process crashes, and database errors without introducing inconsistent state.
The central requirement is atomicity: either the message marker and business effects commit together, or neither does.
Atomic Deduplication and Business Updates
The correct processing order for a PostgreSQL-backed consumer is:
- Begin a transaction.
- Attempt to insert the processed-message marker.
- If the marker already exists, stop without changing business data.
- Otherwise, perform the required business updates.
- Commit the transaction.
- Acknowledge the broker message after successful commit.
When a business update raises an exception, the transaction rolls back, including the newly inserted marker. The broker can then redeliver the message without encountering a false "already processed" result.
Concurrent Consumers and Database Constraints
A preliminary query such as SELECT ... WHERE message_id = ... is insufficient for concurrency safety. Two workers can both observe that the marker does not exist.
Instead, use an atomic INSERT protected by a unique constraint:
INSERT INTO processed_messages (
consumer_name, message_id
)
VALUES ('inventory', 'evt-1001')
ON CONFLICT DO NOTHING
RETURNING message_id;
In PostgreSQL, conflicting concurrent inserts are coordinated through the unique index. Once the winning transaction commits, another insert for the same key returns no row.
The consumer must check the RETURNING result. ON CONFLICT DO NOTHING alone does not prevent the application from accidentally continuing into duplicate business updates.
Acknowledgements, Retries, and Failure Recovery
The acknowledgement boundary determines what happens after a failure.
| Failure | Expected Recovery |
|---|---|
| Crash before database commit | Rollback; broker redelivers for retry |
| Crash after commit, before ACK | Redelivery becomes an idempotent no-op |
| Insufficient inventory | Business rejection; stop automatic retry or route to workflow compensation |
| Temporary database outage | No ACK; retry with backoff |
| Duplicate message | Acknowledge without repeating the effect |
A consumer must distinguish permanent business rejections from transient infrastructure errors. Retrying insufficient inventory indefinitely is not a sensible recovery policy.
Production Example: Inventory Reservations
Consider an inventory service receiving OrderCreated messages. Each message requests a reservation for one order and product.
The system needs to tolerate duplicate deliveries without reducing stock more than once.
Business Requirements
- Each event contains a stable event ID, order ID, SKU, and positive quantity.
- A reservation for one order and SKU must be unique.
- Available stock must never become negative.
- Processed markers and inventory changes share one transaction.
- Concurrent deliveries must not create duplicate reservations.
- Insufficient inventory is a business rejection, not a successful reservation.
The example uses PostgreSQL to enforce uniqueness and atomic conditional inventory updates. It uses a local HTTP endpoint to simulate message delivery; a real message broker would invoke the same consumer function.
Failure Scenarios
Suppose inventory initially contains ten keyboards. Event evt-1001 requests two units for order order-1001.
The first delivery should reduce stock from ten to eight and create one reservation. Every later delivery of the same event should leave stock at eight.
If a different event ID attempts the same order and SKU reservation, the business uniqueness constraint should prevent another decrement.
If the requested quantity exceeds availability, the operation must roll back completely, leaving the event eligible for correction or an appropriate business-level failure workflow.
Ready-to-Use Python and PostgreSQL Implementation
This implementation uses Python 3.12+, FastAPI, psycopg 3, PostgreSQL, and pytest. It deliberately uses explicit SQL so the deduplication and transaction guarantees remain visible.
The application runs locally without SQS or Kafka. Message delivery is simulated through an API endpoint, but the consumer's core logic is transport-independent.
Project Structure and Dependencies
Create the following files:
idempotent-consumer/
consumer.py
api.py
schema.sql
docker-compose.yml
requirements.txt
tests/
test_consumer.py
Create requirements.txt:
fastapi>=0.115,<1
uvicorn>=0.30,<1
psycopg[binary]>=3.2,<4
pytest>=8,<10
httpx>=0.27,<1
For production, pin and test dependency versions rather than relying on broad version ranges.
Database Schema
Create schema.sql. PostgreSQL constraints protect both message identity and business reservation identity.
CREATE TABLE IF NOT EXISTS inventory (
sku TEXT PRIMARY KEY,
available INTEGER NOT NULL CHECK (available >= 0)
);
CREATE TABLE IF NOT EXISTS inventory_reservations (
order_id TEXT NOT NULL,
sku TEXT NOT NULL REFERENCES inventory(sku),
quantity INTEGER NOT NULL CHECK (quantity > 0),
PRIMARY KEY (order_id, sku)
);
CREATE TABLE IF NOT EXISTS processed_messages (
consumer_name TEXT NOT NULL,
message_id TEXT NOT NULL,
processed_at TIMESTAMPTZ NOT NULL DEFAULT now(),
PRIMARY KEY (consumer_name, message_id)
);
-- Local development inventory seed.
INSERT INTO inventory (sku, available)
VALUES ('keyboard', 10)
ON CONFLICT (sku) DO NOTHING;
For this example, one order reserves a given SKU at most once. In a system supporting partial or repeated reservations, the business key should represent the specific reservation operation instead.
Transactional Message Consumer
Implement the actual consumer in consumer.py. This function can later be called by an SQS handler, a RabbitMQ callback, or a Kafka worker.
import os
from dataclasses import dataclass
import psycopg
DATABASE_URL = os.getenv(
"DATABASE_URL",
"postgresql://app:app@localhost:5432/inventory",
)
class InsufficientInventory(Exception):
pass
class ReservationConflict(Exception):
pass
@dataclass(frozen=True)
class OrderCreated:
event_id: str
order_id: str
sku: str
quantity: int
def __post_init__(self) -> None:
if not all((
self.event_id,
self.order_id,
self.sku,
)):
raise ValueError("Identifiers are required")
if type(self.quantity) is not int or self.quantity <= 0:
raise ValueError("Quantity must be positive")
def process_order_created(
event: OrderCreated,
database_url: str = DATABASE_URL,
) -> str:
# Connection context commits on success
# and rolls back when an exception escapes.
with psycopg.connect(database_url) as conn:
with conn.cursor() as cur:
cur.execute(
"""
INSERT INTO processed_messages (
consumer_name, message_id
)
VALUES (%s, %s)
ON CONFLICT DO NOTHING
RETURNING message_id
""",
("inventory", event.event_id),
)
if cur.fetchone() is None:
return "duplicate"
# Lock any existing business reservation
# before attempting another stock change.
cur.execute(
"""
SELECT quantity
FROM inventory_reservations
WHERE order_id = %s AND sku = %s
""",
(event.order_id, event.sku),
)
existing = cur.fetchone()
if existing is not None:
if existing[0] != event.quantity:
raise ReservationConflict(
"Conflicting reservation payload"
)
# Different event ID, same business
# operation: no additional decrement.
return "already_reserved"
# Atomically reserve only available stock.
cur.execute(
"""
UPDATE inventory
SET available = available - %s
WHERE sku = %s AND available >= %s
RETURNING available
""",
(event.quantity, event.sku, event.quantity),
)
if cur.fetchone() is None:
raise InsufficientInventory(
f"Not enough stock for {event.sku}"
)
cur.execute(
"""
INSERT INTO inventory_reservations (
order_id, sku, quantity
)
VALUES (%s, %s, %s)
""",
(event.order_id, event.sku, event.quantity),
)
return "reserved"
The message marker, inventory decrement, and reservation insert belong to the same PostgreSQL transaction. If any statement fails, the connection context rolls back the transaction.
Two consumers attempting the same event ID cannot both insert the marker successfully. Consumers with different event IDs but the same business reservation are also protected by the reservation primary key.
If two such deliveries race, one may encounter a reservation uniqueness violation after updating inventory. PostgreSQL rolls back that entire transaction, including the decrement and marker. A real broker adapter should classify that conflict as a retryable race or reconcile the existing reservation before acknowledging; the implementation does not silently claim it succeeded.
For a high-contention system, consider a dedicated reservation operation identifier and a more explicit claim protocol to reduce race retries.
FastAPI Delivery Simulator
Create api.py to simulate a broker invoking the consumer with repeated messages.
from fastapi import FastAPI, HTTPException
from pydantic import BaseModel, Field
from psycopg import IntegrityError, OperationalError
from consumer import (
InsufficientInventory,
OrderCreated,
ReservationConflict,
process_order_created,
)
app = FastAPI(title="Idempotent Consumer Demo")
class EventRequest(BaseModel):
event_id: str = Field(min_length=1)
order_id: str = Field(min_length=1)
sku: str = Field(min_length=1)
quantity: int = Field(gt=0)
@app.post("/deliver")
def deliver(request: EventRequest):
event = OrderCreated(**request.model_dump())
try:
status = process_order_created(event)
return {"status": status}
except InsufficientInventory as exc:
raise HTTPException(
status_code=409,
detail=str(exc),
) from exc
except ReservationConflict as exc:
raise HTTPException(
status_code=409,
detail=str(exc),
) from exc
except IntegrityError as exc:
# Could be a race between distinct event
# IDs targeting the same business key.
raise HTTPException(
status_code=503,
detail="Concurrent reservation conflict; retry",
) from exc
except OperationalError as exc:
raise HTTPException(
status_code=503,
detail="Database temporarily unavailable",
) from exc
This endpoint is a development-only delivery simulator. Its HTTP statuses help demonstrate the different processing outcomes, but they are not broker acknowledgements.
In production, a broker adapter should acknowledge successful and duplicate outcomes, retry transient database failures, and handle permanent business rejections through an explicit failure workflow. Authentication and authorization are also required before exposing any inventory-changing API.
Integration Tests
The most useful tests exercise the actual database constraints rather than mocking PostgreSQL behavior.
Create tests/test_consumer.py:
from uuid import uuid4
import psycopg
import pytest
from consumer import (
DATABASE_URL,
InsufficientInventory,
OrderCreated,
process_order_created,
)
@pytest.fixture
def test_stock():
sku = f"test-{uuid4().hex}"
with psycopg.connect(DATABASE_URL) as conn:
conn.execute(
"""
INSERT INTO inventory (sku, available)
VALUES (%s, 10)
""",
(sku,),
)
yield sku
# Cleanup applies only to test-created records.
with psycopg.connect(DATABASE_URL) as conn:
conn.execute(
"""
DELETE FROM inventory_reservations
WHERE sku = %s
""",
(sku,),
)
conn.execute(
"DELETE FROM inventory WHERE sku = %s",
(sku,),
)
def available(sku: str) -> int:
with psycopg.connect(DATABASE_URL) as conn:
row = conn.execute(
"SELECT available FROM inventory WHERE sku = %s",
(sku,),
).fetchone()
return row[0]
def test_duplicate_event(test_stock):
event = OrderCreated(
event_id=str(uuid4()),
order_id=str(uuid4()),
sku=test_stock,
quantity=2,
)
assert process_order_created(event) == "reserved"
assert process_order_created(event) == "duplicate"
assert available(test_stock) == 8
def test_same_business_operation(test_stock):
order_id = str(uuid4())
first = OrderCreated(
str(uuid4()), order_id, test_stock, 2
)
second = OrderCreated(
str(uuid4()), order_id, test_stock, 2
)
assert process_order_created(first) == "reserved"
assert process_order_created(second) == "already_reserved"
assert available(test_stock) == 8
def test_failed_reservation_rolls_back(test_stock):
event = OrderCreated(
str(uuid4()),
str(uuid4()),
test_stock,
20,
)
with pytest.raises(InsufficientInventory):
process_order_created(event)
assert available(test_stock) == 10
# A failed transaction did not persist its marker.
with psycopg.connect(DATABASE_URL) as conn:
row = conn.execute(
"""
SELECT 1 FROM processed_messages
WHERE consumer_name = 'inventory'
AND message_id = %s
""",
(event.event_id,),
).fetchone()
assert row is None
These tests verify the essential guarantees: repeated delivery does not decrement inventory again, different event IDs representing the same reservation do not create a second business effect, and failed reservations do not leave a processed marker behind.
For a production system, also test concurrent consumers against the same message ID and business key, interrupted transactions, connection failures, and broker redelivery after commit.
Run and Verify the Example
Create docker-compose.yml to run PostgreSQL. The schema is initialized automatically only when the database volume is first created.
services:
postgres:
image: postgres:16-alpine
environment:
POSTGRES_DB: inventory
POSTGRES_USER: app
POSTGRES_PASSWORD: app
ports:
- "127.0.0.1:5432:5432"
volumes:
- ./schema.sql:/docker-entrypoint-initdb.d/schema.sql:ro
healthcheck:
test: ["CMD-SHELL", "pg_isready -U app -d inventory"]
interval: 5s
timeout: 3s
retries: 10
Start the database, install dependencies, and run the API:
# Start a local PostgreSQL container.
docker compose up -d --wait
# Install dependencies in an isolated environment.
python3 -m venv .venv
source .venv/bin/activate
pip install -r requirements.txt
# Configure the database.
export DATABASE_URL="postgresql://app:app@localhost:5432/inventory"
# Run PostgreSQL integration tests.
pytest -q
# Start the delivery simulator.
uvicorn api:app --reload
From a second terminal, deliver the same event twice:
# First delivery creates the reservation.
curl -X POST http://127.0.0.1:8000/deliver \
-H 'Content-Type: application/json' \
-d '{
"event_id": "evt-1001",
"order_id": "order-1001",
"sku": "keyboard",
"quantity": 2
}'
# Repeated delivery returns duplicate.
curl -X POST http://127.0.0.1:8000/deliver \
-H 'Content-Type: application/json' \
-d '{
"event_id": "evt-1001",
"order_id": "order-1001",
"sku": "keyboard",
"quantity": 2
}'
# Confirm inventory remains at eight.
docker compose exec postgres \
psql -U app -d inventory \
-c "SELECT * FROM inventory WHERE sku='keyboard';"
On a fresh database, the first response should contain "status":"reserved", while the second should contain "status":"duplicate". Inventory should decrease from ten to eight only once.
To repeat the demonstration from scratch, use a new event ID and order ID or reset the disposable local database. Do not reuse these development credentials in production.
Production Design Trade-offs
Idempotency is a business correctness property, but implementing it involves operational trade-offs in message identity, storage retention, ordering, and interactions with external services.
A processed-message table is a strong solution when the consumer's side effects occur in one transactional database. It is not a universal solution for every distributed workflow.
Ordering, Retention, and Event Identity
Message deduplication does not solve out-of-order delivery. A consumer can correctly ignore duplicates and still apply event version 7 before version 6.
For ordered aggregate updates, store and validate a stream version or sequence number. Depending on the business semantics, reject gaps, defer processing, or safely skip obsolete events.
Retention is another important design choice. If processed markers are deleted too early, a replayed old event can be mistaken for new work.
Finally, use stable logical event or operation identifiers. Broker offsets, receipt handles, and delivery-attempt IDs may change during redelivery or republishing and should not automatically be treated as permanent business identities.
Idempotent Consumer vs. Transactional Outbox
The Transactional Outbox and Idempotent Consumer Patterns solve opposite sides of reliable message processing.
| Pattern | Responsibility |
|---|---|
| Transactional Outbox | Durably record outgoing messages with local business changes |
| Idempotent Consumer | Prevent duplicate incoming deliveries from repeating business effects |
| Inbox | Persist incoming work for processing and recovery |
The outbox does not guarantee that a broker delivers a message only once. Publishers can crash after sending but before marking an outbox record as published.
That is why an outbox publisher commonly operates with at-least-once delivery while downstream consumers enforce idempotency.
This combination is particularly valuable in distributed workflows such as those described in What Is a Saga Pattern?.
External Side Effects and Exactly-Once Claims
The database transaction in the example makes the marker, stock change, and reservation atomic. It cannot also atomically commit an unrelated external API call.
Suppose the consumer charges a payment provider and then attempts to write its processed marker. If the database fails after the charge succeeds, redelivery could charge the customer again.
For external side effects, use provider-supported idempotency keys tied to the logical business operation and persist enough state to reconcile uncertain outcomes.
Do not assume Kafka's exactly-once processing features, SQS FIFO deduplication, or a successful broker acknowledgement automatically guarantee exactly-once effects in an external database or payment system.
Common Mistakes and Production Checklist
The biggest idempotency failures typically come from incorrect transaction boundaries or unstable message identities.
Review the consumer's behavior after each possible crash point, not just the normal successful path.
Common Mistakes
- Checking message existence before inserting, without an atomic uniqueness constraint.
- Committing the processed marker separately from the business update.
- Ignoring the result of
ON CONFLICT DO NOTHING. - Using delivery-attempt IDs instead of stable logical event IDs.
- Assuming message deduplication also guarantees event ordering.
- Retrying permanent business failures indefinitely.
Production Checklist
- Use stable event IDs and meaningful business operation keys.
- Enforce deduplication through database constraints.
- Commit markers and database effects atomically.
- Acknowledge messages only after durable processing or a safely classified terminal outcome.
- Test concurrency, crash recovery, and duplicate deliveries.
- Monitor retry rates, poison messages, and deduplication retention.
Frequently Asked Questions
The following questions address common assumptions about idempotency and delivery guarantees.
Does Idempotency Guarantee Exactly-Once Delivery?
No. A broker may deliver the same event repeatedly. Idempotency ensures repeated processing does not repeat the protected business effect. Delivery and business-effect guarantees are different concepts.
Should You Use the Broker Message ID?
Use a stable logical message or business operation ID. A broker-provided ID may work for redelivery of the same broker message, but republishing the same logical operation can produce a new broker ID. Business-level uniqueness provides additional protection.
Can Redis Handle Deduplication?
Yes, Redis can provide fast atomic marker operations such as SET with NX. However, a Redis marker and an unrelated PostgreSQL business update do not share one atomic transaction. Crashes between them can cause lost work or duplicates. Use Redis when its failure and consistency trade-offs are acceptable.
How Long Should Processed IDs Be Retained?
Retention must cover the maximum period during which a message or equivalent business operation can return, including delayed retries, dead-letter replay, and manual reprocessing. Some business operations require permanent uniqueness constraints even after message markers expire.
Does This Pattern Work with DynamoDB?
Yes. DynamoDB conditional writes and TransactWriteItems can atomically combine a deduplication marker with related item updates, subject to transaction limits. For simpler models, a conditional write on the business entity itself may be sufficient.
Conclusion
The Idempotent Consumer Pattern makes duplicate message delivery safe by separating delivery attempts from logical business effects. It is a fundamental building block for reliable event-driven applications and distributed workflows.
For consumers that update PostgreSQL, a processed-message table with a unique key and a shared database transaction provides a straightforward solution. Business-level uniqueness constraints add another layer of protection against duplicate logical operations.
Idempotency does not eliminate the need for event ordering, retry classification, dead-letter handling, external API idempotency, or operational monitoring. It addresses one critical guarantee: repeated deliveries should not repeat the same business effect.
Key Takeaway: Assume messages can arrive more than once. Use stable operation identities, enforce uniqueness atomically, and commit deduplication state with the business changes it protects.
More Articles to Read
- Must-Known Message Broker Patterns — Overview of deduplication, retries, dead-letter queues, and consumer patterns.
- What Is the Unit of Work Pattern? Transactions and Consistency Explained — Coordinate multiple persistence changes in one transaction.
- What Is Event Sourcing? Event Stores, Replay, and Practical Examples — Understand durable events, state reconstruction, and replay.
- What Is CQRS? — Separate write-side commands and read-side queries.
Comments (0)