What Is Message Deduplication?

Message deduplication is the process of detecting repeated deliveries of the same logical message and preventing them from producing duplicate business effects. It is a core reliability technique in message queues, event-driven systems, and distributed applications that use at-least-once delivery.

Duplicate delivery is normal in many reliable messaging systems. A consumer may successfully process a message but crash before acknowledging it, causing the broker to deliver the same message again. Deduplication allows the consumer to recognize that the work has already been completed.

Table of Contents

• Why Duplicate Messages Happen
• How Message Deduplication Works
• Message ID and Deduplication Key
• Idempotency vs Deduplication
• Consumer-Side Deduplication
  • Deduplication Table
  • Atomic Business Update
• Broker-Side Deduplication
• Deduplication Window
• Choosing Deduplication Storage
• Deduplication and Retries
• Message Deduplication in Kafka
• Deduplication and Concurrent Consumers
• Designing a Deduplication Key
• Production Design Example
• Common Message Deduplication Mistakes
• Production Checklist
• Frequently Asked Questions
  • Are Duplicate Messages Normal?
  • Does Deduplication Provide Exactly-Once Processing?
  • Should Deduplication IDs Be Stored Forever?
  • Can Redis Be Used for Message Deduplication?
  • Are Messages with the Same Payload Duplicates?
• Conclusion
  • More Articles to Read

Why Duplicate Messages Happen

Distributed messaging systems cannot always know whether a consumer completed its business operation before a failure occurred.

Consider a payment event:

Message Broker
      │
      ↓
payment.completed
      │
      ↓
Consumer
      │
      ↓
Add loyalty points  ✓
      │
      ↓
Consumer crashes    ✗
      │
      ↓
Acknowledgement never reaches broker

The business operation succeeded, but the broker does not know that.

With at-least-once delivery, the safe action is to deliver the message again:

payment.completed
      │
      ↓
Consumer
      │
      ↓
Add loyalty points again?

Without protection, the customer can receive the same points twice.

Duplicates can also appear because of producer retries, network failures, acknowledgement loss, consumer restarts, broker recovery, offset handling, dead-letter replay, or application-level retry logic.

This is a normal consequence of reliable distributed processing rather than necessarily a broker defect. The delivery trade-offs are covered in Message Delivery Guarantees: At-Most-Once vs At-Least-Once vs Exactly-Once.

How Message Deduplication Works

The basic idea is to give each logical message a stable identifier and remember which identifiers have already been processed.

Message
event_id = evt-8172
      │
      ↓
Has evt-8172 been processed?
      │
   ┌──┴──┐
   │     │
  no    yes
   │     │
   ↓     ↓
process skip

On the first delivery:

evt-8172
   │
   ↓
not found
   │
   ↓
process
   │
   ↓
store evt-8172

On redelivery:

evt-8172
   │
   ↓
already stored
   │
   ↓
skip duplicate

The important requirement is that retries of the same logical event carry the same deduplication identifier. Generating a new identifier for every delivery attempt defeats the mechanism.

Message ID and Deduplication Key

A message should usually contain an identifier representing the logical event:

{
  "event_id": "evt-8172",
  "type": "payment.completed",
  "payment_id": "PAY-4812",
  "order_id": "ORD-9918",
  "amount": 79.95
}

If the producer retries publishing this event, event_id should remain evt-8172.

Original publish
event_id = evt-8172

Network failure

Retry publish
event_id = evt-8172

A new ID on every retry makes the two messages appear unrelated:

Attempt 1 → evt-8172
Attempt 2 → evt-9914

Consumer sees two unique events

The deduplication key does not always have to be a generated event ID. A natural business identifier can sometimes be used when it accurately represents the uniqueness requirement.

For example:

payment_id + operation

PAY-4812:capture

This could express the rule that the capture operation for a particular payment should happen only once.

Idempotency vs Deduplication

Deduplication and idempotency solve closely related problems, but they are not identical.

Deduplication detects that a message has already been handled and avoids processing it again.

Message ID seen before?
       │
      yes
       │
       ↓
Do not execute again

Idempotency means repeating an operation produces the same intended state as performing it once.

Set order status = PAID
Set order status = PAID
Set order status = PAID

Final state = PAID

Compare that with a non-idempotent operation:

balance = balance + $100

Run once  → +$100
Run twice → +$200

Deduplication can make a non-idempotent operation safer by preventing the second execution.

In production systems, both techniques are often used together. Idempotency and Deduplication in Distributed Systems covers the broader relationship between these patterns.

Consumer-Side Deduplication

Consumer-side deduplication is one of the most common approaches because the application understands which business effects must not be repeated.

A consumer checks a durable store before applying the operation:

def process_message(message):
    event_id = message["event_id"]

    if processed_events.exists(event_id):
        return

    apply_business_operation(message)
    processed_events.insert(event_id)

This looks correct but contains a dangerous failure window.

1. Check event ID       → not found
2. Business operation   → success
3. Application crashes
4. Store event ID       → never happens

After redelivery, the consumer cannot tell that step 2 already happened.

Deduplication Table

A relational database can store processed message IDs with a unique constraint:

CREATE TABLE processed_messages (
    consumer_name VARCHAR(100) NOT NULL,
    message_id VARCHAR(100) NOT NULL,
    processed_at TIMESTAMP NOT NULL,
    PRIMARY KEY (consumer_name, message_id)
);

The consumer name can be included because different logical consumers may independently need to process the same event.

For example:

evt-8172

fulfillment-service   → processed
analytics-service     → processed
notification-service  → processed

The fact that fulfillment processed an event should not cause the notification service to skip it.

Atomic Business Update

When the business state and deduplication record are stored in the same database, both changes can often be written in one transaction.

def process_payment(event):
    with database.transaction():
        inserted = processed_events.try_insert(
            consumer="loyalty-service",
            message_id=event["event_id"],
        )

        if not inserted:
            return

        loyalty.add_points(
            customer_id=event["customer_id"],
            amount=event["amount"],
        )

The transaction provides an important property:

Deduplication record
        +
Business update
        │
        ↓
same transaction
        │
   ┌────┴────┐
   │         │
 commit   rollback
   │         │
 both      neither

If the transaction commits, both the business operation and deduplication marker exist. If it rolls back, neither exists.

This removes the failure gap between independently writing the business result and recording that the message was processed.

Broker-Side Deduplication

Some messaging technologies provide broker-level mechanisms that can reduce duplicate publication or delivery under specific conditions.

Conceptually:

Producer
   │
   ├── msg-123
   └── msg-123 retry
          │
          ↓
       Broker
          │
          ↓
    detect duplicate
          │
          ↓
   store/deliver once

This is useful, but application design should not automatically assume that broker deduplication guarantees exactly-once business effects.

The broker may only deduplicate within a particular time window, producer session, transaction, or configuration. Duplicates can also be introduced elsewhere in the processing pipeline.

The business application is the component that ultimately knows whether an operation such as charging a payment, issuing a refund, creating a shipment, or sending a reward must be unique.

Deduplication Window

Deduplication records consume storage, so many systems do not keep every processed message ID forever.

A system might retain IDs for a defined period:

Processed message IDs

Day 1 ───────────────────────────── Day 7
  │                                  │
  └──── deduplication window ─────────┘

Older IDs → expire

The correct window depends on how long duplicate deliveries can realistically occur.

If retries and replays can happen for 24 hours, a five-minute deduplication window is insufficient.

The retention decision should consider:

• broker retry duration;
• message retention;
• dead-letter retention;
• manual replay procedures;
• producer retry behavior;
• business requirements;
• storage cost.

For financial or other high-value operations, a business-level uniqueness record may need to remain much longer than the messaging system's normal retry window.

Choosing Deduplication Storage

The deduplication store must support the required correctness, throughput, and retention period.

For example, Redis can implement a temporary deduplication window using an atomic set-if-not-exists operation:

SET dedup:evt-8172 1 NX EX 86400

If the key is created, processing can continue. If it already exists, the message appears to be a duplicate.

However, writing the Redis key and updating a separate database are two independent operations:

Redis dedup marker ✓
        │
        ↓
Database update    ✗

A failure between them can incorrectly mark unfinished work as completed.

The fastest deduplication store is therefore not automatically the safest choice. Transaction boundaries matter.

Deduplication and Retries

Retries and deduplication solve different parts of message reliability.

Retries answer:

Processing failed.
Should another attempt be made?

Deduplication answers:

This message arrived again.
Has its business effect already happened?

Consider a timeout:

Attempt 1
   │
   ↓
Database update succeeds
   │
   ↓
Consumer loses connection
   │
   ↓
Result appears failed
   │
   ↓
Retry
   │
   ↓
Deduplication detects completed operation

This is particularly important because distributed systems often cannot determine whether a timeout means an operation failed or whether the operation succeeded but its response was lost.

Messages that continue failing after retries may eventually move to a dead-letter path. Dead-Letter Queues, Retries, and Poison Messages covers this failure lifecycle.

Message Deduplication in Kafka

Kafka provides several mechanisms related to duplicate prevention, but they address different layers of the system.

An idempotent Kafka producer can protect against certain duplicate records caused by producer retries:

Producer
   │
   ├── publish record
   │
   X acknowledgement lost
   │
   └── retry
          │
          ↓
        Kafka
          │
          ↓
avoid duplicate append
under supported producer semantics

Consumer-side duplicate processing is a different problem.

Suppose a consumer performs a database update and crashes before its Kafka progress is committed:

Kafka record
    │
    ↓
Consumer
    │
    ↓
Database update       ✓
    │
    ↓
Consumer crashes      ✗
    │
    ↓
Offset not committed
    │
    ↓
Record delivered again

Producer idempotence does not prevent this duplicate consumer execution.

The consumer still needs an appropriate idempotency or deduplication strategy for external side effects.

Kafka Ordering Guarantees and Message Deduplication covers Kafka-specific deduplication and ordering concerns in more detail.

Deduplication and Concurrent Consumers

A simple check followed by an insert is unsafe when multiple workers can process duplicates concurrently.

Consumer A                  Consumer B

check evt-8172              check evt-8172
not found                   not found

process                     process

insert ID                   insert ID

Both consumers saw the message as new before either inserted the marker.

The deduplication decision must be atomic.

A database unique constraint can provide this:

INSERT INTO processed_messages (
    consumer_name,
    message_id,
    processed_at
)
VALUES (
    'fulfillment-service',
    'evt-8172',
    CURRENT_TIMESTAMP
)
ON CONFLICT DO NOTHING;

Only one concurrent attempt can create the unique record.

The application can then determine whether it won the right to process the message.

The same principle applies to other storage systems: use conditional writes, atomic set-if-absent operations, or another mechanism that prevents a check-then-act race.

Designing a Deduplication Key

A good deduplication key represents the logical operation that must happen once.

For generic events, a unique event ID is often appropriate:

evt-8172

For a business operation, a natural key may be stronger:

payment:PAY-4812:capture

or:

order:ORD-9918:refund:REF-82

The key should be:

• stable across retries;
• unique for different logical operations;
• available before the side effect is executed;
• scoped to the consumer or operation when necessary;
• stored long enough to cover the duplicate-delivery window.

A payload hash can sometimes help identify identical content, but identical payloads do not necessarily represent the same logical event.

Two legitimate purchases could have identical amounts and product data while still being two separate transactions.

Production Design Example

Consider a payment platform consuming payment.completed events.

Each event awards loyalty points:

{
  "event_id": "evt-91827",
  "type": "payment.completed",
  "payment_id": "PAY-1842",
  "customer_id": "CUS-291",
  "amount": 120.00
}

The messaging infrastructure uses at-least-once delivery.

The consumer stores both loyalty changes and processed event IDs in PostgreSQL:

CREATE TABLE processed_messages (
    consumer_name VARCHAR(100) NOT NULL,
    message_id VARCHAR(100) NOT NULL,
    processed_at TIMESTAMP NOT NULL,
    PRIMARY KEY (consumer_name, message_id)
);

Processing occurs inside one database transaction:

def handle_payment(event):
    with database.transaction():
        inserted = processed_events.try_insert(
            consumer="loyalty-service",
            message_id=event["event_id"],
        )

        if not inserted:
            return

        loyalty.add_points(
            customer_id=event["customer_id"],
            amount=event["amount"],
        )

The first delivery follows the normal path:

evt-91827
    │
    ↓
insert dedup record
    │
    ↓
add loyalty points
    │
    ↓
commit transaction
    │
    ↓
acknowledge message

Now assume the transaction commits, but the consumer crashes before acknowledgement:

Database transaction  ✓
Acknowledgement        ✗
Consumer crashes       ✗

The broker delivers evt-91827 again.

evt-91827 redelivered
        │
        ↓
insert dedup record
        │
        X unique constraint
        │
        ↓
already processed
        │
        ↓
skip loyalty update

The second delivery does not create duplicate points.

Now consider two consumers receiving duplicate copies nearly simultaneously:

Consumer A                 Consumer B
    │                           │
    └──── evt-91827   evt-91827 ┘
              │
              ↓
      unique database key
          ┌────┴────┐
          │         │
       succeeds  conflict
          │         │
       process     skip

The unique constraint makes the deduplication decision concurrency-safe.

Operational monitoring should include:

A sudden increase in detected duplicates may indicate consumer instability, acknowledgement failures, producer retry problems, or an aggressive replay operation.

Common Message Deduplication Mistakes

• Generating a new event ID on every retry. Retries become indistinguishable from new events.
• Using check-then-insert without atomicity. Concurrent consumers can both process the same message.
• Writing the deduplication marker before an unrelated business operation. A crash can mark unfinished work as complete.
• Writing the marker after the business operation without atomicity. A crash can repeat an already completed side effect.
• Assuming broker deduplication solves consumer-side duplicates. Duplicate business execution can occur after successful delivery.
• Using Kafka offsets as business event IDs. Offsets identify positions within partitions, not logical operations across the application.
• Using payload hashes without understanding semantics. Identical payloads can represent different legitimate events.
• Expiring deduplication records too early. Delayed retries or DLQ replays can bypass the protection.
• Keeping every ID forever without a retention strategy. Deduplication storage can grow continuously.
• Ignoring duplicate metrics. A large increase can indicate broader reliability problems.
• Assuming deduplication automatically provides exactly-once processing. External side effects and transaction boundaries still matter.

Production Checklist

• Give each logical event a stable identifier.
• Reuse the same identifier across retries.
• Define exactly what constitutes a duplicate.
• Scope deduplication to the appropriate consumer or business operation.
• Use atomic conditional writes or unique constraints.
• Avoid non-atomic check-then-insert logic.
• Coordinate the deduplication record with business state when possible.
• Make critical side effects idempotent where possible.
• Define a deduplication retention period.
• Make the retention period longer than the expected retry and replay window.
• Consider DLQ replay when choosing retention.
• Do not rely only on broker-level duplicate prevention.
• Test consumer crashes before acknowledgement.
• Test crashes before and after business commits.
• Test concurrent duplicate deliveries.
• Test delayed retries.
• Test DLQ replays.
• Monitor detected duplicate counts.
• Monitor deduplication storage latency and errors.
• Monitor storage growth.
• Document how duplicate business operations are prevented.

Frequently Asked Questions

Message deduplication is most useful when duplicate delivery is expected and repeating the corresponding business operation would be unsafe or expensive.

Are Duplicate Messages Normal?

Yes. Duplicate delivery is expected in many systems that prioritize reliable at-least-once delivery.

A duplicate does not necessarily indicate a broken broker. It can be the correct recovery behavior when the system cannot determine whether an earlier processing attempt completed successfully.

Does Deduplication Provide Exactly-Once Processing?

Not by itself. Deduplication can prevent many repeated business operations, but exactly-once behavior depends on the complete processing boundary, including message delivery, storage transactions, external APIs, retries, and acknowledgements.

It is often more practical to design for at-least-once delivery with idempotent or deduplicated processing.

Should Deduplication IDs Be Stored Forever?

Not necessarily. The retention period should cover the realistic duplicate-delivery and replay window.

Some business operations may require long-term uniqueness, while high-volume low-risk events may use a shorter TTL-based deduplication window.

Can Redis Be Used for Message Deduplication?

Yes. Atomic set-if-not-exists operations and TTL support make Redis useful for high-throughput deduplication.

However, if the business state lives in another database, failures between the Redis write and business update must be considered. A fast lookup does not solve cross-system atomicity.

Are Messages with the Same Payload Duplicates?

Not necessarily. Two independent business events can contain identical data.

Deduplication should normally use a stable event or business-operation identity rather than assuming identical content means identical intent.

Conclusion

Message deduplication prevents repeated deliveries of the same logical message from producing duplicate business effects. It is particularly important in at-least-once messaging systems, where failures around acknowledgements, retries, and consumer commits can legitimately cause the same event to arrive more than once.

Reliable deduplication requires a stable message identity, an atomic way to claim that identity, an appropriate retention window, and careful coordination with the business operation being protected.

Key takeaway: duplicate delivery should often be treated as an expected failure mode. Design consumers so receiving the same logical message twice does not mean performing the business operation twice.

More Articles to Read

• What Is an Idempotency Key?
• Message Queue Best Practices for Production Systems