Kafka Delivery Semantics: At-Most-Once, At-Least-Once, and Exactly-Once

Kafka delivery semantics describe what can happen to a record when producers, brokers, consumers, networks, or downstream systems fail. The familiar terms at-most-once, at-least-once, and exactly-once are useful only when the processing boundary is defined precisely.

A Kafka record can be written exactly once while its database update happens twice, or a consumer can process a record once while an external payment request is repeated. Production correctness therefore depends on understanding where duplicates and losses can occur across the complete workflow.

Table of Contents

• Delivery Semantics Are About Failure
• Where Message Loss and Duplicates Come From
  • Producer Failures
  • Consumer Failures
• At-Most-Once Delivery
• At-Least-Once Delivery
• Exactly-Once Semantics
  • Idempotent Kafka Producers
  • Kafka Transactions
  • Read-Committed Consumers
• Exactly-Once Stops at System Boundaries
• Idempotent Consumers for Business Correctness
• Database and Kafka Dual Writes
• Practical Payment Processing Example
• Choosing the Right Delivery Model
• Common Production Mistakes
• Monitoring Delivery Correctness
• Conclusion
  • More Articles to Read

Delivery Semantics Are About Failure

During normal operation, all three delivery models can appear identical. A producer sends one record, a consumer receives it, the application processes it, and everything succeeds once.

The difference appears when a failure happens between operations that cannot be committed atomically.

Consider a consumer processing an order:

Kafka read → Database update → Offset commit

If the process crashes after the database update but before the offset commit, Kafka cannot automatically know that the business operation already succeeded. After recovery, the consumer can receive the same record again.

Reversing the sequence changes the failure:

Kafka read → Offset commit → Database update

Now a crash after the offset commit can cause the database update to never happen.

This is the fundamental trade-off behind delivery semantics: when two independent systems cannot commit one operation atomically, failure can create either duplicates or loss unless another coordination mechanism is introduced.

Consumer offsets and commit behavior are covered in Kafka Consumers and Consumer Groups Explained.

Where Message Loss and Duplicates Come From

Delivery semantics are often discussed only from the consumer perspective, but ambiguity exists on both sides of Kafka. Producers can be uncertain whether a write succeeded, and consumers can be uncertain whether processing completed before failure.

Producer Failures

Suppose a producer sends a record to a broker. The broker appends the record successfully, but the acknowledgement is lost because the network connection fails.

The producer sees a timeout. It cannot distinguish between these two cases:

• the broker never stored the record;
• the broker stored the record but the acknowledgement was lost.

If the producer retries, the first case needs the retry while the second can create a duplicate unless producer idempotence protects it.

This is why disabling retries is usually not a good reliability strategy. It converts ambiguous writes into failed publications rather than solving the underlying distributed-systems problem.

Producer acknowledgements, retries, and batching are covered in Kafka Producers Explained: Partitioning, Batching, and Delivery Guarantees.

Consumer Failures

Consumers face the same ambiguity around business processing and offset commits.

Suppose a consumer processes offset 8120:

1. Kafka delivers offset 8120.
2. The consumer inserts a database row.
3. The database commits.
4. The consumer attempts to commit its Kafka offset.
5. The process crashes before the commit succeeds.

Another consumer takes the partition and resumes from the previous committed offset. Record 8120 appears again.

Kafka behaved correctly. The duplicate exists because the database transaction and Kafka offset were two separate durable operations.

At-Most-Once Delivery

At-most-once means an operation is processed zero or one time. Duplicate processing is avoided, but failures can cause work to be lost.

At the consumer boundary, this can be approximated by advancing the Kafka position before executing the business operation.

message = consumer.poll(1.0)

if message is not None and not message.error():
    consumer.commit(
        message=message,
        asynchronous=False,
    )

    process_event(message.value())

If the process crashes after the commit but before process_event() completes, Kafka will not normally redeliver that record to the group from the old position.

The main advantage is simplicity when duplicate processing is more undesirable than occasional loss. The disadvantage is exactly that: accepted work can disappear during failure.

At-most-once behavior can be reasonable for data where individual records have little value, such as some high-frequency telemetry, approximate metrics, or non-critical sampling workloads.

It is usually inappropriate for:

• payments;
• inventory reservations;
• financial ledger updates;
• order creation;
• security events requiring complete audit history.

For critical business operations, losing an event is generally harder to repair than detecting and safely ignoring a duplicate.

At-Least-Once Delivery

At-least-once means the system attempts to ensure that an event is processed, accepting that failures may cause it to be processed more than once.

The common consumer sequence is:

Read → Process durably → Commit offset

A simplified implementation looks like:

message = consumer.poll(1.0)

if message is not None and not message.error():
    event = deserialize(message.value())

    process_event(event)

    consumer.commit(
        message=message,
        asynchronous=False,
    )

If processing fails, the offset is not intentionally advanced and the record can be retried. If processing succeeds but the process crashes before committing, the record can be processed again.

This creates a useful reliability property: the system favors duplicate attempts over silent loss.

At-least-once delivery is a strong practical default for many business systems when combined with idempotent processing.

For example, creating a shipment label twice is dangerous. But instead of choosing at-most-once and risking no label being created, the consumer can use a stable idempotency key such as shipment_id when calling the carrier integration.

The operation can then be safely retried while the carrier or application recognizes repeated requests for the same logical shipment.

Exactly-Once Semantics

Exactly-once is often misunderstood as a universal guarantee that an event causes every possible side effect exactly one time. Kafka cannot provide that guarantee across arbitrary external systems.

Kafka exactly-once semantics are strongest when processing stays within a Kafka-controlled transactional workflow, such as consuming records, transforming them, producing new Kafka records, and committing consumed offsets as part of the same Kafka transaction.

Several mechanisms participate in this model.

Idempotent Kafka Producers

An idempotent producer prevents supported producer retries from creating duplicate records in Kafka.

Conceptually, Kafka associates producer identity and sequence information with writes. If a producer retries the same batch after an ambiguous acknowledgement, the broker can recognize the repeated sequence rather than append another copy.

from confluent_kafka import Producer

producer = Producer({
    "bootstrap.servers": "kafka-1:9092,kafka-2:9092,kafka-3:9092",
    "acks": "all",
    "enable.idempotence": True,
})

This protects against protocol-level retry duplication. It does not recognize that two separate application requests happen to represent the same business operation.

If an API processes the same POST /orders request twice and intentionally publishes two new Kafka records, producer idempotence does not merge them. Business-level idempotency is still required.

Kafka Transactions

Kafka transactions allow a producer to atomically publish records to multiple Kafka partitions and coordinate consumed offsets with produced output.

Consider a stream processor:

orders topic → Validation → validated-orders topic

Without transactions, the processor can produce an output record and crash before committing its input offset. After restart, it reads the same input again and produces another output.

A Kafka transaction can bind the produced output and consumed-offset advancement into one atomic Kafka transaction.

Conceptually:

producer.begin_transaction()

for message in messages:
    output = transform(message)
    producer.produce(
        topic="validated-orders",
        key=message.key(),
        value=serialize(output),
    )

producer.send_offsets_to_transaction(
    offsets=current_offsets,
    group_metadata=consumer.consumer_group_metadata(),
)

producer.commit_transaction()

If the transaction commits, both the output records and offset changes become committed. If it aborts, consumers configured for committed transactional data do not treat those output records as committed results.

This is much stronger than independently producing output and committing offsets.

Read-Committed Consumers

Kafka can contain records belonging to transactions that later abort. Consumers participating in exactly-once processing need appropriate isolation so aborted transactional records are not exposed as committed application results.

A consumer can be configured to read committed transactional data:

consumer = Consumer({
    "bootstrap.servers": "kafka-1:9092,kafka-2:9092,kafka-3:9092",
    "group.id": "validated-order-consumer",
    "isolation.level": "read_committed",
})

This allows downstream processing to observe committed transactional output rather than records from aborted Kafka transactions.

The guarantee is powerful, but the boundary remains important: it coordinates Kafka state. It does not automatically include PostgreSQL, Redis, an HTTP API, an email provider, or a payment gateway in the same transaction.

Exactly-Once Stops at System Boundaries

Consider a Kafka consumer that charges a card through an external payment provider:

Kafka event → Payment API → Kafka offset commit

The payment API succeeds, but the consumer crashes before committing the Kafka offset.

Kafka redelivers the event.

No Kafka transaction can undo the already completed external card charge. Retrying the API without protection may charge the customer again.

The practical solution is usually business-level idempotency. The consumer sends a stable payment operation ID to a provider that supports idempotency:

payment_provider.charge(
    amount_cents=14990,
    payment_method_id="pm_8291",
    idempotency_key="payment_ord_92814",
)

If the same logical request is repeated, the provider can return the original result instead of creating another charge.

The same principle applies to:

• sending webhooks;
• creating shipping labels;
• submitting bank transfers;
• calling external fulfillment systems;
• provisioning cloud resources.

Exactly-once business effects usually come from idempotent operations and durable state, not from messaging guarantees alone.

Idempotent Consumers for Business Correctness

An idempotent consumer makes repeated delivery of the same logical event safe.

A common technique stores a stable event ID in the same database transaction as the business update.

BEGIN;

INSERT INTO processed_events (
    consumer_name,
    event_id,
    processed_at
)
VALUES (
    'inventory-service',
    'evt_73912',
    CURRENT_TIMESTAMP
)
ON CONFLICT (consumer_name, event_id) DO NOTHING;

-- Execute the reservation only when the event marker was inserted.

INSERT INTO inventory_reservations (
    order_id,
    product_id,
    quantity
)
VALUES (
    'ord_92814',
    'prd_501',
    2
);

COMMIT;

The implementation must verify that the processed_events insertion actually created a row before executing the reservation.

Another approach makes the domain operation itself idempotent:

INSERT INTO inventory_reservations (
    order_id,
    product_id,
    quantity
)
VALUES (
    'ord_92814',
    'prd_501',
    2
)
ON CONFLICT (order_id, product_id) DO NOTHING;

This can be stronger because the database constraint represents the business rule directly: one reservation for that order and product.

Idempotency keys should represent stable business operations. Kafka offsets are usually poor deduplication keys because replaying or republishing the same logical event can give it a different topic, partition, or offset.

Database and Kafka Dual Writes

Delivery guarantees also matter before a record reaches Kafka.

Suppose an Order Service performs:

1. Insert order into PostgreSQL.
2. Publish order.created to Kafka.

A crash between the two operations leaves a committed order without an event.

Reversing the order does not solve the problem:

1. Publish order.created to Kafka.
2. Insert order into PostgreSQL.

Now Kafka may contain an event for an order whose database transaction failed.

This is a dual-write problem. Producer delivery settings cannot make two independent storage systems atomic.

A transactional outbox stores the business state and pending event in the same database transaction:

BEGIN;

INSERT INTO orders (
    id,
    customer_id,
    total_amount
)
VALUES (
    'ord_92814',
    'cus_441',
    149.90
);

INSERT INTO outbox_events (
    event_id,
    aggregate_id,
    event_type,
    payload
)
VALUES (
    'evt_73912',
    'ord_92814',
    'order.created',
    '{"order_id":"ord_92814","total":149.90}'
);

COMMIT;

A separate publisher reliably forwards pending outbox records to Kafka. Publication may happen more than once during crashes, so stable event IDs and idempotent consumers remain valuable.

This pattern is covered in Transactional Outbox Pattern for Reliable Messaging.

Practical Payment Processing Example

Consider a Payment Service consuming payment.requested events. The payment provider supports idempotency keys, and the local database stores payment state.

An event contains:

{
  "event_id": "evt_pay_7812",
  "payment_id": "pay_9821",
  "order_id": "ord_92814",
  "amount_cents": 14990,
  "currency": "USD",
  "payment_method_id": "pm_8291"
}

The unsafe implementation is straightforward:

result = payment_provider.charge(
    amount_cents=event["amount_cents"],
    payment_method_id=event["payment_method_id"],
)

save_payment_result(result)

consumer.commit(message=message)

If the charge succeeds and the process crashes before local state or the Kafka offset is safely recorded, the event can be delivered again and create another charge.

A safer design gives the external operation a stable idempotency key:

result = payment_provider.charge(
    amount_cents=event["amount_cents"],
    payment_method_id=event["payment_method_id"],
    idempotency_key=event["payment_id"],
)

The local database can then store the result under the same stable payment identity:

INSERT INTO payments (
    payment_id,
    order_id,
    provider_transaction_id,
    status,
    amount_cents
)
VALUES (
    'pay_9821',
    'ord_92814',
    'txn_728812',
    'succeeded',
    14990
)
ON CONFLICT (payment_id)
DO UPDATE SET
    provider_transaction_id = EXCLUDED.provider_transaction_id,
    status = EXCLUDED.status;

Only after durable local processing should the consumer advance its Kafka progress.

If the consumer crashes and Kafka redelivers the event, the same payment_id is used with the provider. The provider returns the result of the existing logical operation rather than creating a second charge.

This design is still based on at-least-once delivery. What makes it safe is that duplicate delivery does not imply duplicate business effect.

The strongest practical architecture often combines at-least-once messaging with idempotent business operations.

Choosing the Right Delivery Model

The correct delivery model depends on the consequence of losing or repeating work.

For an analytics counter, occasional duplicate processing might be acceptable or correctable later. For an email notification, duplicates are undesirable but usually less severe than a duplicate payment. For a financial transfer, both loss and duplication require explicit controls.

The decision should therefore start from the business operation:

• Can the event be safely lost?
• Can the operation be safely repeated?
• Can a stable idempotency key be defined?
• Does processing stay entirely inside Kafka?
• Does an external database or API participate?
• Can historical events be replayed safely?

These answers are more useful than selecting "exactly-once" as a general architecture goal.

Common Production Mistakes

Delivery-semantics bugs are dangerous because they often remain invisible until a rare failure occurs.

• Calling a workflow exactly-once without defining the boundary. Kafka transactions may protect Kafka records while an external database or API still receives duplicate operations. Document which systems participate in the guarantee.
• Assuming idempotent producers make consumers idempotent. Producer idempotence addresses supported producer retry duplicates, not repeated downstream business effects.
• Committing offsets before critical processing. A crash can permanently skip work. Prefer durable processing before commits when loss is unacceptable.
• Using Kafka offsets as business idempotency keys. Offsets identify log positions, not logical operations. Use stable event or business-operation IDs.
• Retrying external side effects without idempotency. Network timeouts create ambiguous outcomes. Use provider-supported idempotency keys or durable application coordination.
• Assuming Kafka transactions solve database dual writes. Kafka cannot automatically include an arbitrary database transaction. Use an outbox or another explicit consistency strategy.
• Ignoring replay behavior. A consumer that is safe during ordinary retries may still resend years of notifications when offsets are reset. Define replay semantics separately.

Monitoring Delivery Correctness

Infrastructure metrics alone cannot prove that delivery semantics are working correctly. Kafka may be healthy while the application repeatedly applies the same business event.

Useful signals include:

• Duplicate event detections. Count deduplication conflicts by consumer and event type.
• Idempotency-key reuse. Unexpected growth may expose upstream retry storms or duplicate command generation.
• Producer retry rate. Increasing retries indicate more ambiguous publication attempts even when idempotence prevents Kafka duplicates.
• Consumer retry rate. Persistent retries can create duplicate side-effect attempts and reduce throughput.
• Offset commit failures. Successful processing combined with failed commits predicts redelivery.
• Aborted transaction rate. For transactional Kafka workflows, increases can expose producer or infrastructure instability.
• Dead letter volume. Growth indicates events that normal processing cannot complete.
• Outbox age. The oldest unpublished event reveals database-to-Kafka delivery delay.
• Business reconciliation differences. Compare expected payments, reservations, shipments, or other effects against source events.

Reconciliation is especially important for high-value workflows. A zero Kafka error rate does not prove that every order produced exactly one payment or that every payment produced exactly one ledger entry.

Delivery correctness should therefore be observable at both the messaging layer and the business layer.

Conclusion

At-most-once favors avoiding duplicates but can lose work. At-least-once favors avoiding loss but makes duplicate delivery an expected failure mode. Kafka exactly-once semantics provide stronger atomicity for transactional Kafka workflows, but they do not automatically extend to arbitrary databases and external APIs.

For many production business systems, the practical design is at-least-once delivery combined with stable event identities, idempotent consumers, database constraints, transactional outboxes, and idempotency support at external service boundaries.

The central principle is: delivery semantics must be defined around the business side effect, not merely around the Kafka record. A record appearing once in Kafka is useful, but correctness ultimately depends on whether the corresponding payment, reservation, shipment, database update, or other operation occurs with the intended semantics.

More Articles to Read

• Apache Kafka Explained: How Kafka Works
• Kafka Producers Explained: Partitioning, Batching, and Delivery Guarantees
• Kafka Consumers and Consumer Groups Explained
• Transactional Outbox Pattern for Reliable Messaging