Kafka Producers Explained: Partitioning, Batching, and Delivery Guarantees

A Kafka producer does much more than send individual messages to a broker. It chooses partitions, accumulates records into batches, compresses data, retries failed requests, waits for acknowledgements, and can prevent many retry-generated duplicates through idempotent publishing.

These behaviors directly control throughput, latency, ordering, and durability. A production producer should therefore be configured around the application's failure requirements and traffic profile rather than treated as a thin network client.

Table of Contents

• What Happens When a Producer Sends a Record
• Partition Selection and Record Keys
  • Keyed Records
  • Unkeyed Records
  • Choosing a Production Partition Key
• Batching for Throughput
  • Batch Size and Linger Time
  • Compression
• Acknowledgements and Durability
• Retries and Duplicate Records
• Idempotent Producers
• Delivery Guarantees in Practice
• Producer Ordering During Failures
• Backpressure and Producer Buffering
• Practical Order Event Producer
• Production Tuning by Workload
• Producer Failures and Monitoring
• Conclusion
  • More Articles to Read

What Happens When a Producer Sends a Record

Application code may appear to publish one record with one function call, but the Kafka client performs several operations before that record becomes durable.

A typical producer:

1. serializes the key and value;
2. selects the target partition;
3. places the record into an in-memory batch for that partition;
4. optionally compresses the batch;
5. sends the batch to the broker leading the partition;
6. waits for the configured acknowledgement;
7. retries eligible failures when necessary;
8. reports success or failure to the application.

The important detail is that Kafka is fundamentally optimized around batches rather than isolated messages. Records can be grouped before network transmission and written sequentially by brokers, allowing Kafka to achieve much higher throughput than a design based on synchronous per-record network round trips.

The partition remains the ordering and scaling boundary throughout this process. A deeper explanation is available in Kafka Topics, Partitions, and Offsets Explained.

Partition Selection and Record Keys

Every record written to a multi-partition topic must be assigned to one partition. This decision determines which records can remain ordered together and how producer traffic is distributed across the cluster.

Partitioning is therefore an application architecture decision, not merely a load-balancing mechanism.

Keyed Records

When a record contains a key, Kafka producers normally use that key to choose a partition deterministically.

For an order event:

producer.produce(
    topic="order-events",
    key=order_id,
    value=event_payload,
)

Using order_id means events for the same order are normally mapped to the same partition while the topic's partitioning configuration remains compatible.

This is useful for sequences such as:

order.created → order.paid → order.packed → order.shipped

Kafka can preserve that sequence because the records share one partition.

A common mistake is choosing a key based on convenient data rather than the actual ordering requirement. For example, using event_type would group every order.created event together instead of grouping events belonging to one order.

Unkeyed Records

When ordering by business entity is unnecessary, records can be produced without keys. Modern producer partitioning strategies can keep batches efficient while distributing traffic among available partitions.

This works well for workloads such as independent telemetry samples where one event has no ordering relationship with another.

producer.produce(
    topic="application-metrics",
    value=metric_payload,
)

Removing the key solely to improve distribution is dangerous when ordering matters. It can place two related events on different partitions, where Kafka provides no relative ordering guarantee.

Choosing a Production Partition Key

A strong partition key usually has two properties: it matches the required ordering boundary and has enough cardinality to distribute traffic.

Low-cardinality keys deserve particular attention. Using country for a workload dominated by one country can concentrate most records into one partition. The cluster may have dozens of partitions while one partition leader receives most producer traffic.

A partition key defines both an ordering boundary and a potential bottleneck.

Batching for Throughput

Kafka producers collect records destined for the same partition into batches. Sending a larger batch usually requires fewer network requests and less per-record protocol overhead.

Suppose an application produces 20,000 small events per second. Sending each event independently would require an enormous number of network interactions. Combining them into batches lets the producer amortize request, compression, and broker processing overhead across many records.

This creates one of Kafka's central performance trade-offs: waiting briefly can improve throughput substantially, but waiting too long increases end-to-end latency.

Batch Size and Linger Time

Producer clients commonly expose controls conceptually similar to batch size and linger time.

Batch size limits how much data the producer attempts to accumulate for a partition batch. Linger time allows the producer to wait briefly for more records before sending a batch that is not yet full.

Consider two workloads.

A payment authorization stream with low traffic may care about minimizing event publication latency. Waiting tens of milliseconds just to build a larger batch may provide little benefit.

A clickstream pipeline producing hundreds of thousands of records per second can often tolerate a few additional milliseconds while benefiting significantly from fuller batches.

The correct values depend on actual event rate and latency objectives. Producer tuning should be benchmarked using realistic record sizes and traffic patterns rather than copied from another system.

Compression

Kafka producers can compress record batches before sending them to brokers. Compression reduces network bandwidth and broker storage at the cost of CPU used to compress and decompress data.

Compression becomes particularly valuable for JSON and other repetitive event formats.

Suppose 100 MB/s of uncompressed events can be reduced to 30 MB/s. That reduction affects more than producer bandwidth. Replication traffic and retained storage can also decrease significantly.

Compression effectiveness depends on batch quality. Larger batches often compress better because repeated field names and similar values provide more redundancy.

Production testing should measure:

• compressed bytes per second;
• compression ratio;
• producer CPU;
• broker CPU;
• produce latency;
• network utilization.

The best codec is therefore not simply the one with the highest compression ratio. The relevant question is whether the reduction in network and storage pressure justifies its CPU cost for the workload.

Acknowledgements and Durability

The producer's acknowledgement configuration determines when a write is considered successful from the producer's perspective.

The commonly discussed modes are acks=0, acks=1, and acks=all.

With acks=1, the leader can acknowledge a record before followers have replicated it. If the leader fails at the wrong moment and an older replica becomes leader, the acknowledged record may not survive.

With acks=all, success depends on the current in-sync replica requirements and the topic or broker's minimum in-sync replica configuration. This is commonly preferred for important business events.

For example, a payment system may prefer temporary write failure over acknowledging an event that has weak replication protection.

acks=all does not mean every configured replica must always acknowledge. Durability depends on the relationship between acknowledgement behavior, in-sync replicas, replication factor, and minimum in-sync replica requirements.

Replication mechanics and broker failure behavior are covered in Kafka Replication and Fault Tolerance Explained.

Retries and Duplicate Records

Distributed writes often fail ambiguously. The producer may send a record, the broker may store it, and the acknowledgement may disappear because of a network interruption.

From the producer's perspective, the result is unknown.

If the producer retries, two outcomes are possible:

• the original write never succeeded, so the retry correctly creates one record;
• the original write succeeded but its acknowledgement was lost, so the retry can create a duplicate.

Disabling retries avoids some duplicate scenarios but replaces them with more failed publications. That is rarely a good trade for durable business events.

The better approach is normally to retain retry behavior and make retry-generated duplicates safer through Kafka producer idempotence.

Idempotent Producers

An idempotent Kafka producer allows brokers to recognize duplicate writes generated by producer retries within the producer protocol's supported scope.

Conceptually, Kafka tracks producer identity and sequence information so that retrying the same batch does not append it again as a new record.

A Python producer configuration can look like:

from confluent_kafka import Producer

producer = Producer({
    "bootstrap.servers": "kafka-1:9092,kafka-2:9092,kafka-3:9092",
    "acks": "all",
    "enable.idempotence": True,
})

Idempotence is highly valuable because retries are normal in distributed systems. Broker leader changes, transient network failures, request timeouts, and overloaded infrastructure should not automatically create duplicate Kafka records.

Producer idempotence does not make the entire business workflow idempotent.

For example:

1. An HTTP request creates order ord_100.
2. The application publishes order.created.
3. The HTTP client times out.
4. The client retries the entire HTTP request.
5. The application creates another logical event and calls the producer again.

Kafka producer idempotence cannot know that two separate application calls represent the same business operation. The application still needs business-level idempotency keys or stable event identifiers.

This distinction is essential: Kafka idempotence protects the producer protocol from certain retry duplicates; it does not deduplicate arbitrary application events.

Delivery Guarantees in Practice

Terms such as at-most-once, at-least-once, and exactly-once are useful only when the exact boundary is specified.

A producer can make publication durable, but that does not guarantee exactly-once business processing in a database, payment gateway, email provider, or another external system.

Consider an Order Service publishing an event after a database commit:

PostgreSQL commit → Kafka publish → Inventory consumer → Inventory database

Even a perfectly idempotent Kafka producer does not make the PostgreSQL commit and Kafka publication atomic. The application can crash after committing the order but before publishing its event.

A transactional outbox is a common solution. The business state and event are committed to the same database transaction, then a publisher reliably forwards the event to Kafka.

BEGIN;

INSERT INTO orders (
    id,
    customer_id,
    total_amount
)
VALUES (
    'ord_92814',
    'cus_441',
    149.90
);

INSERT INTO outbox_events (
    event_id,
    aggregate_id,
    event_type,
    payload
)
VALUES (
    'evt_73912',
    'ord_92814',
    'order.created',
    '{"order_id":"ord_92814","total":149.90}'
);

COMMIT;

The outbox publisher can retry Kafka publication without losing the source event. Consumers still need to tolerate duplicate logical delivery because crashes can occur around publication-state updates.

The full pattern is explained in Transactional Outbox Pattern for Reliable Messaging.

Producer Ordering During Failures

Partitioning provides an ordering boundary, but producer retry behavior must also preserve that order correctly.

Suppose two records for the same order are sent:

order.paid → order.shipped

If the first request fails temporarily while a later request succeeds, careless retry and in-flight request settings can create unexpected ordering behavior.

Modern Kafka idempotent producer configurations coordinate sequence numbers and compatible in-flight request behavior to maintain ordering under supported retry conditions.

The practical rule is to avoid independently changing producer settings related to retries, idempotence, acknowledgement, and in-flight requests without understanding their interaction.

Ordering also ends at the partition boundary. If order.paid and order.shipped use different keys and reach different partitions, producer configuration cannot restore global ordering afterward.

Backpressure and Producer Buffering

Kafka producers usually buffer records in memory before transmitting them. This improves throughput, but memory is finite.

If the application produces data faster than Kafka can accept it, the producer's buffer begins to fill.

Possible causes include:

• broker overload;
• network degradation;
• partition leader changes;
• insufficient broker storage performance;
• replication pressure;
• traffic spikes beyond planned capacity.

Once buffering limits are reached, producer calls may block, time out, or fail depending on the client and configuration.

This behavior should not be hidden with an enormous buffer. A larger buffer can absorb a short spike, but it also consumes memory and delays the moment when upstream systems learn that Kafka cannot keep up.

For an HTTP API, producer backpressure can eventually become increased request latency or 5xx responses. For a file-ingestion pipeline, slowing ingestion may be preferable to dropping data.

Backpressure should propagate according to business priority rather than becoming uncontrolled memory growth.

Practical Order Event Producer

Consider a service publishing order lifecycle events where losing acknowledged records is unacceptable, duplicate Kafka records from retries should be minimized, and per-order ordering is required.

A practical producer can be configured as follows:

import json
from typing import Any

from confluent_kafka import KafkaError, KafkaException, Producer


producer = Producer({
    "bootstrap.servers": "kafka-1:9092,kafka-2:9092,kafka-3:9092",
    "acks": "all",
    "enable.idempotence": True,
    "compression.type": "lz4",
    "linger.ms": 5,
})


def delivery_report(error: KafkaError | None, message: Any) -> None:
    if error is not None:
        raise KafkaException(error)


def publish_order_event(
    event_id: str,
    order_id: str,
    event_type: str,
    payload: dict[str, Any],
) -> None:
    event = {
        "event_id": event_id,
        "event_type": event_type,
        "order_id": order_id,
        "payload": payload,
    }

    producer.produce(
        topic="order-events",
        key=order_id.encode("utf-8"),
        value=json.dumps(event).encode("utf-8"),
        on_delivery=delivery_report,
    )

    producer.poll(0)

Several decisions are encoded here.

• order_id is the key. Events for the same order share a partition and ordering boundary.
• acks=all favors durability. Successful publication requires the configured in-sync replication conditions.
• Idempotence is enabled. Producer retries are protected against supported duplicate-write scenarios.
• Compression is enabled. Network and storage usage can be reduced for repetitive event payloads.
• A short linger is allowed. Nearby events can form better batches without intentionally adding large latency.
• Each event has an application ID. Downstream systems can implement business-level deduplication independently of Kafka offsets.

The example intentionally does not call flush() after every event. Flushing each record defeats much of producer batching and can turn asynchronous Kafka publishing into a sequence of expensive waits.

Applications should normally keep the producer alive, continuously poll or service delivery callbacks according to the client library, and flush during controlled shutdown when necessary.

Production Tuning by Workload

Producer configuration should follow the workload rather than one universal set of "best" values.

A high-throughput configuration should still be tested under broker failure. A producer benchmark against a healthy empty cluster measures only the easiest operating condition.

Useful load tests include a broker restart, partition leader election, network latency increase, temporary broker saturation, and traffic bursts above steady-state throughput.

The objective is not merely to find maximum records per second. Production capacity should maintain acceptable p99 publication latency, retry rates, memory usage, and durability behavior during expected failures.

Producer Failures and Monitoring

A producer can appear healthy while publication quality is degrading. Application request success alone is therefore insufficient for monitoring Kafka publishing.

• Produce request latency. Rising p95 and p99 latency can indicate broker, network, or replication pressure.
• Record error rate. Persistent publication failures should be separated from temporary retries.
• Retry rate. A rising retry rate often reveals degradation before requests fail permanently.
• Record queue time. Increasing time inside the producer buffer can indicate that Kafka cannot keep up.
• Batch size. Very small batches at high request volume may indicate inefficient batching configuration.
• Compression ratio. This helps quantify whether compression CPU is providing useful network and storage savings.
• Buffer utilization. Sustained high usage indicates producer-side backpressure.
• Records and bytes per partition. Large differences expose partition-key skew and hot partitions.

Application-level metrics should complement Kafka client metrics. For example, an outbox-based publisher should expose the number and age of unpublished rows.

If the oldest outbox record is 20 minutes old while Kafka producer metrics look healthy, the failure may be in the publisher itself rather than the brokers.

The most useful alert therefore often measures business publication delay: the time between creating an event and successfully making it available for downstream processing.

Conclusion

Kafka producer behavior is shaped by several connected decisions. Partition keys determine ordering and traffic distribution, batching and compression determine efficiency, acknowledgements determine when writes are considered successful, and retries determine how transient failures are handled.

Idempotent producers make retries much safer, but they do not provide business-level exactly-once behavior. Database transactions, duplicate HTTP requests, outbox publishing, and downstream side effects still require explicit application design.

Producer tuning should balance throughput, latency, durability, memory, and failure behavior rather than optimize one metric in isolation. A configuration that produces impressive throughput during a healthy benchmark can still behave poorly during leader changes or broker saturation.

The most important production principle is: configure the producer around the consequence of losing, duplicating, delaying, or reordering an event. Those business consequences should drive partition keys, acknowledgements, idempotence, batching, and retry behavior.

More Articles to Read

• Apache Kafka Explained: How Kafka Works
• Kafka Topics, Partitions, and Offsets Explained
• Kafka Consumers and Consumer Groups Explained
• Kafka Delivery Semantics: At-Most-Once, At-Least-Once, and Exactly-Once