Kafka Best Practices for Production Systems

Running Kafka in production is less about finding a perfect configuration and more about designing predictable behavior under load, failures, deployments, retries, and traffic growth. Partitioning, producer durability, consumer idempotency, schema evolution, retention, and observability all affect whether the system remains reliable when conditions stop being ideal.

This article focuses on practical Kafka engineering rules: how to choose partition keys, size capacity, protect business side effects, handle failures without retry storms, evolve events safely, and operate the system with enough headroom to survive failures and recover backlog.

Table of Contents

• Design Topics Around Business Boundaries
• Choose Partition Keys Deliberately
  • Preserve Only Required Ordering
  • Prevent Hot Partitions
• Configure Producers for Durability and Throughput
• Make Consumers Idempotent
• Commit Offsets After Successful Processing
• Design Retries as Part of the Architecture
• Protect Downstream Systems
• Treat Event Schemas as APIs
• Plan Retention, Storage, and Replay Together
• Size for Failure and Recovery
  • Leave Broker Headroom
  • Leave Consumer Recovery Capacity
• Make Deployments Boring
• Monitor the Event Path, Not Only Kafka
• Production Readiness Checklist
• Conclusion
  • More Articles to Read

Design Topics Around Business Boundaries

Topic design should start from the event contract and business ownership rather than from database tables or arbitrary infrastructure grouping.

A topic such as order-events has a clear domain boundary when it contains events such as order.created, order.confirmed, and order.cancelled. A topic called application-events containing payments, shipments, authentication events, notifications, and analytics creates unrelated retention, security, scaling, and ownership requirements inside one stream.

Separate topics are useful when workloads require materially different:

• retention periods;
• partition counts;
• access controls;
• throughput profiles;
• consumer ownership;
• compaction policies;
• availability or durability requirements.

Too many tiny topics create operational overhead, while giant catch-all topics couple unrelated systems. Topic boundaries should reflect operational and business differences that actually matter.

Choose Partition Keys Deliberately

The partition key determines which records share an ordered log and strongly influences load distribution. It is therefore both a correctness decision and a scalability decision.

A logistics system might partition shipment events by shipment_id:

producer.produce(
    topic="shipment-events",
    key=shipment_id,
    value=serialized_event,
)

All events for one shipment can reach the same partition while unrelated shipments are distributed across the topic.

Preserve Only Required Ordering

Global ordering is rarely necessary and severely limits parallelism.

If a payment workflow requires events for one payment to remain ordered, use payment_id. There is usually no reason for payment A to be ordered relative to payment B.

A useful rule is: use the smallest business key whose events must remain ordered relative to each other.

This preserves correctness without serializing unrelated work.

Prevent Hot Partitions

A technically valid key can still distribute traffic poorly.

Suppose a delivery platform partitions by carrier_id, but one carrier processes 55% of all shipments. One partition may receive most traffic while others remain underutilized.

Monitor records and bytes per partition instead of relying on cluster-wide averages. If ordering by carrier is unnecessary, shipment_id may provide a much better distribution.

Adding partitions cannot fully solve a single dominant key because all events for that key still need the same ordering boundary.

Configure Producers for Durability and Throughput

Producer configuration should reflect the acceptable durability and latency trade-off. For important business events, a common baseline is idempotent production with acknowledgements from the required in-sync replicas.

from confluent_kafka import Producer

producer = Producer({
    "bootstrap.servers": "kafka-1:9092,kafka-2:9092,kafka-3:9092",
    "acks": "all",
    "enable.idempotence": True,
    "compression.type": "lz4",
    "linger.ms": 5,
})

acks=all prioritizes durability over the lowest possible acknowledgement latency. Idempotent production protects against certain duplicates caused by producer retries.

Batching and compression can significantly improve throughput. Sending one tiny synchronous request per event wastes network and broker capacity.

However, producer buffers should not be used to hide sustained overload. When Kafka cannot accept traffic quickly enough, the application needs an intentional backpressure or failure policy rather than unbounded buffering.

Track producer p95 and p99 latency, retry rate, batch size, compression ratio, and send failures. A producer that is still technically succeeding while latency and retries increase may already be approaching a capacity problem.

Make Consumers Idempotent

A production Kafka consumer should normally assume that the same logical event can appear more than once.

Consider an inventory consumer:

1. Read inventory.reserve.
2. Create the reservation in PostgreSQL.
3. Crash before committing the Kafka offset.
4. Restart and receive the event again.

The correct fix is not hoping that redelivery never happens. The reservation operation should be safe to repeat.

A database constraint can express the business invariant directly:

CREATE UNIQUE INDEX inventory_reservation_order_product
ON inventory_reservations (order_id, product_id);

The write becomes idempotent:

INSERT INTO inventory_reservations (
    order_id,
    product_id,
    quantity
)
VALUES (
    'ord_92814',
    'prd_501',
    2
)
ON CONFLICT (order_id, product_id) DO NOTHING;

When no natural business constraint exists, consumers can store stable event IDs and reject events that have already been processed.

External operations need the same protection. Payment providers, shipping APIs, and other services should receive stable idempotency keys when supported.

Kafka delivery guarantees do not automatically make business side effects idempotent.

Commit Offsets After Successful Processing

Offset handling determines what happens when a consumer crashes between receiving an event and completing its work.

Committing before processing can lose work:

1. Consumer receives event X.
2. Offset is committed.
3. Consumer crashes.
4. Business processing never completes.
5. Kafka considers X consumed.

For workloads where losing an event is unacceptable, the consumer should generally advance the offset only after the required processing succeeds.

This creates the opposite possibility: processing may succeed and the consumer may crash before committing. Event X is then delivered again.

That is why offset strategy and idempotency belong together:

Read → Process idempotently → Commit offset

This commonly produces at-least-once processing with protection against duplicate business effects.

Design Retries as Part of the Architecture

Retries should be based on failure classification rather than wrapping every exception in an infinite loop.

A database connection reset may succeed immediately. An HTTP 429 may need delayed retry. Invalid JSON will not become valid after waiting ten seconds.

Use exponential backoff with jitter for transient failures to prevent synchronized retry storms.

import random


def retry_delay(attempt: int) -> float:
    maximum = min(2 ** attempt, 60)
    return maximum + random.uniform(0, 1)

Long delays should generally not block a consumer thread indefinitely. Retry topics can move delayed work away from the primary stream when ordering requirements permit it.

Dead letter topics need ownership, alerts, investigation, and replay procedures. A DLT that nobody checks merely converts visible processing failures into stored failures.

Protect Downstream Systems

Kafka can absorb large traffic spikes, but consumers can transfer that pressure directly into PostgreSQL, Elasticsearch, third-party APIs, or other services.

Suppose a topic accumulates 30 million events during a consumer outage. When the service returns, aggressively scaling from 10 consumers to 100 may drain Kafka quickly but overwhelm the database.

Consumer throughput should respect downstream capacity.

Useful techniques include:

• bounded consumer concurrency;
• database batch writes;
• connection-pool limits;
• rate limiting;
• adaptive backpressure;
• circuit breakers for unstable dependencies;
• controlled backlog recovery rates.

For example, if PostgreSQL safely handles 20,000 updates per second, Kafka consumers should not generate 80,000 individual updates per second simply because more partitions are available.

Batching can improve the relationship between Kafka throughput and downstream load:

INSERT INTO analytics_events (
    event_id,
    user_id,
    event_type,
    occurred_at
)
VALUES
    ('evt_1', 'usr_10', 'page_view', '2026-09-09T10:00:00Z'),
    ('evt_2', 'usr_11', 'checkout', '2026-09-09T10:00:01Z'),
    ('evt_3', 'usr_12', 'purchase', '2026-09-09T10:00:02Z')
ON CONFLICT (event_id) DO NOTHING;

Kafka consumer lag is often safer than destroying a critical downstream dependency. Controlled backlog is recoverable; an overloaded database can turn one incident into several.

Treat Event Schemas as APIs

Kafka events are long-lived integration contracts. Producers and consumers deploy independently, and retained records may be replayed after the producer code that created them no longer exists.

Prefer backward-compatible evolution whenever possible:

• add optional fields instead of immediately requiring them;
• avoid renaming fields in place;
• do not silently change field meaning;
• handle unknown enum values intentionally;
• preserve old schema support for the required replay window;
• validate compatibility in CI when using a schema registry.

For example, changing:

{
  "total_amount": "149.90"
}

from "subtotal before tax" to "final total including tax" breaks the event contract even though the JSON schema remains identical.

Structural compatibility tools cannot detect semantic changes. Contract reviews still matter.

Use explicit event versions when consumers genuinely require different decoding logic. When the underlying business fact changes, a new event type is often clearer than forcing unrelated semantics into version 2 or version 3.

Plan Retention, Storage, and Replay Together

Retention is not only a storage setting. It determines how far consumers can recover or replay from Kafka.

Suppose a topic produces 40 MB/s continuously.

throughput_mb_per_second = 40
retention_days = 7
seconds_per_day = 86_400

logical_storage_mb = (
    throughput_mb_per_second
    * seconds_per_day
    * retention_days
)

logical_storage_tb = logical_storage_mb / 1024 / 1024

print(round(logical_storage_tb, 2))  # About 23.07 TB

With replication factor three, the physical storage requirement is much larger before headroom, filesystem overhead, and temporary recovery needs are considered.

Longer retention provides more replay flexibility but increases storage cost and recovery scope.

Retention should be chosen from actual requirements such as:

• maximum expected consumer outage;
• replay requirements;
• audit requirements;
• data reconstruction strategy;
• storage budget.

Consumers should alert well before lag approaches the retention boundary. Discovering that a consumer is eight days behind on a seven-day topic means Kafka may already have deleted required data.

Size for Failure and Recovery

A cluster that can barely handle normal traffic is underprovisioned for production. Kafka must continue operating while brokers restart, replicas recover, partitions move, consumers deploy, and traffic spikes occur.

Leave Broker Headroom

Suppose three brokers collectively handle 300 MB/s while each normally receives roughly 100 MB/s of work.

If one broker disappears, leadership and traffic shift toward the survivors. A cluster already running close to disk or network limits may become overloaded exactly when redundancy is reduced.

Capacity testing should therefore include:

• one broker unavailable;
• replica catch-up;
• leader movement;
• peak producer traffic;
• simultaneous consumer reads.

Healthy-cluster benchmark throughput is not the same as safe production throughput.

Leave Consumer Recovery Capacity

Consumer groups need spare capacity to drain backlog.

If traffic arrives at 100,000 events per second and consumers can process exactly 100,000, any outage creates permanent lag.

With 140,000 events per second of processing capacity, 40,000 events per second remain available for recovery while normal traffic continues.

incoming_rate = 100_000
maximum_processing_rate = 140_000

recovery_rate = maximum_processing_rate - incoming_rate

print(recovery_rate)  # 40,000 events/second

Recovery capacity should be tested against realistic outage durations and downstream limitations.

Make Deployments Boring

Consumer deployments can trigger partition reassignment and temporarily reduce throughput. Large consumer groups with frequent restarts may spend meaningful time rebalancing instead of processing.

Production deployment practices should minimize unnecessary membership churn.

• Shut consumers down gracefully. Finish or safely abandon in-flight work before termination.
• Avoid restarting the entire consumer fleet simultaneously. Roll gradually when practical.
• Respect processing-time limits. Long handlers can make healthy consumers appear dead.
• Test rebalancing under load. Measure how much lag a normal deployment creates.
• Watch downstream load during recovery. Consumers may temporarily process faster after deployment to drain lag.

Producer and schema changes should also be deployable independently. Backward-compatible event evolution reduces the need for coordinated releases across multiple services.

Operationally, a normal deployment should look like a small temporary change in consumer ownership, not a Kafka incident.

Monitor the Event Path, Not Only Kafka

A Kafka dashboard can show healthy brokers while the business pipeline is several hours behind because consumers are blocked on a database.

Observability should cover the full event path:

Producer → Kafka → Consumer → Database/API → Business result

Important producer signals include:

• records and bytes per second;
• p50, p95, and p99 send latency;
• retry rate;
• batch size;
• send error rate.

Important broker signals include:

• network ingress and egress;
• disk utilization and latency;
• under-replicated partitions;
• offline partitions;
• request latency;
• partition distribution.

Important consumer signals include:

• lag per partition;
• oldest unprocessed event age;
• processing throughput;
• p95 and p99 processing latency;
• retry and dead letter rates;
• rebalance frequency;
• downstream dependency latency.

Event age is often more meaningful than raw lag. A backlog of one million events could represent five seconds for a high-throughput analytics pipeline or several hours for a low-volume payment workflow.

Business metrics should complete the picture. If payment.authorized events are being consumed successfully but orders are not transitioning to paid, infrastructure metrics alone will not reveal the correctness problem.

Production Readiness Checklist

• Define topic ownership. Every important topic should have a team responsible for its contract and production behavior.
• Choose partition keys from ordering requirements. Verify both business correctness and key distribution.
• Measure per-partition traffic. Detect hot keys before cluster averages hide them.
• Use appropriate replication. Size storage and network for replica traffic and recovery.
• Enable producer idempotence where duplicate producer retries matter. Combine it with suitable acknowledgement settings.
• Make consumer side effects idempotent. Use business uniqueness, event IDs, or external idempotency keys.
• Commit offsets according to processing semantics. Avoid acknowledging work that has not completed.
• Classify retryable errors. Do not retry permanent validation failures indefinitely.
• Own dead letter topics. Alert, investigate, repair, and replay instead of accumulating forgotten failures.
• Protect downstream dependencies. Bound concurrency and recovery throughput.
• Validate schema evolution. Test old producers, new producers, old consumers, new consumers, and historical replay where required.
• Plan retention from recovery requirements. Alert before lag approaches the retention window.
• Maintain capacity headroom. Test with broker failures and consumer backlog rather than only healthy steady state.
• Test deployments under traffic. Measure rebalancing, lag growth, and catch-up behavior.
• Monitor business freshness. Track how old unprocessed events are, not only how many exist.

Conclusion

Production Kafka systems are reliable when correctness and failure behavior are designed across the entire event path. Topic and partition design determine scalability and ordering; producer settings determine durability and batching behavior; consumers must make repeated processing safe; and retry policies must distinguish recoverable failures from permanent ones.

Capacity planning should include degraded brokers, replica recovery, consumer outages, and backlog catch-up. Schema compatibility should support independent deployments and historical replay. Observability should connect Kafka infrastructure metrics with downstream saturation and business event freshness.

The most useful production rule is: design Kafka for the failure path, not only the healthy path. A system that remains predictable during retries, rebalances, broker loss, replay, schema migration, and traffic spikes is much more valuable than one that only achieves impressive throughput in a steady-state benchmark.

More Articles to Read

• Apache Kafka Explained: How Kafka Works
• Designing Event-Driven Systems with Kafka
• Kafka Performance and Scaling
• Kafka Reliability: Retries, Dead Letter Topics, and Failure Handling
• Kafka Schema Evolution and Event Versioning