Kafka Performance and Scaling

Kafka performance depends on how producers batch records, how partitions distribute work, how brokers use disk and network, and how fast consumer groups can process data. Scaling Kafka successfully requires identifying which layer is saturated instead of assuming that more brokers or more partitions automatically solve every throughput problem.

A production system should be designed for sustained throughput, acceptable p95 and p99 latency, broker failures, consumer recovery, uneven partition traffic, and future growth. The goal is not maximum benchmark throughput under ideal conditions, but predictable performance while the cluster is degraded, rebalancing, or draining backlog.

Table of Contents

• Where Kafka Performance Comes From
• Partitions Create Parallelism
  • Partition Count and Throughput
  • Hot Partitions
• Producer Performance
  • Batching and Linger
  • Compression
• Broker Performance
  • Disk Throughput
  • Network Throughput
  • Replication Cost
• Consumer Performance
• Scaling Consumer Groups
• Backlog and Recovery Capacity
• Partition Count Is Not Free
• Practical Capacity Planning Example
• Finding the Real Bottleneck
• Performance Testing Under Failure
• Production Mistakes to Avoid
• What to Monitor
• Conclusion
  • More Articles to Read

Where Kafka Performance Comes From

Kafka achieves high throughput because its design favors sequential log writes, batching, compression, partition-level parallelism, and efficient transfer of large record batches.

A producer does not need to send every event as an isolated synchronous request. Multiple records targeting the same partition can be combined into a batch, compressed, sent together, appended sequentially by the broker, replicated, and then fetched by consumers in batches.

This reduces per-record overhead across several layers:

• fewer network requests;
• fewer protocol headers per record;
• better compression ratios;
• more sequential disk access;
• more efficient consumer fetches.

This is why Kafka performance tuning should normally focus on batch efficiency and parallelism before low-level micro-optimizations.

Kafka's basic data model is covered in Apache Kafka Explained: How Kafka Works.

Partitions Create Parallelism

A Kafka topic is divided into partitions. Each partition has one active leader at a time and acts as one ordered log.

Partitions allow Kafka to distribute traffic across brokers and allow consumer groups to process several streams concurrently.

This makes partition count one of the most important scaling parameters in Kafka.

Partition Count and Throughput

Suppose one partition can sustain 15 MB/s of the target workload while maintaining acceptable latency. If the topic needs 120 MB/s, one partition is insufficient even if the cluster has many brokers.

import math

required_throughput_mb = 120
partition_capacity_mb = 15

required_partitions = math.ceil(
    required_throughput_mb / partition_capacity_mb
)

print(required_partitions)  # 8

Eight partitions provide the minimum theoretical parallelism for that throughput assumption.

The same principle applies to consumers. If the topic has eight partitions, a consumer group can have at most eight consumers actively owning partitions at one time.

Partition count must therefore support both producer-side distribution and consumer-side parallelism.

Hot Partitions

Having many partitions does not guarantee balanced throughput.

Suppose a logistics system partitions events by carrier_id. If one carrier handles 60% of all shipments, whichever partition receives that carrier's key may receive far more traffic than the others.

The cluster could look like:

Adding more consumers does not solve partition 2 because only one consumer in the group can own it at a time.

The better solution is often a higher-cardinality key such as shipment_id when business ordering allows it.

Partition count creates possible parallelism; partition-key distribution determines whether that parallelism is actually usable.

Partition design is covered in Kafka Topics, Partitions, and Offsets Explained.

Producer Performance

Producer throughput depends heavily on batching, compression, acknowledgement requirements, record size, and the number of active partitions.

A producer optimized only for minimum per-record latency may send many small requests and waste network and broker capacity. A producer optimized only for throughput may add unacceptable delay by waiting too long for large batches.

Batching and Linger

Kafka producers collect records for the same partition into batches. Larger batches reduce per-record overhead and often improve compression.

Two settings commonly influence this behavior:

• batch size controls how much data can accumulate into a batch;
• linger time allows the producer to wait briefly for more records before sending an incomplete batch.

For a high-volume event stream, waiting several milliseconds can significantly improve efficiency because hundreds of nearby records may be sent together.

For a low-volume payment stream, waiting for a large batch may offer almost no throughput benefit while directly increasing publication latency.

The correct tuning should be based on p95 and p99 event publication latency under realistic traffic, not only records per second.

Compression

Compression can substantially reduce producer network traffic, replication traffic, and retained storage.

Suppose a stream produces 200 MB/s of JSON records but compression reduces the payload to 70 MB/s. That reduction can free considerable network and disk capacity.

The trade-off is CPU.

A practical test should compare:

• bytes produced per second;
• compressed bytes per second;
• producer CPU utilization;
• broker CPU utilization;
• p95 and p99 produce latency;
• broker network saturation.

Compression can also improve as batches become fuller because repeated JSON keys and similar payload structures provide more redundancy.

Producer batching and reliability settings are covered in Kafka Producers Explained: Partitioning, Batching, and Delivery Guarantees.

Broker Performance

A Kafka broker sits between producers, followers, and consumers. Its capacity is constrained by several shared resources, especially storage and network bandwidth.

CPU can matter, particularly with encryption, compression-related work, many connections, and heavy request processing, but Kafka clusters are frequently limited by disk or network before pure CPU.

Disk Throughput

Kafka writes partition logs sequentially, which is efficient for modern storage. But disks are still finite.

Each broker may simultaneously handle:

• leader writes from producers;
• replica writes fetched from other leaders;
• consumer reads;
• replication reads for follower brokers;
• segment cleanup and retention work;
• replica recovery after failures.

A broker whose disk is comfortable during normal traffic may become saturated when a recovering replica starts reading hundreds of gigabytes from it.

Disk benchmarking should therefore include mixed read/write load and recovery behavior instead of measuring only sequential write throughput on an idle machine.

Network Throughput

Kafka can move several copies of the same logical data through the network.

Suppose producers write 100 MB/s to a topic with replication factor three.

The logical producer input is 100 MB/s, but brokers also need to transfer replica data to followers. Consumers then read additional copies of the stream.

If three independent consumer groups each process the full topic, the network impact is much larger than the original producer input alone.

A simplified capacity model might consider:

producer_input_mb = 100
replication_factor = 3
consumer_groups = 3

replica_copy_mb = producer_input_mb * (replication_factor - 1)
consumer_output_mb = producer_input_mb * consumer_groups

approx_cluster_data_transfer_mb = (
    producer_input_mb
    + replica_copy_mb
    + consumer_output_mb
)

print(approx_cluster_data_transfer_mb)  # 600 MB/s

This is intentionally simplified because actual broker placement and network paths vary, but it illustrates why Kafka network sizing cannot be based only on producer traffic.

Replication Cost

Replication improves fault tolerance but consumes storage, network, and recovery capacity.

A topic producing 10 TB of retained logical data with replication factor three requires roughly 30 TB of replica storage before filesystem overhead, headroom, temporary recovery needs, and other topics are considered.

Replication traffic also grows with write throughput.

Strong durability may additionally require producers to wait for multiple in-sync replicas, making slow followers visible as higher produce latency or failed writes when minimum ISR requirements cannot be maintained.

Replication and failure behavior are covered in Kafka Replication and Fault Tolerance Explained.

Consumer Performance

A consumer group is only as fast as the work performed after records are fetched.

Kafka may deliver records quickly, while the application spends most of its time on PostgreSQL writes, external API calls, JSON transformation, search indexing, or CPU-heavy processing.

Consider a consumer where one event requires:

• 1 ms of parsing;
• 4 ms of business logic;
• 12 ms of database work;
• 8 ms waiting for an external API.

The broker is not the main bottleneck. Optimizing Kafka fetch size while ignoring 20 ms of downstream I/O produces little improvement.

Useful consumer tuning often includes:

• processing records in batches where the business operation supports it;
• reducing database round trips;
• using bulk inserts or updates;
• controlling concurrency against downstream services;
• avoiding long blocking work inside the poll loop;
• keeping business processing idempotent.

For example, writing 500 analytics events with one database statement may be dramatically cheaper than executing 500 individual inserts.

INSERT INTO analytics_events (
    event_id,
    user_id,
    event_type,
    occurred_at
)
VALUES
    ('evt_1', 'usr_1', 'page_view', '2026-09-08T20:00:00Z'),
    ('evt_2', 'usr_2', 'page_view', '2026-09-08T20:00:01Z'),
    ('evt_3', 'usr_3', 'checkout', '2026-09-08T20:00:02Z');

Batching must still respect offset safety. Committing the entire batch before its durable processing completes can convert performance optimization into message loss during crashes.

Scaling Consumer Groups

The number of partitions creates the maximum number of active partition owners within one consumer group.

Suppose a topic has 12 partitions.

This leads to a practical scaling process:

1. measure sustainable throughput of one consumer instance;
2. calculate required consumer parallelism;
3. verify enough partitions exist;
4. verify the downstream database or API can handle the added concurrency;
5. leave headroom for recovery.

Suppose one consumer handles 10,000 events per second and input is 70,000 events per second.

import math

incoming_rate = 70_000
consumer_capacity = 10_000

minimum_consumers = math.ceil(
    incoming_rate / consumer_capacity
)

print(minimum_consumers)  # 7

At least seven active partition owners are needed in steady state. A topic with only four partitions cannot expose that concurrency without increasing partition count or changing the processing model.

Consumer-group behavior is covered in Kafka Consumers and Consumer Groups Explained.

Backlog and Recovery Capacity

A Kafka consumer group needs more than enough capacity to keep up with normal traffic. It also needs enough spare capacity to drain backlog after failures.

Suppose a service normally receives 100,000 events per second and can process exactly 100,000.

If it is unavailable for ten minutes, backlog becomes:

incoming_rate = 100_000
outage_seconds = 10 * 60

backlog = incoming_rate * outage_seconds

print(backlog)  # 60,000,000 events

After recovery, if processing capacity is still exactly 100,000 events per second, the service can process only the new traffic. The 60 million-record backlog never shrinks.

If recovery capacity is 140,000 events per second, only 40,000 events per second are available for draining backlog while 100,000 new events continue arriving.

backlog = 60_000_000
incoming_rate = 100_000
processing_capacity = 140_000

recovery_rate = processing_capacity - incoming_rate
recovery_seconds = backlog / recovery_rate

print(recovery_seconds)  # 1500 seconds
print(recovery_seconds / 60)  # 25 minutes

This means a ten-minute outage creates about 25 minutes of catch-up time after service restoration.

Recovery throughput should be part of normal capacity planning, not treated as emergency capacity.

Partition Count Is Not Free

Adding partitions can increase parallelism, but every partition creates operational overhead.

More partitions mean more:

• partition replicas;
• leader assignments;
• log segments and files;
• metadata;
• replication streams;
• recovery work;
• consumer assignment state.

Consider 200 topics with 100 partitions each and replication factor three.

topics = 200
partitions_per_topic = 100
replication_factor = 3

partition_replicas = (
    topics
    * partitions_per_topic
    * replication_factor
)

print(partition_replicas)  # 60,000

That is a very different operational workload from 200 topics with 12 partitions each.

Choosing an unnecessarily high partition count also reduces flexibility because reducing partition count is not a simple in-place operation.

Increasing it later is possible, but keyed partition mapping may change and affect long-lived ordering assumptions.

The practical goal is enough partitions to support required throughput and concurrency with reasonable headroom, not the maximum partition count the cluster can technically sustain.

Practical Capacity Planning Example

Consider a logistics platform ingesting 150,000 shipment events per second. Average compressed event size is 1 KB.

The workload therefore produces roughly:

events_per_second = 150_000
event_size_kb = 1

throughput_mb = (
    events_per_second
    * event_size_kb
    / 1024
)

print(round(throughput_mb, 2))  # 146.48 MB/s

Suppose production tests show that one partition can safely sustain 18,000 events per second while maintaining target producer latency and consumer processing behavior.

import math

events_per_second = 150_000
safe_partition_rate = 18_000

minimum_partitions = math.ceil(
    events_per_second / safe_partition_rate
)

print(minimum_partitions)  # 9

Nine partitions are the theoretical minimum. Choosing 12 may provide practical headroom and align with expected consumer concurrency.

Now consider the consumer side. One shipment-processing consumer safely handles 11,000 events per second because each event requires a PostgreSQL transaction.

incoming_rate = 150_000
consumer_rate = 11_000

required_consumers = math.ceil(
    incoming_rate / consumer_rate
)

print(required_consumers)  # 14

Twelve partitions are now insufficient if the consumer must scale to 14 active instances.

This reveals an important design fact: partition count should be sized from both broker throughput and consumer processing requirements.

Suppose the topic is therefore created with 18 partitions. That provides room for 18 active consumers and some future growth.

The next check is PostgreSQL. If 18 consumers each execute 11,000 transactions per second, the theoretical downstream request rate becomes almost 200,000 transactions per second. If PostgreSQL safely sustains only 130,000, Kafka partitioning is no longer the bottleneck.

The design may need:

• batching multiple shipment updates into fewer transactions;
• partitioning the database workload;
• reducing write amplification;
• using asynchronous secondary projections;
• limiting consumer concurrency to protect the database.

Kafka scaling must therefore be performed end to end.

Finding the Real Bottleneck

Kafka systems often scale poorly because engineers optimize the wrong layer.

A useful diagnosis starts by correlating producer, broker, consumer, and downstream metrics.

Adding brokers helps when broker resources are saturated and partitions can be redistributed effectively.

Adding consumers helps when consumer processing is the bottleneck and additional partitions are available.

Adding partitions helps when more partition-level parallelism is required.

None of these helps when PostgreSQL is already saturated by the consumer workload.

Scaling should target the saturated resource, not the component with the most visible dashboard.

Performance Testing Under Failure

A Kafka cluster benchmarked only under healthy conditions produces an incomplete capacity number.

Production testing should include failure scenarios because those conditions change load distribution and resource consumption.

Useful tests include:

• restart one broker during peak load;
• force partition leadership movement;
• reduce network bandwidth temporarily;
• introduce consumer processing slowdown;
• restart half of a consumer group;
• create a controlled backlog and measure catch-up time;
• recover a broker with replicas significantly behind;
• generate intentionally skewed partition-key traffic.

Suppose a three-broker cluster handles 300 MB/s comfortably under normal conditions. If one broker fails, the other two may need to handle approximately 50% more leadership work depending on partition distribution.

If the surviving brokers were already running at 80% disk or network saturation, the cluster may become overloaded precisely when redundancy is reduced.

This means nominal maximum throughput should be lower than healthy-cluster benchmark throughput.

A strong capacity target asks: what throughput can the cluster sustain while one expected failure is already happening?

Production Mistakes to Avoid

Kafka performance problems are often architectural rather than configuration-level problems.

• Adding partitions without checking key distribution. More partitions do not help when most traffic belongs to one hot key. Measure records and bytes per partition first.
• Scaling consumers beyond partition count. Extra instances cannot own partitions and add no partition-level throughput. Increase partitions only when the ordering model allows it.
• Sizing only for steady-state traffic. A consumer group that barely keeps up cannot recover from backlog. Preserve catch-up capacity.
• Benchmarking only healthy brokers. Broker failure changes leader distribution and adds recovery work. Test degraded operation.
• Ignoring downstream systems. More consumers can overload databases and external APIs. Treat the full processing path as one capacity system.
• Flushing the producer after every record. This destroys batching and adds synchronous round trips. Keep producers long-lived and allow batching where latency permits.
• Using enormous producer buffers to hide backpressure. This delays failure visibility and increases memory consumption. Let backpressure propagate intentionally.
• Creating excessive partitions for future-proofing. Partitions have metadata, storage, replication, recovery, and operational cost. Add only justified headroom.
• Watching averages only. One hot partition or saturated broker can fail while cluster-wide averages remain healthy. Keep per-partition and per-broker visibility.

What to Monitor

Kafka performance monitoring should show where latency, throughput, and backlog are changing across the whole path.

• Produce throughput. Track records and bytes per second by topic and partition.
• Produce latency. Monitor p50, p95, and p99 rather than averages alone.
• Producer batch size. Small batches under high traffic may indicate inefficient batching.
• Producer retry rate. Rising retries can reveal broker or network degradation before hard failures.
• Broker network utilization. Watch both ingress and egress because replication and consumers multiply traffic.
• Broker disk throughput and latency. Storage saturation can affect writes, reads, and replica catch-up.
• Under-replicated partitions. Replication lag can increase latency and reduce fault tolerance.
• Consumer lag per partition. Aggregate lag can hide individual hot partitions.
• Oldest-event age. Translate backlog into business delay.
• Consumer processing throughput. Compare it directly with incoming event rate.
• Consumer processing latency. Track p95 and p99 handler time.
• Rebalance frequency. Frequent rebalances can create throughput interruptions.
• Downstream saturation. Monitor database pool utilization, query latency, rate limits, and external API latency.

Capacity alerts should ideally fire before throughput collapses. For example, sustained consumer utilization near maximum combined with growing traffic can be more actionable than waiting until lag reaches millions of records.

Business-specific freshness metrics are equally important. A recommendation pipeline being 15 minutes behind may be acceptable, while a fraud-detection pipeline being 15 minutes behind may be a severe incident.

Conclusion

Kafka scales through batching, partitioned parallelism, efficient sequential storage, replication, and independent consumer groups. Performance depends on how effectively these mechanisms are matched to the workload.

Partitions determine available parallelism, but poor keys can create hot partitions. Producers gain throughput from batching and compression, while brokers are commonly constrained by storage and network. Consumers frequently become limited by databases or external services rather than Kafka itself.

Production capacity must also include failure and recovery behavior. A cluster that handles normal traffic but collapses after one broker failure, or a consumer group that cannot drain backlog after an outage, is not truly sized for its workload.

The central scaling principle is: measure the bottleneck across the entire event path, then add parallelism or capacity where that bottleneck actually exists. More brokers, consumers, or partitions are useful only when they relieve the resource that limits end-to-end throughput.

More Articles to Read

• Apache Kafka Explained: How Kafka Works
• Kafka Topics, Partitions, and Offsets Explained
• Kafka Producers Explained: Partitioning, Batching, and Delivery Guarantees
• Kafka Consumers and Consumer Groups Explained
• Kafka Replication and Fault Tolerance Explained