Kafka Replication and Fault Tolerance Explained

Kafka fault tolerance is built around partition replication. Instead of storing a partition on only one broker, Kafka keeps multiple replicas across brokers so another replica can take over when the current leader becomes unavailable.

Replication alone does not guarantee durability. Production behavior depends on the replication factor, leader and follower state, the in-sync replica set, producer acknowledgements, minimum in-sync replica requirements, and leader-election policy. These settings determine whether Kafka favors continued writes or protection against acknowledged data loss during failures.

Table of Contents

• Why Kafka Needs Replication
• Leaders, Followers, and Replicas
  • Partition Leaders
  • Follower Replicas
  • In-Sync Replicas
• What Happens When a Broker Fails
• Choosing a Replication Factor
• Acks and Minimum In-Sync Replicas
  • acks=1
  • acks=all
  • min.insync.replicas
• Availability vs Durability During Failures
• Unclean Leader Election and Data Loss
• Replication Does Not Eliminate Failure Domains
• Recovery After Broker Failure
• Practical Three-Broker Production Design
• Capacity Planning for Failures
• Production Mistakes to Avoid
• Monitoring Kafka Replication
• Conclusion
  • More Articles to Read

Why Kafka Needs Replication

A partition stored on one broker has a simple failure mode: when that broker becomes unavailable, the partition becomes unavailable with it. If the broker's storage is permanently lost, the records disappear as well.

Kafka replication creates additional copies of each partition on other brokers. A topic with replication factor three has three replicas of each partition, normally placed on different brokers.

For example, a six-partition payment-events topic might be distributed across three brokers:

If Broker A disappears, partitions led by Broker A can elect eligible replicas on Brokers B or C. Applications continue using the same logical topic while Kafka moves partition leadership underneath them.

Topics and partition behavior are covered in Kafka Topics, Partitions, and Offsets Explained.

Leaders, Followers, and Replicas

Replication is organized independently for every partition. One replica is the current leader and the remaining replicas follow it.

This leader-based design gives Kafka a clear authority for the partition's current log while still maintaining redundant copies for failover.

Partition Leaders

The leader handles normal producer writes for its partition. Producers discover the current partition leader through Kafka metadata and send records to that broker.

Suppose orders-4 currently has:

• Broker B as leader;
• Broker A as follower;
• Broker C as follower.

A producer writing to orders-4 sends its batch to Broker B. Broker B appends the records to its local partition log.

Followers then fetch the new data from the leader.

Follower Replicas

Followers maintain copies of the leader's partition log. They continuously fetch new records and attempt to stay close enough to the leader to remain eligible for safe failover.

A follower being configured as a replica does not automatically mean it contains every record currently accepted by the leader.

For example, a follower may fall behind because of:

• slow disk I/O;
• network congestion;
• broker CPU saturation;
• long garbage-collection or runtime pauses;
• broker restart;
• large traffic spikes.

Kafka therefore needs to distinguish replicas that are sufficiently caught up from replicas that are too far behind.

In-Sync Replicas

The in-sync replica set, commonly called the ISR, represents replicas currently considered sufficiently synchronized with the leader according to Kafka's replication rules.

Suppose a partition has replication factor three:

Leader A → Follower B → Follower C

Under healthy conditions, all three replicas may belong to the ISR.

If Broker C becomes slow and can no longer keep up, it can leave the ISR. The partition still has three configured replicas, but only A and B are currently in sync.

This distinction is critical because producer durability with acks=all depends on the in-sync replica set and minimum ISR configuration, not simply on the nominal replication factor.

What Happens When a Broker Fails

Consider a partition with Broker A as leader and Brokers B and C as in-sync followers.

A producer has been sending records successfully. Broker A then loses power.

Kafka's controller detects that the leader is unavailable and selects an eligible replica, for example Broker B, as the new leader. Cluster metadata changes to reflect the new leadership.

Producers and consumers may briefly receive errors or experience increased latency while metadata is refreshed. They then communicate with Broker B.

The application does not need to know which physical server permanently owns the partition because leadership is movable.

After Broker A returns, it does not immediately become authoritative based on its old state. It must reconcile and catch up with the current leader before it can again function as an in-sync replica.

This failover mechanism is what allows Kafka to survive ordinary broker failures without requiring every producer and consumer application to implement its own storage failover.

Choosing a Replication Factor

The replication factor determines how many copies of each partition Kafka maintains.

A replication factor of one means the broker holding a partition is a single point of failure for that partition.

Replication factor three is a common production choice because it creates enough redundancy to maintain a useful balance between durability, availability, maintenance flexibility, and cost.

Higher replication factors can provide additional copies but increase storage, network replication traffic, recovery work, and cluster resource consumption.

The correct value depends on data criticality and failure domains. A disposable telemetry stream and a financial transaction stream may reasonably use different durability policies.

Replication factor should express the cost of losing data, not simply follow one cluster-wide habit.

Acks and Minimum In-Sync Replicas

Replication factor describes how many replicas should exist. Producer acknowledgement settings determine what replication state is required before a producer considers a write successful.

These settings must be designed together.

acks=1

With acks=1, the producer considers a request successful after the partition leader accepts it.

Consider this sequence:

1. Producer sends event X to leader A.
2. Leader A appends X.
3. A acknowledges success to the producer.
4. A fails before follower B replicates X.
5. B becomes leader without X.

The producer observed success, but the record can disappear from the active partition history.

This may be acceptable when throughput or latency matters more than the durability of every individual record. It is risky for events whose acknowledged loss would violate business correctness.

acks=all

With acks=all, the leader waits for the required in-sync replication condition before acknowledging success.

This provides stronger durability because records are not acknowledged based solely on the leader's local copy.

A typical critical-event producer may use:

from confluent_kafka import Producer

producer = Producer({
    "bootstrap.servers": "kafka-1:9092,kafka-2:9092,kafka-3:9092",
    "acks": "all",
    "enable.idempotence": True,
})

Producer reliability behavior is discussed in more detail in Kafka Producers Explained: Partitioning, Batching, and Delivery Guarantees.

min.insync.replicas

min.insync.replicas defines the minimum number of in-sync replicas required for a write using acks=all to succeed.

A common durability-oriented configuration is:

replication.factor = 3, min.insync.replicas = 2, producer acks = all

Under healthy conditions, three replicas exist. If one broker fails, two in-sync replicas can remain and writes can continue.

If another replica becomes unavailable and only one in-sync replica remains, writes requiring acks=all are rejected.

This rejection is intentional. Kafka is choosing not to acknowledge new data when the configured durability level cannot be satisfied.

A subtle but important point is that acks=all by itself is not enough to express a strong replication requirement. If the minimum ISR requirement is too low, the cluster may still acknowledge writes while only one in-sync copy is available.

Availability vs Durability During Failures

Kafka replication exposes a fundamental distributed-systems trade-off: when redundancy degrades, should the system continue accepting writes or stop until safer replication is restored?

Consider a three-replica partition with only one healthy replica remaining.

One policy can continue accepting writes. Availability remains high, but that remaining broker becomes the only current copy of new records. Its failure can lose acknowledged data.

Another policy rejects writes until another in-sync replica becomes available. Durability protection remains stronger, but producer requests fail temporarily.

For clickstream events, continuing with reduced durability may sometimes be acceptable. For financial ledger events, rejecting writes may be preferable to reporting success for data that no longer meets the intended durability requirement.

The important engineering decision is not whether availability or durability is universally better. It is which failure is safer for the specific business stream.

Unclean Leader Election and Data Loss

A particularly difficult failure occurs when no in-sync replica is available for a partition but an out-of-sync replica still exists.

That replica may be missing records previously present on the old leader.

Promoting it can restore partition availability, but the new leader's log may not contain all previously acknowledged history. This is known as an unclean leader election.

The trade-off is direct:

• wait for an in-sync replica and keep the partition unavailable;
• promote an out-of-sync replica and risk losing records.

For durable business topics, allowing stale replicas to become leaders can undermine the entire purpose of strong replication settings.

For workloads where availability is substantially more important than retaining every event, the trade-off may be different.

This setting should therefore be an explicit business reliability decision rather than a tuning switch changed during an incident simply to make a red dashboard green.

Replication Does Not Eliminate Failure Domains

Three replicas provide little protection if all three fail together.

Replica placement should consider infrastructure failure domains such as racks, availability zones, power domains, or physical hosts.

Suppose three Kafka brokers run on three virtual machines but all three virtual machines are placed on the same underlying failure domain. A single infrastructure outage can still remove every replica.

A stronger deployment spreads replicas so that one infrastructure failure does not remove all copies of a partition.

The same principle applies to cloud availability zones. A three-broker cluster spread across three zones provides a different failure profile from three brokers in one zone.

However, spreading replicas across zones increases cross-zone replication traffic and may increase latency and network cost. Replication architecture therefore affects both reliability and operating cost.

Replication also does not automatically protect against:

• accidental topic deletion;
• bad retention configuration;
• application publishing corrupted data;
• operator mistakes affecting the whole cluster;
• security compromise;
• region-wide failure.

High availability, backup, and disaster recovery solve related but different failure classes.

Recovery After Broker Failure

Failover is only the first part of handling a broker failure. The cluster must eventually restore the intended replica count and distribution.

Suppose Broker A fails while holding hundreds of partition replicas. Brokers B and C take over leadership where possible.

When Broker A returns, its replicas may be behind. They must fetch missing log data from current leaders before returning to healthy synchronized state.

This recovery consumes:

• disk read bandwidth on leaders;
• disk write bandwidth on recovering replicas;
• network bandwidth;
• CPU for request processing and compression-related work;
• broker resources also needed for normal producer and consumer traffic.

A cluster that handles normal traffic comfortably can become overloaded during recovery if it has no spare capacity.

This creates an important production requirement: Kafka should be sized for degraded and recovering operation, not only healthy steady state.

Recovery speed also matters. A broker that takes six hours to rebuild leaves the cluster in a reduced-redundancy state for six hours, increasing exposure to a second failure.

Practical Three-Broker Production Design

Consider a Kafka cluster processing order and payment events across three brokers.

For a critical payment-events topic, the intended policy is:

• three replicas per partition;
• at least two in-sync replicas required for acknowledged writes;
• producers use acks=all;
• producer idempotence is enabled;
• unclean leader election is avoided for the durability-sensitive topic;
• brokers are distributed across independent infrastructure failure domains.

A topic configuration can conceptually include:

kafka-topics.sh \
  --bootstrap-server kafka-1:9092 \
  --create \
  --topic payment-events \
  --partitions 12 \
  --replication-factor 3 \
  --config min.insync.replicas=2

The producer then uses strong acknowledgement behavior:

from confluent_kafka import Producer

producer = Producer({
    "bootstrap.servers": "kafka-1:9092,kafka-2:9092,kafka-3:9092",
    "acks": "all",
    "enable.idempotence": True,
    "compression.type": "lz4",
})

Under normal operation, all three replicas can remain synchronized.

If one broker fails, two replicas remain. Writes can continue while satisfying the minimum ISR requirement.

If a second required replica becomes unavailable before redundancy is restored, producers begin receiving errors rather than having critical events acknowledged with only one in-sync copy.

The application must be designed for this outcome. A producer receiving a temporary Kafka failure should not silently discard the payment event.

Depending on the architecture, the source operation may remain safely retryable, or an outbox can retain the unpublished event until Kafka becomes writable again.

This illustrates why broker configuration and application reliability cannot be designed independently. Strong Kafka durability is useful only when producers handle temporary unavailability safely.

Capacity Planning for Failures

A common capacity-planning mistake is sizing every broker close to its normal maximum load.

Suppose a three-broker cluster handles 240 MB/s of combined workload, approximately 80 MB/s per broker under balanced conditions.

If one broker fails and its leadership moves to the other two, the surviving brokers may need to absorb significantly more work.

normal_cluster_load = 240
healthy_brokers = 3
remaining_brokers = 2

normal_per_broker = normal_cluster_load / healthy_brokers
degraded_per_broker = normal_cluster_load / remaining_brokers

print(normal_per_broker)    # 80 MB/s
print(degraded_per_broker)  # 120 MB/s

If each broker can safely sustain only 100 MB/s, the cluster was healthy only while every broker was available. One failure pushes the survivors beyond safe capacity.

Recovery can add even more load because missing replicas need to copy data while normal producers and consumers continue operating.

Capacity planning should therefore consider:

• normal producer and consumer traffic;
• replication traffic;
• one-broker or one-zone failure;
• replica catch-up traffic;
• traffic spikes during degraded operation;
• disk space needed during retention and recovery;
• network limits between failure domains.

Peak benchmark throughput is not safe production throughput if the cluster cannot survive an expected broker failure at that rate.

Production Mistakes to Avoid

Replication failures often expose configuration combinations that looked reasonable when considered separately.

• Using replication factor one for important topics. One broker failure can make partitions unavailable and permanent storage loss can destroy their data. Replicate business-critical streams.
• Assuming replication factor three means every acknowledged record has three copies. Replica count and acknowledgement policy are separate. Design acks and minimum ISR together.
• Using strong producer acknowledgements with weak minimum ISR requirements. The cluster may acknowledge writes with less redundancy than expected. Define the minimum acceptable in-sync state explicitly.
• Placing replicas in the same failure domain. Multiple copies provide limited value when one infrastructure failure removes all of them. Spread replicas across meaningful failure boundaries.
• Allowing stale replicas to become leaders without accepting the data-loss trade-off. Unclean election can restore availability by sacrificing log history.
• Sizing brokers only for normal traffic. Surviving brokers must absorb failed-broker leadership and recovery traffic. Maintain degraded-mode capacity.
• Treating replication as backup. Replicas can reproduce deletion or bad application data. Design separate recovery mechanisms for cluster-wide logical failures.
• Ignoring application behavior when Kafka rejects writes. Strong durability policies intentionally create temporary write failures. Producers need safe retry or durable buffering strategies.

Monitoring Kafka Replication

Replication monitoring should detect degraded redundancy before another failure turns it into an outage or data-loss event.

• Under-replicated partitions. Detect partitions whose replicas are not fully synchronized.
• ISR shrink and expansion rate. Frequent changes can indicate unstable brokers, disk pressure, or network problems.
• Offline partitions. Any partition without an available leader requires immediate investigation.
• Leader-election rate. Unexpected elections can reveal broker instability.
• Unclean leader elections. Treat these as serious events when durability matters.
• Replication lag. Identify followers that cannot keep up with leaders.
• Broker disk utilization. High utilization can slow replication and eventually prevent healthy operation.
• Network utilization. Replication shares network capacity with producer and consumer traffic.
• Produce error rate. Increases may be the expected result of minimum ISR protection during degraded replication.
• Recovery duration. Measure how long the cluster remains below its intended redundancy after failure.

Cluster-wide averages can hide dangerous conditions. One broker with saturated disk may cause only the partitions it follows to leave the ISR while average disk utilization remains moderate.

Alerts should therefore identify affected brokers, topics, and partitions rather than report only a cluster-wide percentage.

For critical topics, a useful operational objective is not simply "Kafka is available." A stronger objective is all critical partitions have the intended number of healthy replicas and can accept writes at the required durability level.

Conclusion

Kafka fault tolerance comes from replicating partition logs across brokers and moving leadership when failures occur. Leaders handle normal partition traffic, followers replicate their logs, and the in-sync replica set identifies replicas eligible for safe participation in durable writes and failover.

Replication factor, acks=all, and min.insync.replicas must be designed together. A common durability-oriented design uses three replicas, requires at least two in-sync replicas, and rejects writes when that protection can no longer be maintained.

This introduces an intentional trade-off: stronger durability can create temporary write unavailability during severe failures. Applications must therefore be capable of retrying or durably retaining events instead of assuming Kafka is always writable.

The central production principle is: fault tolerance is not the ability to survive a broker failure in theory; it is the ability to preserve the required correctness and throughput while failures and recovery are actually happening.

More Articles to Read

• Apache Kafka Explained: How Kafka Works
• Kafka Topics, Partitions, and Offsets Explained
• Kafka Producers Explained: Partitioning, Batching, and Delivery Guarantees
• Kafka Delivery Semantics: At-Most-Once, At-Least-Once, and Exactly-Once