Apache Kafka Explained: How Kafka Works

Apache Kafka is a distributed event-streaming platform designed for moving large volumes of records between systems while keeping those records durable, ordered within defined boundaries, and available for later replay.

The practical value of Kafka comes from a different model than a traditional task queue. Producers append records to persistent logs, consumers independently track how far they have read, and partitions allow the workload to scale across brokers and consumer instances. Understanding these mechanics is essential before choosing partition keys, delivery guarantees, replication settings, or consumer architecture.

Table of Contents

• What Kafka Actually Does
• Core Kafka Components
  • Topics and Records
  • Partitions
  • Brokers
  • Producers
  • Consumers and Consumer Groups
• How a Record Moves Through Kafka
  • Producing a Record
  • Storing the Record
  • Consuming the Record
• Partitioning Is the Foundation of Kafka Scaling
• Offsets and Replay
• Consumer Groups and Parallel Processing
• Replication and Broker Failures
• Kafka Is a Log, Not Just a Queue
• Practical Order Processing Example
• What Kafka Does Not Solve
• When Kafka Is a Good Fit
• When Kafka Is Not a Good Fit
• Production Signals That Matter
• Conclusion
  • More Articles to Read

What Kafka Actually Does

Kafka provides a durable intermediary between systems that produce data and systems that consume it. A producer does not normally send an event directly to a specific consumer. It writes the event to a Kafka topic, and consumers read that topic independently.

Consider an e-commerce platform after an order is created. Several systems may need the same event:

• Inventory reserves products.
• Payments starts payment processing.
• Analytics records the purchase.
• Fraud detection evaluates the transaction.
• Notifications prepares an order confirmation.

Making the Order Service call every downstream system directly creates synchronous dependencies. A slow analytics service should not prevent an order from being created.

Kafka changes the interaction to a durable event publication:

Order Service → Kafka → Inventory / Payments / Analytics / Notifications

The producer only needs Kafka to accept the event. Each downstream system can process that event at its own rate and recover independently after temporary failures.

This broader architectural pattern is covered in Event-Driven Architecture in Distributed Systems.

Core Kafka Components

Most Kafka architectures can be understood through five concepts: records, topics, partitions, brokers, and consumers. The important part is not memorizing the terminology, but understanding which responsibility and failure boundary belongs to each component.

Topics and Records

A record is the unit of data stored in Kafka. It normally contains a key, value, timestamp, and optional headers.

An order event may look like:

{
  "event_id": "evt_73912",
  "event_type": "order.created",
  "order_id": "ord_92814",
  "customer_id": "cus_441",
  "total": 149.90,
  "created_at": "2026-09-07T16:40:21Z"
}

A topic is a logical stream of related records. Examples might include:

• orders
• payments
• shipment-events
• user-activity

A topic is not one physical file or one queue. It is divided into partitions that may live across several Kafka brokers.

Partitions

A partition is an append-only ordered log. New records are added to the end, and each receives an increasing numeric offset.

For example, one partition might contain:

Offset 81 → Offset 82 → Offset 83 → Offset 84 → Offset 85

Kafka guarantees ordering inside a partition. It does not provide one global ordering across every partition in a topic.

This distinction has major architectural consequences. If all events for one order must be processed in order, they can use order_id as the record key so Kafka consistently maps that order to the same partition.

producer.produce(
    topic="orders",
    key=order_id,
    value=payload,
)

Events for different orders can then occupy different partitions and be processed concurrently.

Brokers

A Kafka server is commonly called a broker. A production cluster normally contains multiple brokers, with topic partitions distributed between them.

Suppose an orders topic contains six partitions:

Distributing partitions allows storage, network traffic, and request processing to be spread across machines rather than concentrated on a single server.

Modern Kafka clusters use KRaft for cluster metadata management. Controllers maintain metadata such as broker membership, topic definitions, partition assignments, and leadership information, while brokers handle the data-plane workload.

Producers

A producer publishes records to Kafka. In production, a producer does considerably more than opening a connection and sending individual messages.

The producer can:

• choose a partition using the record key;
• batch multiple records together;
• compress batches;
• retry transient failures;
• wait for different levels of broker acknowledgement;
• use idempotent publishing to reduce duplicate writes caused by retries.

Batching is one reason Kafka can achieve high throughput. Sending 1,000 records as efficient batches creates very different network and disk behavior from issuing 1,000 independent synchronous requests.

Consumers and Consumer Groups

A consumer reads records from topic partitions. Kafka does not normally remove a record simply because one consumer read it.

Instead, each consumer group maintains its own progress through the topic using offsets. That allows multiple applications to independently consume the same data.

For example, the same orders topic might have three consumer groups:

• inventory-service
• analytics-service
• notification-service

Each group receives the order events independently. Analytics reading offset 50,000 has no effect on Inventory reading offset 49,500.

How a Record Moves Through Kafka

The complete Kafka path becomes much easier to reason about when separated into producer, broker, and consumer behavior. Consider an order.created event keyed by order_id=ord_92814.

Producing a Record

The Order Service creates the event and sends it through a Kafka producer.

import json
from confluent_kafka import Producer

producer = Producer({
    "bootstrap.servers": "kafka-1:9092,kafka-2:9092,kafka-3:9092",
    "acks": "all",
    "enable.idempotence": True,
})

event = {
    "event_id": "evt_73912",
    "event_type": "order.created",
    "order_id": "ord_92814",
    "customer_id": "cus_441",
    "total": 149.90,
}

producer.produce(
    topic="orders",
    key=event["order_id"],
    value=json.dumps(event),
)

producer.flush()

The key matters. Kafka's partitioner uses it to select a partition consistently. If all events for ord_92814 use the same key, those events continue through the same ordering boundary.

The configuration also requests acknowledgement from the required replicas and enables idempotent producer behavior. These decisions trade some latency and coordination for stronger durability and retry behavior.

Storing the Record

The producer obtains Kafka metadata and determines which broker currently leads the selected partition. The record is sent to that leader.

The partition leader appends the record to its log. Replica brokers copy the partition so that the data can survive broker failure according to the configured replication and acknowledgement policy.

After the write satisfies the producer's acknowledgement requirements, Kafka confirms the operation.

The important production insight is that Kafka durability is configurable. A producer receiving success means only what its acknowledgement, replication, and broker durability configuration define. Stronger acknowledgement settings generally reduce the chance of acknowledged data loss at the cost of waiting for additional replication work.

Consuming the Record

A consumer assigned to that partition fetches batches of records from Kafka. After successfully processing them, it advances or commits its consumer position according to the application's chosen offset strategy.

import json
from confluent_kafka import Consumer

consumer = Consumer({
    "bootstrap.servers": "kafka-1:9092,kafka-2:9092,kafka-3:9092",
    "group.id": "inventory-service",
    "enable.auto.commit": False,
    "auto.offset.reset": "earliest",
})

consumer.subscribe(["orders"])

while True:
    message = consumer.poll(1.0)

    if message is None:
        continue

    if message.error():
        raise RuntimeError(message.error())

    event = json.loads(message.value())

    reserve_inventory(
        order_id=event["order_id"],
        event_id=event["event_id"],
    )

    consumer.commit(message=message, asynchronous=False)

Committing after the business operation means a crash between successful processing and the offset commit can cause the same event to be delivered again. This is why Kafka consumers commonly need idempotent business logic.

Committing before processing changes the failure mode: a crash can cause the event to be skipped permanently. Offset timing is therefore a reliability decision, not merely client configuration.

Partitioning Is the Foundation of Kafka Scaling

Kafka scales topics primarily through partitions. A single partition has one active leader at a time and represents one ordered stream. Adding partitions creates additional independent streams that can be distributed across brokers and processed concurrently.

Suppose an order workload receives 120,000 events per second. If one consumer instance sustainably processes 20,000 events per second, running multiple partitions allows several consumer instances to divide the workload.

More partitions are not free. They increase metadata, open files, replication work, leader management, recovery work, and operational complexity. Partition count should be driven by expected throughput, required consumer parallelism, ordering boundaries, and future growth rather than an arbitrary large number.

The partition key also determines workload distribution. A poor key can defeat horizontal scaling even when many partitions exist.

For example, using country as the key for a globally distributed logistics stream may create one extremely busy partition if most traffic originates in one market. A higher-cardinality key such as shipment_id may distribute traffic much more evenly while still preserving per-shipment ordering.

Offsets and Replay

Every record in a partition receives an offset. The offset identifies its position within that partition.

orders-2: 10491 → 10492 → 10493 → 10494

Offsets are what make Kafka consumers fundamentally different from destructive queue consumers. Reading record 10492 does not normally delete it. Kafka keeps the record according to topic retention policy, while the consumer stores its progress separately.

This allows a consumer to move its position backward and process historical data again.

Replay is useful for practical scenarios such as:

• rebuilding a search index;
• recalculating analytics after fixing a bug;
• creating a new consumer from historical events;
• recovering a downstream projection after data corruption;
• testing a new processing version against recorded production events.

Replay also creates risk. An event that originally sent an email, charged a card, or called an external carrier API may trigger that side effect again if replay behavior is not designed explicitly.

Replayability does not automatically mean every consumer is safe to replay. Consumers with external side effects usually need idempotency, deduplication, replay modes, or separation between state reconstruction and irreversible actions.

Consumer Groups and Parallel Processing

A consumer group represents one logical subscriber. Kafka assigns each partition in a topic to at most one active consumer within that group at a time.

Suppose a topic has six partitions and the Shipping Service runs three consumer instances. Kafka can assign two partitions to each instance.

If six consumers run, each can receive one partition. If ten consumers run while only six partitions exist, four consumers cannot increase partition-level parallelism because there are no additional partitions to assign.

This leads to an important capacity rule: partition count establishes an upper bound on active consumer parallelism within a consumer group.

When consumers join, leave, crash, or become unhealthy, partition ownership can change through a rebalance. During these transitions, processing may pause or move between instances. Long-running handlers, unstable consumers, frequent deployments, and poor consumer configuration can therefore produce noticeable lag even when broker capacity is healthy.

Consumer scaling should be based on processing throughput and lag rather than CPU utilization alone. A consumer can have moderate CPU usage while falling hours behind because it is blocked on database calls or an external dependency.

Replication and Broker Failures

Partitions can be replicated across multiple brokers. One replica acts as the leader for normal reads and writes, while followers copy the leader's data.

With a replication factor of three, one partition might have replicas on Brokers A, B, and C.

If Broker A currently leads the partition and fails, Kafka can elect an eligible replica as the new leader. Producers and consumers refresh metadata and continue against the new leader.

This is how Kafka separates logical topic availability from individual machine availability. The topic does not need to become unavailable merely because one server disappears.

Replication still has limits. If too many replicas are unavailable, the cluster may reject writes instead of weakening configured durability guarantees. That is usually preferable for critical events such as payments or inventory updates because silently accepting under-replicated data can turn infrastructure failure into acknowledged data loss.

Replication also does not replace backups or cross-region disaster recovery. Replicas protect primarily against failures inside the replication topology. Software bugs, operator mistakes, accidental deletion, corrupted application events, or entire-region failures require separate recovery strategies.

Kafka Is a Log, Not Just a Queue

A traditional work queue usually focuses on delivering a task to a worker and removing or acknowledging it after successful processing. Kafka's core abstraction is different: records remain in an ordered log for a configured retention period, and consumers track their own position independently.

This makes Kafka particularly useful when an event has value beyond one immediate task. An order.created record may drive fulfillment today, rebuild analytics tomorrow, and become an input to a new fraud model months later while still within the retained dataset.

Traditional message queues remain useful for many job-processing workloads. A practical overview of producers, consumers, and brokers is available in Message Queues Explained: Producers, Consumers, and Brokers.

Practical Order Processing Example

Consider a checkout system that creates an order in PostgreSQL and wants to publish order.created to Kafka.

A naive implementation performs two independent operations:

1. Commit the order to PostgreSQL.
2. Publish the event to Kafka.

This appears reasonable until the application crashes after step one. The order exists, but Kafka never receives the event. Inventory, analytics, and notifications never learn about it.

Reversing the sequence creates the opposite problem: Kafka may receive an event for an order whose database transaction later fails.

A practical production design can use a transactional outbox. The order and an outbox record are stored in one database transaction:

BEGIN;

INSERT INTO orders (
    id,
    customer_id,
    status,
    total_amount
)
VALUES (
    'ord_92814',
    'cus_441',
    'created',
    149.90
);

INSERT INTO outbox_events (
    event_id,
    aggregate_id,
    event_type,
    payload,
    created_at
)
VALUES (
    'evt_73912',
    'ord_92814',
    'order.created',
    '{"order_id":"ord_92814","customer_id":"cus_441","total":149.90}',
    CURRENT_TIMESTAMP
);

COMMIT;

A separate publisher reads pending outbox rows and sends them to Kafka. Once publication succeeds, the row can be marked as published or removed according to the implementation.

This changes the dangerous database-to-Kafka gap into durable local state. If the publisher crashes, another process can retry from the outbox.

Publication can still occur more than once. Consumers should therefore treat duplicate delivery as an expected distributed-systems condition instead of an exceptional bug.

The complete database-to-broker reliability pattern is covered in Transactional Outbox Pattern for Reliable Messaging.

Now consider the Inventory Service. It receives evt_73912, reserves inventory, but crashes before committing its Kafka offset. Kafka later delivers the event again.

A consumer that blindly executes the reservation can reserve inventory twice. One solution is to atomically record processed event identifiers together with the business change.

BEGIN;

INSERT INTO processed_events (
    consumer_name,
    event_id
)
VALUES (
    'inventory-service',
    'evt_73912'
)
ON CONFLICT DO NOTHING;

-- Continue only when the processed_events row was inserted.

UPDATE inventory
SET reserved = reserved + 2
WHERE product_id = 'prd_291';

COMMIT;

The exact implementation depends on the database and domain model, but the important principle is stable: Kafka acknowledgement and business-side-effect atomicity are separate concerns.

What Kafka Does Not Solve

Kafka provides durable streaming infrastructure, but several common distributed-systems problems remain application responsibilities.

• Business idempotency. A consumer may receive the same logical event more than once. Kafka cannot automatically determine whether charging the same payment twice is acceptable.
• Cross-system atomicity. Updating a database and publishing to Kafka are separate operations unless an explicit coordination pattern is used.
• Global ordering. Ordering exists inside a partition, not across an entire multi-partition topic.
• Correct partition keys. Kafka cannot decide which business entity requires ordering or whether a chosen key will create traffic hotspots.
• Schema compatibility. Producers and consumers still need disciplined event-contract evolution.
• Backpressure planning. Kafka can absorb backlog, but finite disk capacity and finite consumer recovery throughput remain.
• Correct retry behavior. Retrying an invalid event forever can consume capacity without making progress.

This is why adopting Kafka does not automatically create an event-driven architecture that is reliable. Kafka supplies useful primitives; application design determines whether those primitives produce safe business behavior.

When Kafka Is a Good Fit

Kafka is strongest when data should be durable, independently consumable, replayable, and processed at significant scale.

• Business event distribution. Orders, payments, shipments, account changes, or other domain events need several independent consumers.
• High-throughput ingestion. Application activity, telemetry, logs, clickstreams, or device events arrive continuously at high volume.
• Data pipelines. Operational systems need to feed analytics platforms, warehouses, search indexes, caches, or machine-learning pipelines.
• Event-driven microservices. Producers should remain temporally decoupled from downstream consumers.
• Replayable processing. Consumers may need to rebuild derived state from historical events.
• Traffic buffering. Producers can temporarily generate data faster than downstream services can process it.

Kafka becomes especially useful when several of these requirements exist together. Using it solely because asynchronous communication is required can add unnecessary operational complexity.

When Kafka Is Not a Good Fit

Kafka is not automatically the best messaging technology for every asynchronous task.

A simple application that needs to place a few background email jobs into a queue may benefit more from a conventional queue or managed task service. Kafka introduces topics, partition design, offsets, consumer groups, retention, replication, cluster capacity, and operational behavior that may provide little value for such a workload.

Kafka is also a poor replacement for synchronous request-response communication when the caller genuinely needs an immediate answer. A pricing API asking for the current price normally benefits from direct request-response communication rather than publishing an event and waiting for another event to return.

Another weak fit is a workload requiring simple per-task scheduling, priorities, arbitrary delayed execution, or rich task-state semantics. Those requirements often align more naturally with job queues than with an ordered event log.

A useful decision rule is: choose Kafka because persistent event streams, independent consumers, replay, partitioned scale, or high-throughput ingestion solve a real problem—not simply because the architecture contains microservices.

Production Signals That Matter

Kafka monitoring should answer whether producers can publish safely, brokers can keep up, and consumers can process data within the application's acceptable delay.

• Consumer lag. Measure how many records each consumer group remains behind the partition head.
• Oldest-event delay. Record count alone can be misleading; time behind the producer often maps better to business impact.
• Produce and fetch latency. Rising p95 or p99 latency can expose broker saturation, network problems, replication pressure, or storage issues.
• Under-replicated partitions. Detect replicas that are no longer keeping up with their leaders.
• Partition traffic distribution. Uneven bytes or records per partition can reveal poor keys and hot partitions.
• Broker disk usage. Retention, traffic growth, or slow consumers can consume storage faster than expected.
• Producer error and retry rate. A growing retry rate can reveal broker instability before application publishing fails completely.
• Consumer processing latency. Measure application handling time separately from Kafka fetch performance.
• Rebalances. Unexpectedly frequent group changes can create pauses and unstable throughput.

Consumer lag deserves special attention because it connects infrastructure behavior to business freshness. An analytics consumer being five minutes behind may be harmless, while an inventory reservation consumer being five minutes behind may cause orders to be accepted against stale availability.

Alert thresholds should therefore be defined per consumer group rather than using one cluster-wide lag threshold.

Conclusion

Kafka works by storing records in durable partitioned logs rather than delivering each message directly to one destination. Producers append records, brokers store and replicate partitions, and consumer groups independently track their positions through offsets.

The partition is Kafka's most important scaling and ordering boundary. Partition count controls parallelism, partition keys determine which records remain ordered together, and consumer groups determine how partition work is distributed across application instances.

Kafka's persistent-log model enables high-throughput ingestion, multiple independent consumers, traffic buffering, and historical replay. Those capabilities also introduce responsibilities around partition design, duplicate processing, offset management, schema evolution, consumer lag, replication, and safe replay.

The central production principle is simple: Kafka provides durable event transport and storage, but correctness still belongs to the application. Idempotency, business transaction boundaries, partition keys, failure handling, and recovery strategies must be designed explicitly.

More Articles to Read

• Kafka Topics, Partitions, and Offsets Explained
• Event-Driven Architecture in Distributed Systems
• Message Queues Explained: Producers, Consumers, and Brokers
• Transactional Outbox Pattern for Reliable Messaging