Kafka Topics, Partitions, and Offsets Explained

Kafka topics, partitions, and offsets define how data is organized, scaled, ordered, consumed, and replayed. Most Kafka design mistakes eventually come back to one of these three concepts.

A topic is the logical stream, partitions are the physical units of parallelism and ordering, and offsets identify positions inside those partitions. Production design depends on understanding how they interact rather than treating them as independent configuration options.

Table of Contents

• Topics Are Logical Event Streams
• Partitions Are the Real Unit of Scaling
  • How Records Enter Partitions
  • Partition Keys and Ordering
  • Hot Partitions
• Offsets Identify Record Positions
  • Consumer Position and Committed Offsets
  • Offset Commit Timing
• How Partitions Control Consumer Parallelism
• Choosing the Right Number of Partitions
• Increasing Partitions Changes Key Distribution
• Offset Reset and Replay
• Retention and Offsets Are Different
• Practical Shipment Tracking Design
• Production Mistakes to Avoid
• Production Signals to Monitor
• Conclusion
  • More Articles to Read

Topics Are Logical Event Streams

A Kafka topic is a named stream of related records. Producers write records to topics, while consumers subscribe to topics and process their records.

Typical production topics might include:

• orders
• payment-events
• shipment-events
• inventory-updates
• user-activity

The topic name represents the logical stream, but Kafka does not normally store the entire topic as one ordered sequence. Each topic is divided into one or more partitions.

That distinction is important. Applications usually reason about a topic as one business stream, while Kafka processes and stores it as several independent logs.

For example, a shipment-events topic with four partitions is conceptually one stream but physically contains four ordering and processing lanes.

shipment-events → partition-0 / partition-1 / partition-2 / partition-3

The fundamentals of brokers, producers, consumers, and records are covered in Apache Kafka Explained: How Kafka Works.

Partitions Are the Real Unit of Scaling

A partition is an append-only ordered log. Kafka adds new records to the end of the partition, and every record receives an offset that identifies its position.

Partitions matter because Kafka distributes them across brokers and assigns them independently to consumers. More partitions can therefore increase storage distribution, producer parallelism, and consumer parallelism.

They also define Kafka's ordering boundary. Kafka guarantees record order within a partition, not across all partitions in a topic.

How Records Enter Partitions

A producer must eventually choose a partition for every record. The choice may depend on an explicit partition, a record key, or producer partitioning behavior.

For business events, using a meaningful key is common:

producer.produce(
    topic="shipment-events",
    key=shipment_id,
    value=event_payload,
)

The producer hashes or otherwise maps the key according to its partitioning strategy. Records with the same key normally continue to the same partition as long as the relevant topic partitioning remains stable.

If no key is provided, producers can distribute records across partitions for throughput. That may be useful for independent telemetry events where per-entity ordering is irrelevant.

Partition Keys and Ordering

The key should usually represent the entity whose events require ordering.

Suppose a shipment produces these events:

1. shipment.created
2. shipment.picked_up
3. shipment.in_transit
4. shipment.delivered

If all four records use shipment_id as the Kafka key, they are routed to the same partition and Kafka can preserve their relative order.

Using event_type instead would group all shipment.created events together and all shipment.delivered events together. That destroys per-shipment ordering and usually creates poor load distribution.

The best key is not necessarily the most obvious business field. It should balance ordering requirements, cardinality, traffic distribution, and future scalability.

Hot Partitions

A topic can have many partitions and still scale poorly if one partition receives a disproportionate amount of traffic.

Consider a logistics system using carrier_id as the key. If one carrier processes 70% of all shipments, most events may flow into the partition holding that carrier's key.

Other partitions remain lightly loaded while one broker handles much higher network, disk, and request traffic.

This is a hot partition. Adding consumers cannot solve it because one partition can be actively consumed by only one member of a consumer group at a time.

A better key might be shipment_id if ordering is required only within a shipment. If carrier-level ordering genuinely matters, the imbalance may be a necessary trade-off and capacity planning must account for it.

Offsets Identify Record Positions

An offset is a monotonically increasing position assigned to a record inside one partition.

For example:

partition-2: 40128 → 40129 → 40130 → 40131

The same numeric offset can exist in several partitions because offsets are local to a partition. Therefore, the complete position of a Kafka record is effectively identified by topic, partition, and offset.

For example:

{
  "topic": "shipment-events",
  "partition": 2,
  "offset": 40130
}

Offsets are not globally unique event identifiers. Application events should usually contain their own stable IDs such as event_id for deduplication, tracing, and business-level identity.

Consumer Position and Committed Offsets

A consumer's current position represents where it is reading. A committed offset represents progress that the consumer group has persisted so processing can resume after restart or reassignment.

Suppose a consumer has successfully processed records through offset 40130. Depending on client semantics, the committed position may indicate that the next record to consume is 40131.

If that consumer crashes, another instance in the same group can take ownership of the partition and resume from the committed position.

This separation is what allows Kafka consumption to recover independently of the consumer process that originally handled the records.

Offset Commit Timing

The moment an application commits an offset determines what happens during failure.

Consider a payment-events consumer:

1. Read event at offset 900.
2. Update the database.
3. Commit Kafka progress.

If the database update succeeds but the process crashes before step three, Kafka can deliver offset 900 again after restart. This creates at-least-once processing behavior.

If the application commits before updating the database:

1. Read event at offset 900.
2. Commit Kafka progress.
3. Update the database.

A crash after step two can permanently skip the business operation because Kafka considers the record consumed.

The practical default for important business events is often to process first and commit afterward, combined with idempotent consumer logic.

message = consumer.poll(1.0)

if message is not None and not message.error():
    event = json.loads(message.value())

    process_event_idempotently(
        event_id=event["event_id"],
        payload=event,
    )

    consumer.commit(message=message, asynchronous=False)

Offset management therefore belongs to application correctness, not only Kafka client configuration.

How Partitions Control Consumer Parallelism

Within one consumer group, a Kafka partition can be assigned to at most one active consumer at a time. This rule allows each partition to preserve ordered processing while still enabling parallelism across partitions.

Suppose shipment-events has eight partitions.

This produces a hard scaling relationship: the number of partitions limits useful consumer concurrency within a group.

If a consumer processes 5,000 events per second and the topic has four partitions, adding a fifth consumer does not increase partition-level processing capacity.

This makes partition count a capacity-planning decision that should consider both current and future consumer throughput.

Choosing the Right Number of Partitions

There is no universally correct partition count. Choosing one requires estimating producer throughput, consumer processing capacity, ordering requirements, broker capacity, and expected growth.

A simple starting approximation can be based on required consumer parallelism.

Suppose a service expects 100,000 events per second and one consumer instance safely processes 12,000 events per second at target p99 latency.

import math

required_throughput = 100_000
consumer_capacity = 12_000

minimum_consumers = math.ceil(
    required_throughput / consumer_capacity
)

print(minimum_consumers)  # 9

At least nine partitions would be needed for nine consumers to process partitions concurrently. In practice, additional headroom may be appropriate for growth and recovery.

The calculation is only one input. More partitions also create more replica logs, open file segments, metadata, leader assignments, network activity, and recovery work.

For example, 100 topics with 100 partitions each and replication factor three create 30,000 partition replicas. That is a very different operational workload from 100 topics with 12 partitions each.

Partition count should therefore be large enough for required scale but not inflated without a reason.

Increasing Partitions Changes Key Distribution

Increasing a topic's partition count appears simple operationally, but it can change how keyed records are distributed.

Suppose a producer maps a key according to a function conceptually similar to:

partition = hash(key) % partition_count

With four partitions, a shipment might map to partition 1. After increasing the topic to eight partitions, the same key may map to partition 5.

Existing events remain in the original partition while new events for the same business key may enter another partition.

This matters when an application assumes that all historical and future events for one key live in a single partition. Adding partitions can affect key-based ordering across the change boundary.

The exact behavior depends on the producer's partitioner, but production systems should not assume that increasing partition count is semantically invisible.

For strict long-lived ordering requirements, partition expansion should be planned before deployment rather than treated as a harmless emergency scaling mechanism.

Offset Reset and Replay

Because Kafka keeps records independently of consumer progress, a consumer group can intentionally move its offsets backward and reprocess historical data while those records remain available.

This enables practical operations such as rebuilding a database projection after fixing a bug.

Suppose an analytics consumer calculated shipment duration incorrectly for three days. After deploying corrected code, its offsets can be moved back to the beginning of the affected interval and the events can be processed again.

The same mechanism is useful when launching a new consumer. A newly created consumer group can start from older retained data rather than receiving only future records.

Replay becomes dangerous when processing causes irreversible external side effects. Replaying a notification topic may send old emails again. Replaying payment commands may attempt duplicate financial operations.

Consumers should therefore separate replay-safe state reconstruction from side-effecting behavior whenever possible.

A robust consumer may also use a stable event identifier to prevent duplicate side effects:

BEGIN;

INSERT INTO processed_events (
    consumer_name,
    event_id,
    processed_at
)
VALUES (
    'shipment-notifications',
    'evt_8a921',
    CURRENT_TIMESTAMP
)
ON CONFLICT (consumer_name, event_id) DO NOTHING;

-- Send or schedule the business action only if this event was newly accepted.

COMMIT;

The exact transaction boundary depends on whether the side effect itself can participate in the same durable transaction. External APIs usually require additional idempotency or coordination.

Retention and Offsets Are Different

Kafka retention decides how long records remain available. Consumer offsets decide how far each consumer group has progressed.

These mechanisms are independent.

A consumer being at offset 40,000 does not cause offsets 0 through 39,999 to disappear. Kafka removes or compacts records according to the topic's configured retention policy, not because a particular consumer processed them.

This separation allows many consumer groups to operate at different speeds.

It also means consumers can fall so far behind that the records they still need are removed by retention.

Suppose:

• topic retention is 24 hours;
• a consumer is offline for 36 hours;
• Kafka continues receiving traffic.

When the consumer returns, part of the required history may already be unavailable.

Retention must therefore be sized not only for storage cost but also for expected outage duration, replay requirements, downstream recovery time, and operational response time.

A system that needs seven days to rebuild a derived database should not retain only one day of source events unless another durable recovery source exists.

Practical Shipment Tracking Design

Consider a logistics platform consuming shipment status updates from multiple carriers. Events arrive through a Kafka topic called shipment-events.

Each event contains:

{
  "event_id": "evt_80bc4",
  "shipment_id": "shp_238911",
  "carrier_id": "ups",
  "status": "in_transit",
  "event_time": "2026-09-07T16:41:00Z"
}

The system requires lifecycle ordering for each shipment but does not require global ordering across all shipments.

A strong partitioning choice is therefore:

producer.produce(
    topic="shipment-events",
    key=event["shipment_id"],
    value=json.dumps(event),
)

This preserves per-shipment ordering while distributing different shipments across partitions.

Suppose traffic reaches 60,000 records per second. One tracking consumer instance can safely process 8,000 records per second because each event triggers validation and a PostgreSQL update.

A deployment with twelve partitions allows up to twelve active consumers in the same group, providing theoretical aggregate processing capacity above the current input rate while leaving recovery headroom.

A consumer might persist state with event-level deduplication:

BEGIN;

INSERT INTO processed_shipment_events (
    event_id,
    shipment_id,
    processed_at
)
VALUES (
    'evt_80bc4',
    'shp_238911',
    CURRENT_TIMESTAMP
)
ON CONFLICT (event_id) DO NOTHING;

UPDATE shipments
SET
    status = 'in_transit',
    last_event_at = '2026-09-07T16:41:00Z'
WHERE
    id = 'shp_238911'
    AND last_event_at < '2026-09-07T16:41:00Z';

COMMIT;

The timestamp condition protects the shipment projection from older carrier events that may arrive late even though Kafka preserved the order in which Kafka itself received records.

This distinction matters: partition ordering preserves Kafka record order, not real-world event-time correctness. Distributed producers, carrier APIs, network retries, and delayed integrations can still send business events out of chronological order.

The consumer commits its Kafka progress only after the database transaction succeeds. A crash may result in duplicate delivery, but the database logic is designed to tolerate it.

Production Mistakes to Avoid

Several design errors repeatedly cause Kafka systems to scale or recover badly.

• Using a low-cardinality partition key. Fields such as region, country, or event type can concentrate most traffic into a few partitions. Prefer a key that matches the required ordering boundary while distributing load.
• Assuming topic-wide ordering. Events across different partitions have no single global order. Put events that require relative ordering onto the same partition.
• Creating too few partitions. Consumer groups cannot scale beyond the available partitions. Estimate future processing concurrency before traffic reaches production limits.
• Creating excessive partitions. More partitions increase broker metadata, replication, storage files, recovery work, and operational overhead. Add them for capacity needs rather than as a default.
• Treating offsets as event IDs. Offsets identify positions within partitions and can repeat across partitions. Use explicit application-level event identifiers.
• Committing offsets before critical processing. A crash after the commit can lose the business operation. Prefer processing first when at-least-once behavior is acceptable.
• Assuming replay is harmless. Replayed records can repeat emails, payments, webhooks, or external API calls. Side-effecting consumers need explicit replay safety.
• Increasing partition count without reviewing key semantics. Key-to-partition mappings may change, affecting long-lived ordering assumptions.

Production Signals to Monitor

Monitoring topics and partitions should reveal both capacity problems and data-distribution problems.

• Consumer lag per partition. Aggregate lag can hide one badly delayed partition.
• Lag in seconds. Business impact is often easier to understand as event age than record count.
• Records per second per partition. Large differences can expose key skew and hot partitions.
• Bytes per second per partition. Equal record counts do not guarantee equal load when event sizes vary.
• Consumer processing time. Track p95 and p99 processing duration to understand sustainable throughput.
• Partition count growth. Unexpected growth can reveal poor topic lifecycle management or automation mistakes.
• Disk growth by topic. Retention and event size directly affect broker storage requirements.
• Rebalance frequency. Frequent reassignments can interrupt partition processing and increase lag.

Hot-partition alerts are especially valuable because cluster-wide CPU or disk averages may look healthy while one partition leader is saturated.

Operational dashboards should therefore retain partition-level visibility for high-volume or business-critical topics rather than relying only on broker-wide averages.

Conclusion

Topics organize Kafka records logically, partitions provide the physical boundaries for ordering and parallelism, and offsets identify positions inside those partitions.

The partition key is one of the most important Kafka architecture decisions. It determines which events remain ordered together and how evenly traffic is distributed across the cluster. A poor key can create hot partitions even when the topic contains plenty of theoretical capacity.

Offsets make independent consumption and replay possible, but offset commit timing determines failure behavior. Processing before committing usually favors at-least-once delivery and requires idempotent consumers, while committing too early can lose business work during crashes.

The central design principle is: choose partitions from business ordering boundaries and required throughput, then design offsets and replay around explicit failure behavior. Partition count, partition keys, retention, and consumer progress should be treated as application architecture decisions rather than Kafka tuning details.

More Articles to Read

• Apache Kafka Explained: How Kafka Works
• Kafka Producers Explained: Partitioning, Batching, and Delivery Guarantees
• Kafka Consumers and Consumer Groups Explained