Kafka Consumers and Consumer Groups Explained

Kafka consumers turn durable event streams into application work. They fetch records from partitions, process them, track progress through offsets, and cooperate through consumer groups to divide a topic across multiple application instances.

The difficult production decisions are not how to call poll(), but when to commit offsets, how much work to perform per record, how to scale without creating idle consumers, and how to survive crashes and rebalances without losing or incorrectly duplicating business operations.

Table of Contents

• How Kafka Consumption Works
• Consumer Groups Distribute Work
  • Partition Assignment
  • Independent Consumer Groups
• Polling and Processing Records
• Offsets and Commit Strategies
  • Commit Before Processing
  • Commit After Processing
  • Automatic vs Manual Commits
• Idempotent Consumers
• Rebalancing and Consumer Failures
• Slow Consumers and Consumer Lag
• Scaling Consumers Correctly
• Handling Slow and Failing Events
• Practical Inventory Consumer
• Production Mistakes to Avoid
• What to Monitor in Production
• Conclusion
  • More Articles to Read

How Kafka Consumption Works

A Kafka consumer reads records from topic partitions. Unlike a traditional queue where successful consumption may remove a message, Kafka normally retains the record according to the topic's retention policy.

The consumer separately tracks its position using offsets.

Suppose partition orders-2 contains:

Offset 8101 → 8102 → 8103 → 8104 → 8105

If a consumer group has successfully processed through offset 8103, its stored progress tells Kafka where processing should resume after a restart or partition reassignment.

This separation between stored records and consumer progress enables replay, multiple independent consumers, and recovery after consumer failures.

Topics, partitions, and offset mechanics are covered in more detail in Kafka Topics, Partitions, and Offsets Explained.

Consumer Groups Distribute Work

A consumer group represents one logical subscriber to a topic. Several application instances can join the same group, and Kafka distributes partitions among them.

This is the main mechanism for horizontally scaling Kafka processing.

For example, an Inventory Service may run four instances using the same group ID:

{
  "group.id": "inventory-service"
}

Kafka treats these processes as cooperating instances of one logical consumer rather than four independent subscribers.

Partition Assignment

Within one consumer group, one partition is assigned to at most one active consumer at a time.

Suppose order-events has six partitions and three Inventory Service instances are running.

Each consumer can process its assigned partitions concurrently with the other instances while Kafka preserves ordering inside each individual partition.

If a fourth consumer joins, Kafka can redistribute the six partitions. If ten consumers join, only six can receive partitions. The remaining four cannot increase partition-level throughput.

Consumer-group parallelism is bounded by partition count.

Independent Consumer Groups

Different applications use different group IDs when each application needs to receive the same events independently.

An order.created event might be consumed by:

• inventory-service
• analytics-service
• notification-service
• fraud-service

Each group maintains its own offsets. Inventory may be completely caught up while Analytics remains 500,000 records behind.

Adding another consumer to inventory-service does not create another logical subscription. It shares Inventory's existing partition workload.

This distinction prevents a common configuration mistake. Two applications that both need every event should not accidentally share the same group ID, because Kafka would distribute partitions between them instead of giving each application the full stream.

Polling and Processing Records

Kafka consumers generally pull data from brokers. The application repeatedly polls for available records rather than Kafka opening an application endpoint and pushing each record individually.

A simplified Python consumer looks like:

import json

from confluent_kafka import Consumer


consumer = Consumer({
    "bootstrap.servers": "kafka-1:9092,kafka-2:9092,kafka-3:9092",
    "group.id": "inventory-service",
    "enable.auto.commit": False,
    "auto.offset.reset": "earliest",
})

consumer.subscribe(["order-events"])

while True:
    message = consumer.poll(1.0)

    if message is None:
        continue

    if message.error():
        raise RuntimeError(message.error())

    event = json.loads(message.value())

    reserve_inventory(event)

    consumer.commit(message=message, asynchronous=False)

The important operation is not the loop itself. It is the relationship between reserve_inventory() and the offset commit.

If inventory processing succeeds but the consumer crashes before committing, the event can be delivered again. If the offset is committed first and the consumer crashes before reserving inventory, the business operation may never happen.

This creates the central Kafka consumer design question: which failure is safer for this operation—possible duplication or possible loss?

Offsets and Commit Strategies

Offset commits record how far a consumer group has progressed. Commit timing strongly affects processing semantics during crashes, deployments, timeouts, and partition reassignments.

The relevant boundary is usually not whether Kafka delivered the record, but whether the application's important side effect completed.

Commit Before Processing

Consider this sequence:

1. Read offset 500.
2. Commit progress beyond offset 500.
3. Update the database.

If the process crashes between steps two and three, Kafka considers the record consumed even though the database was never updated.

This produces at-most-once-like behavior at the application-processing boundary: duplicate processing is reduced, but records can be effectively lost.

That may be acceptable for disposable telemetry where losing an occasional sample has negligible business impact. It is usually dangerous for payments, orders, inventory, or account changes.

Commit After Processing

Now reverse the important operations:

1. Read offset 500.
2. Update the database.
3. Commit progress beyond offset 500.

If the database update succeeds and the consumer crashes before step three, Kafka can deliver the event again.

No business operation is silently lost, but the application may execute the same event more than once.

This is usually preferable for critical business processing when the consumer is designed to be idempotent.

At-least-once delivery moves complexity from message loss to duplicate handling. For many business workflows, that is the safer failure mode because duplicates can be detected while missing events are often difficult to reconstruct.

Automatic vs Manual Commits

Automatic offset commits reduce application code, but they can make the relationship between business completion and offset advancement less explicit.

Manual commits allow the application to define exactly when Kafka progress is considered durable.

For critical processing, this explicit control is often valuable:

event = deserialize(message.value())

process_event(event)

consumer.commit(
    message=message,
    asynchronous=False,
)

Manual commits do not make processing exactly once. A failure can still occur after process_event() succeeds but before the commit reaches Kafka.

Correctness therefore requires consumer-side idempotency or another coordination mechanism when duplicates matter.

Idempotent Consumers

An idempotent consumer can safely receive the same logical event more than once without applying the business effect more than once.

Stable event IDs are one common building block.

Suppose Inventory receives:

{
  "event_id": "evt_73912",
  "event_type": "order.created",
  "order_id": "ord_92814",
  "product_id": "prd_501",
  "quantity": 2
}

The database can record processed event IDs in the same transaction as the inventory update.

BEGIN;

INSERT INTO processed_events (
    consumer_name,
    event_id,
    processed_at
)
VALUES (
    'inventory-service',
    'evt_73912',
    CURRENT_TIMESTAMP
)
ON CONFLICT (consumer_name, event_id) DO NOTHING;

-- Continue with the reservation only when the insert succeeded.

UPDATE inventory
SET reserved_quantity = reserved_quantity + 2
WHERE product_id = 'prd_501';

COMMIT;

The application must actually check whether the event-marker insert succeeded before executing the update. The SQL demonstrates the transaction boundary, but unconditional execution of the second statement would still reserve inventory twice.

Another approach is to make the business operation itself naturally idempotent. Instead of incrementing a counter for every delivery, a system can create a reservation identified by order_id and product_id under a unique constraint.

INSERT INTO inventory_reservations (
    order_id,
    product_id,
    quantity
)
VALUES (
    'ord_92814',
    'prd_501',
    2
)
ON CONFLICT (order_id, product_id) DO NOTHING;

Domain-level idempotency is often stronger because it protects against duplicate business commands even when they arrive with different Kafka metadata.

Rebalancing and Consumer Failures

Consumer-group membership changes over time. Instances start, stop, crash, deploy new versions, lose connectivity, or become too slow to maintain their expected group participation.

Kafka must then redistribute partition ownership among active consumers. This process is called a rebalance.

Suppose three consumers own six partitions. If inventory-2 crashes, its partitions must move to the remaining consumers.

The replacement consumer resumes from the group's committed offsets. Any successfully processed but uncommitted records may therefore appear again.

Rebalancing can also temporarily interrupt processing. Frequent rebalances reduce effective throughput and can produce lag spikes even when brokers have plenty of capacity.

Common causes include unstable instances, aggressive deployment churn, long processing operations, blocked poll loops, and poorly matched consumer timeout settings.

Rebalancing is therefore both a correctness concern and a performance concern.

Slow Consumers and Consumer Lag

A consumer does not need to process events as quickly as producers create them every second, because Kafka can buffer records durably. But over sustained periods, processing capacity must at least match incoming traffic or backlog grows continuously.

Suppose producers write 40,000 events per second while the consumer group processes 30,000.

The backlog grows by:

incoming_rate = 40_000
processing_rate = 30_000

lag_growth_per_second = incoming_rate - processing_rate

print(lag_growth_per_second)  # 10000

After one hour, the group is approximately 36 million records further behind if rates remain constant.

Record count alone does not reveal business impact. Ten million tiny telemetry events may represent two minutes of traffic, while 50,000 payment events may represent hours.

For this reason, production systems should monitor both offset lag and time lag, such as the age of the oldest unprocessed event.

Lag can increase because of:

• insufficient consumer instances;
• too few partitions;
• slow database queries;
• external API latency;
• downstream rate limits;
• lock contention;
• repeated retries;
• large or expensive records;
• frequent consumer rebalances.

Adding consumers helps only when the bottleneck is parallelizable and enough partitions exist.

Scaling Consumers Correctly

Consumer scaling should begin with measured processing capacity rather than instance count.

Suppose a topic receives 90,000 events per second. One consumer instance sustainably processes 15,000 events per second while maintaining acceptable database latency.

import math

incoming_rate = 90_000
consumer_capacity = 15_000

minimum_consumers = math.ceil(
    incoming_rate / consumer_capacity
)

print(minimum_consumers)  # 6

Six active consumers can theoretically keep up, so the topic needs at least six partitions to expose that degree of parallelism.

Production capacity normally needs headroom. Running exactly at sustainable maximum leaves no room for traffic bursts, instance failures, deployments, database slowdown, or catching up after an outage.

Suppose traffic continues at 90,000 events per second while half the consumers are unavailable for ten minutes. Once capacity returns, the group needs throughput above 90,000 events per second to process new traffic and drain accumulated lag simultaneously.

This is an often-missed capacity requirement: a consumer group sized only for steady-state traffic may never recover from a significant backlog.

Scaling also stops helping when the real bottleneck is downstream. Doubling consumers against a PostgreSQL database already at maximum connection or I/O capacity can make performance worse.

Consumer scaling should therefore be evaluated together with database connections, external service limits, CPU, memory, partition count, and per-event processing cost.

Handling Slow and Failing Events

One problematic record can block progress on a partition when processing is strictly sequential.

Suppose a consumer receives a shipment event that repeatedly fails because its payload violates a business constraint. Retrying it forever prevents every later event in that partition from progressing.

Immediate unlimited retries are rarely appropriate. A practical strategy distinguishes between transient and permanent failures.

• Transient database timeout. Retry with bounded backoff.
• External API rate limit. Respect backoff or move work to a delayed retry mechanism.
• Malformed event. Repeated retries will not repair the payload; capture it for investigation.
• Missing dependency that may arrive later. Retry according to a bounded policy that matches the expected delay.

Retry topics or dead letter topics can isolate records that cannot currently be processed, but they change ordering behavior. Moving offset 100 to a retry topic while continuing with offset 101 means the business effects may no longer occur in original partition order.

That may be acceptable for independent notification events but dangerous for account ledger transitions.

Failure handling therefore must start from the business ordering requirement rather than automatically adding a dead letter topic.

Practical Inventory Consumer

Consider an Inventory Service consuming order.created events. The requirements are:

• an event must not be silently lost after Kafka delivery;
• duplicate deliveries must not reserve stock twice;
• processing errors must not commit the offset;
• the consumer should continue after recoverable failures.

A simplified implementation can use a database transaction with a stable event ID:

import json
from typing import Any

import psycopg
from confluent_kafka import Consumer


consumer = Consumer({
    "bootstrap.servers": "kafka-1:9092,kafka-2:9092,kafka-3:9092",
    "group.id": "inventory-service",
    "enable.auto.commit": False,
    "auto.offset.reset": "earliest",
})

consumer.subscribe(["order-events"])


def reserve_inventory(
    connection: psycopg.Connection[Any],
    event: dict[str, Any],
) -> None:
    with connection.transaction():
        inserted = connection.execute(
            """
            INSERT INTO processed_events (
                consumer_name,
                event_id
            )
            VALUES (%s, %s)
            ON CONFLICT (consumer_name, event_id) DO NOTHING
            RETURNING event_id
            """,
            ("inventory-service", event["event_id"]),
        ).fetchone()

        if inserted is None:
            return

        connection.execute(
            """
            INSERT INTO inventory_reservations (
                order_id,
                product_id,
                quantity
            )
            VALUES (%s, %s, %s)
            """,
            (
                event["order_id"],
                event["product_id"],
                event["quantity"],
            ),
        )


with psycopg.connect(
    "postgresql://inventory:password@postgres/inventory"
) as connection:
    while True:
        message = consumer.poll(1.0)

        if message is None:
            continue

        if message.error():
            raise RuntimeError(message.error())

        event = json.loads(message.value())

        reserve_inventory(connection, event)

        consumer.commit(
            message=message,
            asynchronous=False,
        )

The database transaction atomically records the event ID and creates the reservation. If the transaction fails, neither operation is committed.

If the database transaction succeeds but the Kafka offset commit fails, the event may be delivered again. The processed_events constraint recognizes the duplicate and skips the second reservation.

If the consumer crashes before the database commit, Kafka eventually redelivers the event and another attempt can perform the reservation.

This does not create universal exactly-once processing. It works because both deduplication state and the business change share one transactional database. Calling an external payment provider or carrier API requires a different idempotency strategy.

Production Mistakes to Avoid

Consumer problems often appear as Kafka problems even when the actual failure is in application processing.

• Sharing a group ID between independent applications. Kafka divides partitions between them instead of delivering the complete stream to both. Give each logical subscriber its own group.
• Committing before critical work completes. A crash can permanently skip the business operation. Commit after durable processing when loss is unacceptable.
• Assuming manual commits prevent duplicates. A crash can occur after the business transaction but before the commit. Make the consumer idempotent.
• Scaling beyond partition count. Extra instances remain idle and add deployment complexity without increasing partition parallelism.
• Retrying permanent failures forever. One poison event can block an entire partition. Classify errors and use bounded failure policies.
• Calling slow external APIs directly without capacity planning. One slow dependency can create rapidly increasing Kafka lag. Apply timeouts, concurrency limits, and appropriate asynchronous boundaries.
• Monitoring only total lag. One hot partition can be severely delayed while cluster-wide averages appear healthy. Keep partition-level visibility.
• Ignoring recovery throughput. A group that barely handles normal traffic cannot drain backlog after an outage.

What to Monitor in Production

Useful consumer monitoring should reveal whether processing is current, whether instances are stable, and what dependency is limiting throughput.

• Consumer lag per partition. Detect skew and partitions that stop making progress.
• Oldest unprocessed event age. Translate backlog into business delay.
• Records processed per second. Compare actual consumer throughput with producer rate.
• Processing latency. Track p50, p95, and p99 business-handler duration.
• Commit failures and latency. Detect unstable progress persistence.
• Rebalance frequency. Frequent group changes often explain throughput interruptions.
• Processing error rate. Separate transient failures from permanently invalid records.
• Retry volume. Growing retries can consume capacity before overall lag becomes critical.
• Database latency and pool utilization. A consumer often becomes limited by its persistence layer rather than Kafka.
• External dependency latency. Track any API that participates directly in event processing.

Alerts should reflect the business function of each consumer group. Ten minutes of lag in a nightly analytics pipeline may be harmless, while ten minutes of lag in fraud detection or inventory reservation can be operationally critical.

A useful dashboard connects Kafka lag with processing latency and downstream saturation. For example, increasing consumer lag combined with PostgreSQL pool utilization near 100% immediately suggests a different response than increasing lag combined with idle database capacity and high consumer CPU.

Conclusion

Kafka consumers read durable partition logs, while consumer groups divide those partitions across application instances. This creates horizontal processing capacity without sacrificing ordering inside individual partitions.

Offset commits define recovery position, not business correctness. Committing after processing generally favors at-least-once behavior, which makes duplicate delivery an expected condition and places idempotency at the center of consumer design.

Scaling requires enough partitions, but adding consumers helps only when downstream systems can handle the additional concurrency. Database capacity, external APIs, processing latency, retries, and recovery throughput are often the real limits.

The most important production principle is: design the consumer around crashes and duplicate delivery first, then optimize throughput. A fast consumer that loses events or repeats irreversible side effects is not a reliable consumer.

More Articles to Read

• Apache Kafka Explained: How Kafka Works
• Kafka Topics, Partitions, and Offsets Explained
• Kafka Producers Explained: Partitioning, Batching, and Delivery Guarantees
• Kafka Delivery Semantics: At-Most-Once, At-Least-Once, and Exactly-Once