Designing Event-Driven Systems with Kafka

Kafka makes it possible to connect services through durable event streams instead of synchronous request chains. The difficult part is not publishing records; it is deciding what should be an event, who owns it, how services change state safely, how failures are retried, and how schemas evolve without coupling every consumer to the producer.

A production event-driven system should remain useful when consumers are slow, services restart, events are delivered more than once, schemas change, and historical data is replayed. Kafka provides the transport and storage primitives, while the application architecture determines whether those conditions remain safe.

Table of Contents

• From Synchronous Calls to Events
• Design Events Around Business Facts
  • Events vs Commands
  • Event Payload Design
• Define Service Ownership
• Choose Topic and Partition Boundaries
  • Topic Design
  • Partition Key Design
• Publish Events Without Dual-Write Gaps
• Design Consumers for Duplicate Delivery
• Handle Failures Without Blocking the Stream
• Design for Eventual Consistency
• Evolve Event Contracts Safely
• Practical Order Processing Architecture
  • Creating and Publishing the Order
  • Processing Inventory
  • Handling Payment and Failure
• Scale by Bottleneck, Not by Service Count
• Observe the Business Flow
• Production Design Principles
• Conclusion
  • More Articles to Read

From Synchronous Calls to Events

Consider an Order Service that must coordinate inventory, payments, notifications, analytics, and shipping. A synchronous implementation might call each dependency while processing the original request.

Client → Orders → Inventory → Payments → Shipping → Notifications

This architecture is simple when the workflow is small, but its availability and latency increasingly depend on downstream services. A slow notification provider can extend checkout latency, and an unavailable analytics service can become part of the critical path unless every dependency is carefully isolated.

An event-driven design changes the dependency direction. The Order Service commits its own state and publishes an event such as order.created. Independent consumers react asynchronously.

Order Service → Kafka → Inventory / Payments / Analytics / Notifications

The Order Service no longer needs all consumers to be available when the order is created. Kafka retains the event while unavailable consumers recover.

This does not automatically make the system simpler. Synchronous coupling is replaced by temporal decoupling, asynchronous state changes, duplicate delivery, event contracts, consumer lag, and eventual consistency.

Kafka fundamentals are covered in Apache Kafka Explained: How Kafka Works.

Design Events Around Business Facts

A useful event describes something that has already happened and can be meaningful to consumers without exposing the producer's internal implementation.

Examples include order.created, payment.authorized, shipment.dispatched, and customer.email_changed.

Events should not simply mirror every database table update. Publishing orders_row_updated forces consumers to understand the producer's persistence model rather than the business meaning of the change.

Events vs Commands

An event describes a fact. A command requests an action.

This distinction affects ownership. Multiple services may independently react to an event. A command normally has a specific capability owner responsible for deciding whether the requested operation succeeds.

Using event names that secretly contain commands creates unclear semantics. An event named send_order_email is really an instruction to a notification service. order.created describes the business fact and lets Notification decide whether that fact requires an email.

Event Payload Design

An event should contain enough information to identify the business fact and support intended consumers without becoming a copy of the producer's entire database entity.

A practical order event might look like:

{
  "event_id": "evt_73912",
  "event_type": "order.created",
  "event_version": 1,
  "occurred_at": "2026-09-08T19:42:11Z",
  "order_id": "ord_92814",
  "customer_id": "cus_441",
  "currency": "USD",
  "total_amount": "149.90"
}

event_id provides a stable identity for deduplication and tracing. occurred_at records business-event time rather than Kafka ingestion time. event_version provides an explicit contract evolution mechanism when needed.

Avoid putting rapidly changing or sensitive data into events without a reason. Kafka records can be retained and copied into several downstream systems, making later deletion or correction substantially harder than updating one database row.

Define Service Ownership

Event-driven architecture works best when each important piece of business state has a clear owner.

For an e-commerce platform:

• Order Service owns order lifecycle state.
• Inventory Service owns stock and reservations.
• Payment Service owns payment attempts and results.
• Shipping Service owns shipment lifecycle.

Other services learn about state changes through events instead of directly modifying the owner's database.

For example, Payment should not update orders.status directly. It publishes payment.authorized, and Order decides what that means for its own state.

This creates additional events, but it preserves ownership and prevents databases from becoming hidden integration APIs.

A consumer can maintain its own local projection when it needs data from another domain. Shipping may store the delivery address required to create a shipment rather than synchronously querying Order for every operation.

This improves runtime independence at the cost of eventual consistency and duplicated data. That trade-off is usually worthwhile only when service boundaries represent genuine ownership boundaries rather than arbitrary code separation.

Choose Topic and Partition Boundaries

Kafka architecture introduces two related boundaries. Topics organize streams and operational policies, while partitions determine ordering and parallelism inside those streams.

Neither should be derived mechanically from database tables or microservice names.

Topic Design

Topics should group events that belong together operationally and semantically.

A system might use domain streams such as:

• order-events
• payment-events
• shipment-events

Creating one topic for every individual event type can produce hundreds of tiny topics with repetitive configuration and operational overhead. Putting every event in one global topic creates the opposite problem: unrelated domains share retention, partitioning, permissions, scaling, and failure policies.

The right boundary depends on requirements such as retention, security, throughput, ordering, ownership, and consumer patterns.

For example, payment events may deserve stronger access controls and longer retention than high-volume page-view events. Keeping them in separate topics allows those policies to evolve independently.

Partition Key Design

The partition key should normally represent the entity whose events must remain ordered.

For order lifecycle events:

producer.produce(
    topic="order-events",
    key=order_id,
    value=event_payload,
)

This keeps events for one order in the same partition while different orders can be processed in parallel.

Choosing customer_id instead would preserve ordering across all orders belonging to one customer. That may be necessary for some domains, but it creates a larger ordering boundary and potentially more skew.

Choosing country can be much worse if one country generates most traffic. One partition may become hot while others remain underused.

The narrowest business entity that requires ordering is often a strong starting point for the partition key.

Partition design is covered in depth in Kafka Topics, Partitions, and Offsets Explained.

Publish Events Without Dual-Write Gaps

One of the most important event-driven design problems appears when a service must update its database and publish an event.

A naive Order Service performs:

1. Insert the order into PostgreSQL.
2. Publish order.created to Kafka.

If the process crashes after step one, the order exists but consumers never receive the event.

Publishing first creates the inverse failure: consumers can observe an order that never commits to the source database.

A transactional outbox solves this by storing the business change and pending event in one local transaction.

BEGIN;

INSERT INTO orders (
    id,
    customer_id,
    status,
    total_amount
)
VALUES (
    'ord_92814',
    'cus_441',
    'created',
    149.90
);

INSERT INTO outbox_events (
    event_id,
    aggregate_id,
    event_type,
    payload,
    created_at
)
VALUES (
    'evt_73912',
    'ord_92814',
    'order.created',
    '{"order_id":"ord_92814","customer_id":"cus_441","total_amount":"149.90"}',
    CURRENT_TIMESTAMP
);

COMMIT;

An outbox publisher later sends pending events to Kafka. If Kafka is unavailable, the event remains durable in PostgreSQL and publication can resume later.

Publication may occur more than once if the publisher crashes around the Kafka-send and outbox-update boundary. Consumers should therefore remain idempotent.

The complete pattern is explained in Transactional Outbox Pattern for Reliable Messaging.

Design Consumers for Duplicate Delivery

At-least-once processing is a practical default for many important event-driven workflows. It avoids silently losing work but means a consumer can receive the same event more than once.

Consider an Inventory consumer:

1. Read order.created.
2. Reserve inventory in PostgreSQL.
3. Crash before committing the Kafka offset.
4. Restart and receive the event again.

Incrementing reserved_quantity on every delivery would reserve stock twice.

A safer design creates a reservation under a stable business identity:

INSERT INTO inventory_reservations (
    order_id,
    product_id,
    quantity
)
VALUES (
    'ord_92814',
    'prd_501',
    2
)
ON CONFLICT (order_id, product_id) DO NOTHING;

The unique business key makes repeated delivery safe.

Another option stores event_id in a processed-events table in the same transaction as the business change. Domain-level constraints are often preferable when a natural business identity exists because they also protect against duplicate commands that arrive through another path.

Delivery semantics and idempotency are covered in Kafka Delivery Semantics: At-Most-Once, At-Least-Once, and Exactly-Once.

Handle Failures Without Blocking the Stream

Consumer failures should be classified rather than handled with one unlimited retry loop.

A database connection timeout may succeed on the next attempt. An event missing a required field will not become valid after 10,000 retries.

Moving failed records to a retry or dead letter topic is useful only when the resulting ordering change is acceptable.

Suppose account.updated offset 100 fails and is moved aside while offsets 101 and 102 continue. The eventual processing order may become 101, 102, 100.

For independent notification events that may be harmless. For a financial state machine it can be incorrect.

Failure handling must preserve the business invariants, not merely keep consumer lag low.

Design for Eventual Consistency

Asynchronous events mean different services do not change state simultaneously.

An order may be created while inventory reservation is still pending. A payment may succeed before Order receives the corresponding payment.authorized event.

This is not necessarily an error. It is the expected consistency model.

Applications should represent intermediate states explicitly rather than pretending the workflow is immediately consistent.

An order lifecycle might include:

• pending_inventory
• inventory_reserved
• payment_pending
• confirmed
• failed

User-facing APIs should also expose meaningful state. Returning 200 OK with an order described as fully confirmed while asynchronous payment processing is still pending creates a misleading contract.

For workflows that span several services, compensating actions may be required. If payment succeeds but inventory cannot be reserved, the workflow may need to refund or void the payment rather than attempting a distributed database rollback.

The architecture should define these failure transitions explicitly. Kafka provides durable communication between the steps, but it does not decide the business compensation logic.

Evolve Event Contracts Safely

Events become integration contracts. A producer may deploy today while some consumers remain on versions released weeks earlier.

Changing an event therefore requires compatibility planning.

Suppose version one contains:

{
  "event_type": "order.created",
  "order_id": "ord_92814",
  "total_amount": "149.90"
}

Adding an optional currency field is usually easier to evolve than renaming total_amount to amount and immediately removing the original field.

Consumers should generally tolerate fields they do not use. Producers should avoid silently changing field meaning while retaining the same name.

Schema technologies such as Avro or Protobuf and a schema registry can automate compatibility checks, but tooling cannot decide whether a semantic change is safe. Changing total_amount from "before tax" to "after tax" may be syntactically compatible and still break every consumer's assumptions.

A safe event contract requires both structural compatibility and stable business meaning.

Practical Order Processing Architecture

Consider an order workflow involving Order, Inventory, Payment, and Notification services. The goal is to avoid synchronous dependency chains while keeping business state recoverable after failures.

The workflow uses domain events, transactional outboxes, idempotent consumers, and explicit intermediate states.

Creating and Publishing the Order

The Order Service receives a request containing an application-level idempotency key. It creates the order and its outbox event in one transaction.

BEGIN;

INSERT INTO orders (
    id,
    idempotency_key,
    customer_id,
    status,
    total_amount
)
VALUES (
    'ord_92814',
    'checkout_71281',
    'cus_441',
    'pending_inventory',
    149.90
)
ON CONFLICT (idempotency_key) DO NOTHING;

INSERT INTO outbox_events (
    event_id,
    aggregate_id,
    event_type,
    payload
)
VALUES (
    'evt_order_73912',
    'ord_92814',
    'order.created',
    '{"order_id":"ord_92814","customer_id":"cus_441","total_amount":"149.90"}'
);

COMMIT;

In a real implementation, outbox creation must occur only when the corresponding order creation is accepted for that logical request. The example illustrates the shared transaction boundary rather than complete application control flow.

The outbox publisher writes the event to order-events using order_id as the Kafka key.

If Kafka is unavailable, checkout does not need to lose the event. The committed outbox row remains available for later publication.

Processing Inventory

Inventory consumes order.created and attempts to create reservations.

When reservation succeeds, it publishes:

{
  "event_id": "evt_inventory_291",
  "event_type": "inventory.reserved",
  "order_id": "ord_92814",
  "reservation_id": "res_8821",
  "occurred_at": "2026-09-08T19:42:14Z"
}

If inventory is unavailable for five minutes, Order does not need to synchronously wait on an HTTP connection. The event remains in Kafka and consumer lag increases until Inventory recovers.

The trade-off is that the order remains in pending_inventory longer. The business must define how long this state is acceptable and what happens when reservation cannot complete.

Handling Payment and Failure

Payment begins only after the required inventory state is reached. The Payment Service calls an external provider using a stable payment ID as its idempotency key.

result = payment_provider.charge(
    amount_cents=14990,
    payment_method_id="pm_8291",
    idempotency_key="pay_ord_92814",
)

If the consumer crashes after the provider successfully charges the card, Kafka may redeliver the event. Reusing the same idempotency key prevents the retry from becoming a second logical charge when the provider supports idempotent operations.

Payment publishes either payment.authorized or payment.failed. Order consumes the result and changes its own state.

If payment fails after inventory has been reserved, the workflow publishes or triggers a release operation. Inventory remains responsible for its own state and releases the reservation idempotently.

This is a practical example of distributed compensation: no global transaction spans Order, Inventory, Kafka, and the payment provider. Each service performs local durable changes and the workflow repairs partial progress through explicit events and compensating actions.

Scale by Bottleneck, Not by Service Count

Kafka allows consumer groups to scale horizontally, but adding service instances does not automatically increase throughput.

Suppose order-events receives 80,000 records per second and Inventory has eight partitions. At most eight consumers in one group can actively own partitions at the same time.

If each consumer processes 12,000 records per second, eight consumers provide enough theoretical processing capacity. A ninth instance does not add partition-level concurrency.

But the database may become saturated before eight consumers reach that rate. If every event executes two PostgreSQL statements and the database sustains only 60,000 event transactions per second, adding consumers merely increases connection contention.

The real throughput is bounded by the slowest constrained resource:

• partition count;
• consumer CPU;
• database throughput;
• connection-pool capacity;
• external API rate limits;
• broker network and disk capacity.

Consumer capacity also needs recovery headroom. A group that can process exactly 80,000 events per second cannot drain a backlog while producers continue generating 80,000 events per second.

Steady-state capacity and recovery capacity are different requirements.

Observe the Business Flow

Kafka infrastructure can be completely healthy while a business workflow is broken. Monitoring only broker CPU, disk, and network is therefore insufficient.

A useful event-driven system exposes both messaging and business signals.

• Consumer lag by group and partition. Shows which processors are falling behind.
• Oldest-event age. Converts record backlog into business delay.
• Event processing latency. Measure p50, p95, and p99 handler duration.
• Outbox age and depth. Reveals events committed by services but not yet published.
• Retry rate. Detects transient failures consuming processing capacity.
• Dead letter or quarantine volume. Exposes events that cannot complete normal processing.
• Duplicate detections. Shows how frequently idempotency mechanisms are being exercised.
• Workflow age. Measure how long orders, payments, shipments, or other business processes remain in intermediate states.

Correlation identifiers should allow one logical workflow to be traced across services. event_id identifies one event, while identifiers such as order_id or a dedicated correlation ID connect multiple events belonging to the same business process.

For example, an incident investigation should make it possible to determine:

1. when order.created was produced;
2. when Inventory consumed it;
3. whether reservation succeeded;
4. when inventory.reserved was produced;
5. whether Payment processed it;
6. which state currently blocks completion.

Without this visibility, asynchronous decoupling can become operational opacity.

Production Design Principles

A practical Kafka architecture benefits from a small set of rules that connect messaging behavior to business correctness.

• Publish business facts, not database implementation details. Events should communicate stable domain meaning.
• Give every state a clear owner. Other services should react through contracts instead of modifying another service's database.
• Choose partition keys from ordering requirements. Preserve only the ordering boundary the business actually needs while maintaining good distribution.
• Make publication durable before assuming an event exists. Use an outbox or another explicit solution when database changes and Kafka publication must remain consistent.
• Expect duplicate delivery. Stable event IDs, business constraints, and external idempotency keys should make retries safe.
• Represent intermediate states explicitly. Event-driven workflows are eventually consistent and should expose that reality.
• Classify failures before retrying. Temporary infrastructure failures and permanently invalid events need different handling.
• Protect event contracts. Schema compatibility and semantic compatibility are both required.
• Keep recovery capacity. Consumers must be able to process new traffic while draining backlog after an outage.
• Monitor business progress, not Kafka alone. Healthy brokers do not guarantee healthy orders, payments, or shipments.

Conclusion

Kafka enables services to communicate through durable event streams, reducing synchronous runtime dependencies and allowing consumers to process and replay data independently. The architectural benefit comes from decoupling service availability and processing rates, not simply replacing HTTP calls with Kafka records.

The trade-off is that correctness becomes asynchronous. Services must define ownership, event contracts, partition keys, idempotency, intermediate states, retries, compensation, and replay behavior explicitly.

Reliable production systems commonly combine Kafka with transactional outboxes, at-least-once processing, idempotent consumers, stable business identifiers, bounded retries, schema compatibility, and business-level observability.

The central design principle is: events should make services independently recoverable without making business state ambiguous. Kafka provides the durable communication layer; clear ownership and explicit failure behavior make the event-driven system reliable.

More Articles to Read

• Apache Kafka Explained: How Kafka Works
• Kafka Topics, Partitions, and Offsets Explained
• Kafka Consumers and Consumer Groups Explained
• Kafka Delivery Semantics: At-Most-Once, At-Least-Once, and Exactly-Once