Latest Messaging & Event Streaming Articles

Change Data Capture (CDC) is a technique for detecting changes made to a database and delivering those changes to other systems. Instead of repeatedly querying entire tables, CDC captures inserts, updates, and deletes as they happen.

Message brokers are more than infrastructure for moving messages between services. Production systems depend on a set of recurring message broker patterns that determine how work is distributed, how events are broadcast, how failures are retried, how duplicates are handled, and how services remain c

A Transactional Outbox is a reliability pattern that keeps a database change and the event or message describing that change consistent without requiring a distributed transaction between the database and a message broker.

Kafka consumer lag is the difference between the latest offset available in a Kafka partition and the offset a consumer group has processed or committed. It shows how far consumers are behind producers.

A Dead Letter Queue, usually shortened to DLQ , is a separate queue that stores messages that could not be processed successfully after normal processing or retry attempts have been exhausted.

Running Kafka in production is less about finding a perfect configuration and more about designing predictable behavior under load, failures, deployments, retries, and traffic growth. Partitioning, producer durability, consumer idempotency, schema evolution, retention, and observability all affect w

Kafka events often live much longer than the code that originally produced them. Producers and consumers deploy independently, retained records may be replayed months later, and new consumers may start by reading historical events created by older application versions.

Kafka preserves record order inside a partition, not across an entire multi-partition topic. That distinction affects partition-key design, consumer concurrency, retries, scaling, and every workflow where events must be applied in a predictable sequence.

Kafka keeps events durable, but durable transport does not make event processing reliable by itself. Consumers still need a failure strategy for database timeouts, malformed payloads, external API outages, rate limits, poison messages, and business errors that will never succeed on retry.

Kafka performance depends on how producers batch records, how partitions distribute work, how brokers use disk and network, and how fast consumer groups can process data. Scaling Kafka successfully requires identifying which layer is saturated instead of assuming that more brokers or more partitions

Kafka makes it possible to connect services through durable event streams instead of synchronous request chains. The difficult part is not publishing records; it is deciding what should be an event, who owns it, how services change state safely, how failures are retried, and how schemas evolve witho

Kafka fault tolerance is built around partition replication. Instead of storing a partition on only one broker, Kafka keeps multiple replicas across brokers so another replica can take over when the current leader becomes unavailable.

Kafka delivery semantics describe what can happen to a record when producers, brokers, consumers, networks, or downstream systems fail. The familiar terms at-most-once , at-least-once , and exactly-once are useful only when the processing boundary is defined precisely.

Kafka consumers turn durable event streams into application work. They fetch records from partitions, process them, track progress through offsets, and cooperate through consumer groups to divide a topic across multiple application instances.

A Kafka producer does much more than send individual messages to a broker. It chooses partitions, accumulates records into batches, compresses data, retries failed requests, waits for acknowledgements, and can prevent many retry-generated duplicates through idempotent publishing.

Kafka topics, partitions, and offsets define how data is organized, scaled, ordered, consumed, and replayed. Most Kafka design mistakes eventually come back to one of these three concepts.

Apache Kafka is a distributed event-streaming platform designed for moving large volumes of records between systems while keeping those records durable, ordered within defined boundaries, and available for later replay.

Message queues help distributed systems process work asynchronously, absorb traffic spikes, isolate failures, and decouple producers from consumers. They are commonly used for background jobs, order processing, notifications, logistics workflows, data pipelines, payments, and communication between m

Distributed applications often need to update a database and publish a message as part of one business operation. An order service may save a confirmed order and publish an order.confirmed event. A payment service may record a successful charge and notify fulfilment. If one action succeeds while the

Message processing does not always succeed on the first attempt. A database may be temporarily unavailable, an external API may return a rate-limit response, a network request may time out, or a message may contain invalid data that can never be processed successfully. Reliable messaging systems mus

Background workers execute tasks outside the main request-response flow of an application. Instead of making an API client wait while a report is generated, an email is sent, or a video is processed, the application places a job into a queue and returns a response quickly. A separate worker receives

Message delivery guarantees describe what can happen when producers, brokers, consumers, networks, or databases fail. A message may be lost, delivered multiple times, or processed only once under specific conditions. These behaviours are commonly described as at-most-once , at-least-once , and exact

Event-driven architecture allows distributed components to communicate by publishing facts about completed business actions. Instead of calling every dependent service directly, a service publishes an event such as order.created , payment.completed , or shipment.delivered . Other services process th

Modern applications often need to perform work that should not block the main request. An order service may need to reserve inventory, send a confirmation email, update analytics, and notify a warehouse. Executing every operation synchronously increases response time and creates direct dependencies

In Scalability for Dummies - Part 3: Cache , we reduced database load by avoiding unnecessary queries.