What Is Change Data Capture?

Change Data Capture (CDC) is a technique for detecting changes made to a database and delivering those changes to other systems. Instead of repeatedly querying entire tables, CDC captures inserts, updates, and deletes as they happen.

CDC is commonly used to synchronize databases, update search indexes and caches, build analytics pipelines, publish database changes to message brokers, and connect operational databases with downstream services.

Table of Contents

• Why Change Data Capture Exists
• How Change Data Capture Works
• CDC Approaches
  • Polling-Based CDC
  • Trigger-Based CDC
  • Log-Based CDC
• Database Transaction Logs
• CDC Event Structure
• CDC Delivery and Offsets
• Ordering and Duplicates
• Schema Changes
• CDC vs Transactional Outbox
• Common CDC Use Cases
• Production Design Example
• CDC Failure Scenarios
• When to Use CDC
• Common CDC Mistakes
• Frequently Asked Questions
  • Does CDC Require Kafka?
  • Does CDC Capture Deletes?
  • Is CDC Real-Time?
  • Can CDC Lose Events?
  • Should CDC or Transactional Outbox Be Used?
• Conclusion
  • More Articles to Read

Why Change Data Capture Exists

Suppose an application stores orders in PostgreSQL while an analytics platform, search engine, and data warehouse also need information about those orders.

One approach is repeatedly querying the database:

SELECT *
FROM orders
WHERE updated_at > :last_sync;

This can work for simple systems, but it has several weaknesses.

The query must run repeatedly even when nothing changed. Large tables become expensive to scan. Deletes are difficult to detect. Timestamp boundaries can produce missing or duplicate records. Synchronization latency depends on the polling interval.

CDC changes the model.

Database Change
      ↓
CDC
      ↓
Change Event
      ↓
Downstream Systems

Instead of asking the database what changed, the system observes the changes as they occur.

How Change Data Capture Works

At a high level, CDC converts database mutations into a stream of change records.

Application
     ↓
 Database
     ↓
 INSERT / UPDATE / DELETE
     ↓
 CDC Reader
     ↓
 Change Stream
     ↓
 Consumers

Suppose an application updates an order:

UPDATE orders
SET status = 'SHIPPED'
WHERE id = 'order-8472';

A CDC system may produce a change record representing the old and new values:

{
  "operation": "update",
  "table": "orders",
  "key": {
    "id": "order-8472"
  },
  "before": {
    "status": "PAID"
  },
  "after": {
    "status": "SHIPPED"
  }
}

That record can then be consumed by another database, search index, analytics pipeline, cache invalidation service, or message-processing system.

The exact event format depends on the CDC technology and database.

CDC Approaches

There are several ways to detect database changes. The most common are polling, database triggers, and transaction-log reading.

Polling-Based CDC

Polling periodically queries rows that changed after a known position or timestamp.

SELECT *
FROM orders
WHERE updated_at > :last_timestamp
ORDER BY updated_at
LIMIT 1000;

The consumer stores the last processed position and uses it in the next query.

This approach is easy to understand and can be sufficient for small workloads where several seconds or minutes of delay are acceptable.

However, timestamp-based polling has subtle edge cases.

Multiple rows may have the same timestamp. Application clocks can differ. A transaction can begin before the polling boundary but commit afterward. Deletes disappear entirely unless they are represented using soft deletes or another tracking mechanism.

A monotonically increasing change sequence can make polling safer, but the application still repeatedly queries operational tables.

Trigger-Based CDC

Database triggers can record changes into a dedicated audit or change table.

orders
   ↓ INSERT / UPDATE / DELETE
Database Trigger
   ↓
order_changes

A downstream process then reads order_changes.

This approach can capture detailed changes and deletes, but it places additional logic in the database.

Triggers execute as part of database operations, so expensive trigger logic can increase transaction latency. Trigger definitions also become another production component that must be deployed, tested, versioned, and monitored.

Log-Based CDC

Log-based CDC reads the database's transaction log rather than querying application tables.

Application
     ↓
 Database
     ↓
Transaction Log
     ↓
 CDC Connector
     ↓
Change Stream

Databases already maintain transaction logs for durability, recovery, and replication. CDC can consume these logs and convert committed database changes into events.

This approach usually avoids repeatedly scanning business tables and does not require triggers on every tracked table.

For high-throughput production systems, log-based CDC is often the preferred architecture.

Database Transaction Logs

Transaction logs record changes required for database durability and recovery.

Different database engines use different mechanisms and terminology. Conceptually, the flow is similar:

BEGIN

UPDATE orders ...

COMMIT
   ↓
Transaction Log
   ↓
CDC Reader

The important detail is transaction boundaries.

A CDC pipeline should normally expose changes only when the corresponding transaction has committed.

Consider:

BEGIN;

UPDATE accounts
SET balance = balance - 100
WHERE id = 'A';

UPDATE accounts
SET balance = balance + 100
WHERE id = 'B';

COMMIT;

Publishing only the first update before knowing whether the transaction commits could expose downstream systems to state that never became durable.

A CDC implementation therefore needs to understand commit and rollback behavior rather than treating every low-level database record as an independent business event.

CDC Event Structure

A useful CDC record usually contains more than the changed row.

For example:

{
  "database": "commerce",
  "table": "orders",
  "operation": "update",
  "key": {
    "id": "order-8472"
  },
  "before": {
    "status": "PAID",
    "total": 149.99
  },
  "after": {
    "status": "SHIPPED",
    "total": 149.99
  },
  "transaction_id": "tx-991",
  "position": "18472911",
  "timestamp": "2026-09-27T18:42:15Z"
}

Common metadata includes:

• database and table;
• operation type;
• primary key;
• before and after values;
• transaction identifier;
• transaction-log position;
• event timestamp;
• schema information.

Not every consumer needs every field. Large before and after payloads can significantly increase broker traffic and storage requirements.

CDC Delivery and Offsets

A CDC reader needs to know where it is in the database change stream.

This position may be represented by a log sequence number, binary log position, replication offset, or another database-specific cursor.

Database Log

1001
1002
1003
1004 ← last processed position
1005
1006
1007

After a restart, the connector resumes from the stored position rather than reading the entire database again.

Offset durability is critical.

If the CDC process publishes change 1005 and crashes before saving its new position, it may publish 1005 again after restarting.

Publish 1005 ✓
     ↓
Crash before checkpoint
     ↓
Restart from 1004
     ↓
Publish 1005 again

This is one reason CDC consumers should not assume that every change arrives exactly once.

The distinction between transport delivery and actual business processing is also important in ordinary message systems, as described in Message Delivery Guarantees: At-Most-Once vs At-Least-Once vs Exactly-Once.

Ordering and Duplicates

CDC preserves useful ordering information because database changes originate from an ordered transaction log. That does not automatically guarantee simple global ordering throughout the entire downstream architecture.

Once events are partitioned across a broker and processed concurrently, ordering depends on the partition strategy and consumer behavior.

Suppose one order changes twice:

order-8472

PENDING
   ↓
PAID
   ↓
SHIPPED

If both changes must be processed sequentially, they should normally follow the same ordered path.

A partition key such as order_id can help:

order-8472 → Partition 3
order-8472 → Partition 3
order-8472 → Partition 3

Consumers should also tolerate duplicates. A stable source position, transaction identifier, or generated event identifier can be used for deduplication when necessary.

The broader relationship between duplicates and safe processing is covered in Idempotency and Deduplication in Distributed Systems.

Schema Changes

CDC pipelines are tightly connected to database schemas, which means schema changes can affect downstream consumers.

Consider adding a column:

ALTER TABLE orders
ADD COLUMN shipping_method VARCHAR(50);

New CDC events may now contain a field older consumers have never seen.

Renaming or removing columns is more dangerous because downstream processors may still depend on the previous schema.

Safe CDC pipelines should consider:

• backward-compatible schema evolution;
• nullable fields and defaults;
• schema versioning;
• consumer deployment order;
• data-type changes;
• renamed and removed columns;
• historical event compatibility.

When CDC events flow through Kafka, the same compatibility concerns described in Kafka Schema Evolution and Event Versioning become important.

CDC vs Transactional Outbox

CDC and the Transactional Outbox solve related problems, but they operate at different levels.

Suppose an order changes from PENDING to PAID.

Raw CDC might produce:

{
  "table": "orders",
  "operation": "update",
  "before": {
    "status": "PENDING"
  },
  "after": {
    "status": "PAID"
  }
}

An application-level outbox can instead publish:

{
  "event_type": "PaymentCaptured",
  "order_id": "order-8472",
  "payment_id": "payment-991"
}

The second message expresses business meaning rather than exposing the physical database mutation.

The two techniques can also be combined.

Application Transaction
        ↓
Business Tables + Outbox
        ↓
Database Transaction Log
        ↓
CDC
        ↓
Broker

The application atomically writes a business event into the outbox, while CDC reads the transaction log and publishes newly committed outbox rows.

This removes the need for a custom polling publisher while preserving explicit domain events. The application side of this design is covered in What Is a Transactional Outbox?.

Common CDC Use Cases

CDC is useful whenever another system needs a continuous representation of database changes.

Common use cases include:

• Database replication — propagate changes into another database.
• Search indexing — update Elasticsearch or another search engine when source rows change.
• Analytics pipelines — stream operational changes into warehouses or analytical databases.
• Cache synchronization — invalidate or update cached records after database changes.
• Data migration — continuously synchronize old and new systems during migration.
• Materialized views — build read-optimized projections from source changes.
• Event-driven integration — turn committed database changes into downstream messages.

CDC is particularly valuable when direct application changes are difficult. A legacy application may already write to a database without publishing events. CDC can expose those changes without rewriting every write path.

Production Design Example

Consider an e-commerce platform using PostgreSQL as its transactional database.

The system also needs order data in a search engine and an analytical data store.

Order Service
     ↓
 PostgreSQL
     ↓
Transaction Log
     ↓
 CDC Connector
     ↓
 Message Broker
     │
     ├──→ Search Indexer
     │
     └──→ Analytics Consumer

The Order service updates an order:

UPDATE orders
SET
    status = 'SHIPPED',
    shipped_at = NOW()
WHERE id = 'order-8472';

The transaction commits.

The CDC connector reads the committed change from the database log and publishes a record to the broker.

{
  "operation": "update",
  "table": "orders",
  "key": {
    "id": "order-8472"
  },
  "after": {
    "id": "order-8472",
    "status": "SHIPPED"
  },
  "position": "18472911"
}

The Search Indexer updates the searchable representation:

CDC Event
    ↓
Search Indexer
    ↓
orders-search-index

The analytics consumer independently stores the change in the analytical pipeline.

If the Search Indexer is unavailable for 20 minutes, the source database continues processing transactions. CDC events accumulate in the broker rather than forcing the Order service to wait for the search system.

When the indexer returns, it processes the backlog.

This architecture separates the availability of the operational database from the availability of downstream systems.

It also creates eventual consistency. Immediately after an order is updated, PostgreSQL may contain the new state while the search index still contains the old state. That trade-off is discussed further in What Is Eventual Consistency?.

Production monitoring should track:

• source log position;
• CDC connector lag;
• oldest unprocessed change;
• events produced per second;
• connector restart count;
• serialization failures;
• schema errors;
• consumer lag;
• duplicate processing rate;
• downstream synchronization delay.

Lag should be measured in both records and time. A backlog of 10,000 changes may be insignificant during high traffic, while a single change that has been stuck for an hour can indicate a broken pipeline.

CDC Failure Scenarios

CDC removes the need for repeated full-table synchronization, but it introduces its own failure modes.

The connector crashes. It should resume from a durable offset without silently skipping committed changes.

The broker becomes unavailable. The connector must stop advancing its durable checkpoint until publication is safe, or use equivalent recovery semantics.

A consumer crashes after applying a change. The same event may be processed again, requiring idempotent behavior.

The database removes old log segments too quickly. A connector that remains offline for too long may lose the log history required to resume.

A schema migration breaks serialization. CDC may continue reading the database while failing to publish or deserialize new records.

A downstream consumer falls behind. The CDC connector can remain healthy while the actual destination becomes hours behind the source database.

These failures should be observable independently. Monitoring only whether the CDC process is running is not enough.

When to Use CDC

CDC is a strong option when database changes must continuously propagate to other systems without placing repeated query load on operational tables.

It is particularly useful when:

• large tables need near-real-time synchronization;
• inserts, updates, and deletes must all be captured;
• multiple downstream systems need the same database changes;
• legacy applications cannot easily publish events;
• data must flow into analytics or search systems;
• a migration requires continuous synchronization between databases;
• database changes need to become a durable stream.

CDC is less attractive when a small application only needs occasional synchronization and a simple query can solve the problem reliably.

It is also not automatically the right mechanism for domain events. A database row change describes what changed in storage; a domain event describes what happened in the business. Those are not always the same thing.

Common CDC Mistakes

• Treating database changes as domain events. A row update may not communicate the business reason for the change.
• Assuming exactly-once processing. Connector and consumer failures can create duplicate delivery.
• Ignoring deletes. Synchronization logic must explicitly represent deleted records.
• Using timestamps as perfect offsets. Equal timestamps and transaction boundaries can create correctness problems.
• Ignoring transaction boundaries. Downstream systems should not observe changes from transactions that eventually roll back.
• Ignoring schema evolution. Database migrations can break downstream consumers.
• Assuming ordering survives every stage. Broker partitioning and parallel consumers can change processing order.
• Keeping offsets only in memory. Restarts can cause missing or repeated changes.
• Ignoring transaction-log retention. A long connector outage can make the required log position unavailable.
• Sending every database column downstream. This increases coupling, payload size, and exposure of implementation details.
• Monitoring connector uptime instead of end-to-end lag. A running CDC connector does not guarantee synchronized destinations.

Frequently Asked Questions

CDC is conceptually simple, but production implementations raise important questions around infrastructure, latency, reliability, and the relationship between database changes and application events.

Does CDC Require Kafka?

No. CDC describes how database changes are captured, not which messaging platform transports them.

Kafka is commonly used because an ordered, durable stream fits CDC workloads well, but changes can also be delivered through other brokers, streaming platforms, databases, or custom pipelines.

Does CDC Capture Deletes?

Log-based CDC can normally observe deletes because the database transaction log records the operation even though the row no longer exists in the table.

This is an important advantage over simple timestamp polling, where a deleted row disappears before the polling query can discover it.

Is CDC Real-Time?

CDC is usually near-real-time, not instantaneous.

Latency exists between the database commit, log processing, event publication, broker delivery, consumer processing, and downstream write. Depending on the architecture, that delay may range from milliseconds to seconds or considerably longer during failures or backlogs.

Can CDC Lose Events?

A correctly designed CDC pipeline should preserve committed changes across ordinary failures, but reliability depends on durable offsets, source-log retention, publication semantics, and downstream processing.

Misconfigured log retention, incorrectly committed offsets, destructive schema changes, or operational mistakes can still create data loss. CDC therefore requires the same production discipline as other distributed data pipelines.

Should CDC or Transactional Outbox Be Used?

CDC is a strong fit when the goal is to capture database-level changes for replication, analytics, search indexing, migration, or synchronization.

A Transactional Outbox is usually a better abstraction when the application needs to publish explicit business events such as OrderCreated, PaymentCaptured, or ShipmentCancelled.

The approaches are not mutually exclusive. A common production architecture writes domain events to an outbox table and then uses log-based CDC to publish those rows to the messaging system.

Conclusion

Change Data Capture turns committed database mutations into a continuous stream that other systems can process. Log-based CDC is especially powerful because it uses the database's existing transaction log instead of repeatedly scanning application tables.

CDC can simplify database replication, search indexing, analytics pipelines, migrations, cache synchronization, and event-driven integration, but it introduces distributed-system concerns around offsets, duplicates, ordering, schema evolution, lag, and recovery.

The core principle is: capture changes from a durable source, preserve enough position information to resume safely, and design downstream consumers to tolerate asynchronous and repeated delivery.

More Articles to Read

• What Is Database Replication?
• Event-Driven Architecture in Distributed Systems
• What Is CQRS?