Building a Simple CDC Ingestion Pattern: A Practical Guide

In this article, let us walk through building a simple Change Data Capture (CDC) ingestion pipeline. If you have been only running full-load or batch ETL jobs so far, CDC gives you a way to capture row-level changes — inserts, updates, and deletes — from your source database and send them downstream in near real-time.

We will use Debezium to read the Postgres WAL (write-ahead log), push changes into Kafka, and then land those changes as files in a data lake. The pattern is simple enough to get running on your laptop but covers the moving parts you would encounter in a real project.

Why CDC Instead of Batch Dumps

If your source table is small and doesn’t change much, running a full export every few hours works fine. But once the table grows or you need fresher data downstream, full dumps become expensive. You end up re-reading gigabytes of unchanged rows just to pick up a handful of modifications. CDC solves this by only sending what actually changed.

Some common use cases where CDC fits better:

• Keeping a search index in sync with your operational database
• Feeding audit logs or event-sourcing tables into analytics
• Replicating data across regions without full-table copies
• Streaming transactional data into your data lake without a heavy batch window

That said, CDC adds complexity. You now have to manage a connector, a message broker, and schema evolution. If you only need daily snapshots and your table is under a few million rows, stick with batch.

The Pattern We Will Build

We are going to set up this flow:

1. Postgres — the source database with a table we want to track
2. Debezium — reads the WAL and emits change events in a structured JSON format
3. Kafka — buffers the events so we don’t lose anything
4. A consumer — reads from Kafka and writes the changes to a GCS bucket or local storage

Enough talking. Let us get our hands on it.

Step 1: Set Up Postgres for CDC

First, make sure your Postgres instance has logical replication enabled. Debezium needs this to read the WAL. If you are running Postgres locally or in Docker, check these settings:

# postgresql.conf
wal_level = logical
max_replication_slots = 4
max_wal_senders = 4

Restart Postgres after changing those. Then create a table we can play with:

CREATE TABLE orders (
    id SERIAL PRIMARY KEY,
    customer_name TEXT NOT NULL,
    amount DECIMAL(10, 2),
    status TEXT DEFAULT 'pending',
    created_at TIMESTAMPTZ DEFAULT now(),
    updated_at TIMESTAMPTZ DEFAULT now()
);

Debezium needs a replication user with enough privileges:

CREATE USER debezium WITH PASSWORD 'dbzpass' REPLICATION;
GRANT CONNECT ON DATABASE mydb TO debezium;
GRANT USAGE ON SCHEMA public TO debezium;
GRANT SELECT ON ALL TABLES IN SCHEMA public TO debezium;
ALTER TABLE orders REPLICA IDENTITY FULL;

That last line is important. By default, Postgres only includes the values of the primary key and the changed columns in the WAL for UPDATE statements. Setting REPLICA IDENTITY FULL makes sure Debezium gets the entire row — before and after — for every change. This means you can reconstruct the full row downstream without needing a separate lookup.

The trade-off is that FULL writes more data to the WAL. If your table has wide rows and heavy update traffic, this can increase WAL disk usage noticeably. In that case, you might skip this or use DEFAULT and accept that your before payload will be incomplete for non-key columns.

Step 2: Deploy Kafka and Debezium

The quickest way to get everything up is using Docker Compose. Here is a minimal setup with Kafka, Zookeeper, and Debezium:

version: '3'
services:
  zookeeper:
    image: confluentinc/cp-zookeeper:7.4.0
    environment:
      ZOOKEEPER_CLIENT_PORT: 2181

  kafka:
    image: confluentinc/cp-kafka:7.4.0
    depends_on:
      - zookeeper
    ports:
      - "9092:9092"
    environment:
      KAFKA_ZOOKEEPER_CONNECT: zookeeper:2181
      KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://localhost:9092
      KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 1

  debezium:
    image: debezium/connect:2.4
    depends_on:
      - kafka
    ports:
      - "8083:8083"
    environment:
      BOOTSTRAP_SERVERS: kafka:9092
      CONFIG_STORAGE_TOPIC: debezium_config
      OFFSET_STORAGE_TOPIC: debezium_offsets
      STATUS_STORAGE_TOPIC: debezium_status

Bring it up:

docker-compose up -d

Wait a few seconds for everything to be healthy, then register the Postgres connector:

curl -X POST http://localhost:8083/connectors \
  -H "Content-Type: application/json" \
  -d '{
    "name": "orders-connector",
    "config": {
      "connector.class": "io.debezium.connector.postgresql.PostgresConnector",
      "database.hostname": "host.docker.internal",
      "database.port": "5432",
      "database.user": "debezium",
      "database.password": "dbzpass",
      "database.dbname": "mydb",
      "table.include.list": "public.orders",
      "topic.prefix": "cdc",
      "plugin.name": "pgoutput"
    }
  }'

If everything is wired correctly, Debezium will take an initial snapshot of the orders table and then start listening for changes. You can verify by checking the topic:

docker exec -it kafka kafka-console-consumer \
  --bootstrap-server localhost:9092 \
  --topic cdc.public.orders \
  --from-beginning

You should see JSON messages that look something like this (trimmed for clarity):

{
  "payload": {
    "before": null,
    "after": {
      "id": 1,
      "customer_name": "Ashok",
      "amount": 99.99,
      "status": "pending",
      "created_at": "2025-11-10T15:30:00Z",
      "updated_at": "2025-11-10T15:30:00Z"
    },
    "op": "c",
    "ts_ms": 1721253400000
  }
}

The op field tells you the operation: c for create, u for update, d for delete, and r for the initial snapshot read.

Step 3: Consume and Land the Data

Now that changes are flowing into Kafka, we need something to pull them out and write them somewhere useful. For a simple demo, let us write a Python consumer that reads from the topic and writes each change as a JSON line to a file or GCS.

from kafka import KafkaConsumer
import json
from datetime import datetime

consumer = KafkaConsumer(
    'cdc.public.orders',
    bootstrap_servers='localhost:9092',
    group_id='cdc-landing-consumer',
    value_deserializer=lambda m: json.loads(m.decode('utf-8')),
    auto_offset_reset='earliest'
)

# In production you would batch these and write to GCS/S3
output_path = f"/tmp/cdc_landing/orders_{datetime.utcnow().strftime('%Y%m%d_%H%M%S')}.jsonl"

with open(output_path, 'a') as f:
    for msg in consumer:
        payload = msg.value.get('payload', {})
        record = {
            'operation': payload.get('op'),
            'timestamp': payload.get('ts_ms'),
            'before': payload.get('before'),
            'after': payload.get('after')
        }
        f.write(json.dumps(record) + '\n')
        f.flush()

This is deliberately simple. It appends every message to a local file. In a real setup, you would batch writes, rotate files periodically, and push to cloud storage. But this gives you a working end-to-end flow to test with.

Comparison: CDC Approaches at a Glance

Here is how different CDC patterns stack up for the kind of data work we typically do:

Approach	Latency	Setup Complexity	Best For
Debezium + Kafka	Seconds	Medium	Near real-time, multiple consumers
Postgres logical replication + custom consumer	Seconds	High	Full control, no Kafka dependency
Trigger-based audit table	Sub-second	Low	Simple setups, small tables
Batch diff using watermark column	Hours	Very Low	When CDC is overkill
Fivetran / Stitch / Airbyte	Minutes	Very Low	Teams that don’t want to manage infra

For what it is worth, I have used the trigger-based approach for a small internal project where we only had one table to track and did not want to bring in Kafka. It works but becomes a mess once you have more than a handful of tables.

Practical Limitations and Things to Watch Out For

Schema changes are manual. If you add or drop a column on the source table, Debezium does not automatically handle it. New columns appear in the payload, but your downstream consumer might not know what to do with them. You need a process for this — either a schema registry (like Confluent’s) or a manual review step when the source schema changes.

Kafka retention matters. Debezium records every change. If your Kafka topic has a 7-day retention and your consumer is down for 8 days, you will lose data. Plan your retention period and monitoring accordingly.

The initial snapshot can be heavy. When you first register a connector, Debezium takes a full-table snapshot. On a large table, this can lock rows or spike CPU on your source database. For production, you can use snapshot.mode=exported or take the snapshot from a read replica.

WAL disk growth. If your replication slot is not being consumed (consumer is down or slow), Postgres will keep WAL segments around, and your disk can fill up. Always monitor pg_replication_slots and set alerts on replication lag.

Ordering is per-table, not cross-table. Debezium guarantees order within a single table’s topic, but if you need cross-table ordering (for example, an orders row must be processed before its order_items), you have to handle that in your consumer logic.

What Changes in Production

If this were going into a production environment, here is what I would do differently from the demo above:

• Run Kafka as a cluster, not a single broker. Three brokers minimum so you don’t lose data when one goes down.
• Use a schema registry. Hard-coding assumptions about the payload structure in your consumer will break eventually. Avro + Schema Registry gives you versioned, backward-compatible schemas.
• Don’t write one JSONL file per consumer restart. Batch writes into partitioned paths — something like cdc/orders/year=2025/month=11/day=10/hour=15/ — and rotate every 10–15 minutes or every N records.
• Add dead-letter handling. If a message cannot be processed (schema mismatch, null in a required field), don’t let it block the whole pipeline. Send it to a dead-letter topic and alert on it.
• Monitor replication lag. Set up alerts in your monitoring tool of choice (Datadog, Grafana, whatever you use) on the pg_current_wal_lsn vs confirmed_flush_lsn gap.

Wrapping Up

We walked through a minimal CDC ingestion pattern — from enabling logical replication on Postgres to landing changes as JSON files. The whole thing takes under a hundred lines of config and code, but it covers the core concepts you would deal with in a larger deployment.

The main takeaway is that CDC is not magic. It is just reading the database log and shipping those changes somewhere. Once you understand the moving parts — the WAL, the connector, the broker, and the consumer — it becomes just another pipeline pattern in your toolkit.

If you are evaluating CDC for your team, start small. Get one table working end to end on a dev environment first. The operational surprises — schema drift, lag, WAL bloat — are much easier to handle when you have already seen them in a small setup.