Most teams that adopt Kafka still poll their database. They stand up topics, wire up consumers, feel good about being event-driven, and then quietly run a cron job every 30 seconds that does SELECT * FROM orders WHERE updated_at > :last_seen to catch anything the event path dropped. That polling loop is a confession. It means nobody trusts the events to be complete, so the database is still the source of truth and Kafka is a cache that might be stale.
The reason the events are incomplete is almost always the same: application code publishes them, and application code is where the gap lives. The service writes a row, then publishes an event, and sometimes it writes the row and never publishes, because the pod died or the broker blinked. So you add a reconciliation poll to cover the misses, and now you have a real-time system with a 30-second fallback bolted to its side.
Debezium removes the reason for the poll. It reads the Postgres write-ahead log directly and turns every committed insert, update, and delete into a Kafka message, with no publish call anywhere in your application. If it is in the database, it is in Kafka. This post is the practical version: how the WAL path actually works, the connector config line by line, what happens to your consumers when the schema changes, and why pairing CDC with the outbox pattern is the thing that finally lets you delete the poll.
Why application-level publishing leaks
Start with the shape almost everyone ships first.
def place_order(cart):
order = Order(items=cart.items, total=cart.total, status="placed")
db.session.add(order)
db.session.commit() # (A) row is durable
kafka.produce("orders", { # (B) tell the world
"type": "OrderPlaced",
"order_id": order.id,
})
return order
Steps (A) and (B) hit two different systems with no shared transaction. The failure surface is the gap between them:
- The process dies after (A), before (B). The order exists forever; the event never fires. No exception, because the code that would catch it is gone.
- The broker is mid-rebalance and (B) times out. The order is already committed, so you cannot roll it back, and retrying from a handler that might also die just moves the problem.
- You flip the order to publish-first, and now a failed commit emits an event for an order that does not exist. You are broadcasting lies.
You cannot fix this by being careful, because it is not a bug in your code. It is a property of writing to two systems that do not share a transaction. The usual patch is the reconciliation poll, which does not fix the leak, it just scans for the damage after the fact. CDC attacks the root: it stops treating “publish an event” as a second write at all.
What the write-ahead log actually gives you
Postgres already solves the exactly-this-happened problem internally. Every change is written to the write-ahead log before it touches the data files, so a crash can be replayed to a consistent state. The WAL is a durable, totally ordered record of every committed change in the database. That is precisely the event stream you were trying to reconstruct from application code.
Debezium does not poll tables. It connects as a replication client, the same interface a Postgres read replica uses, and streams the WAL as it is written. Because the WAL is ordered by commit, the events come out in commit order for free. Because the WAL is the same thing that makes your data durable, there is no change that is in the database but not in the stream.
To turn physical WAL into something a consumer can read, you need logical decoding. Set it on the primary and restart:
# postgresql.conf
wal_level = logical # emit enough to decode row-level changes
max_wal_senders = 10 # replication connections, including Debezium
max_replication_slots = 10 # one slot per Debezium connector
Two concepts carry the rest of the design, and you have to understand both or you will get paged:
Replication slot. A named, persistent bookmark on the server. It records the oldest WAL position the consumer has not yet confirmed. Postgres will not recycle WAL past that position until the slot advances. This is what guarantees Debezium never misses a change across restarts - and it is also the thing that fills your disk if Debezium stops, because the WAL it needs cannot be deleted. The slot is a feature and a loaded gun.
Publication. A Postgres object that declares which tables are in scope. Debezium creates one automatically, or you manage it yourself to control exactly which tables stream.
INSERT/UPDATE/DELETE
|
v
+-------------+ logical +-----------+ +-------+
| Postgres | ---> decoding ---> | Debezium | ---> | Kafka |
| WAL + slot | (pgoutput) | connector | | topic |
+-------------+ +-----------+ +-------+
^ |
| confirms flushed LSN |
+---------------------------------------+
The feedback arrow matters: Debezium periodically tells Postgres the log sequence number (LSN) it has safely handed to Kafka, and only then does the slot advance and the WAL become collectable.
The connector config, line by line
Debezium runs as a connector inside Kafka Connect. You deploy it by POSTing JSON to the Connect REST API. Here is a real Postgres connector, with the fields that actually decide behavior called out.
{
"name": "orders-connector",
"config": {
"connector.class": "io.debezium.connector.postgresql.PostgresConnector",
"database.hostname": "postgres.internal",
"database.port": "5432",
"database.user": "debezium",
"database.password": "${file:/secrets/db:password}",
"database.dbname": "shop",
"plugin.name": "pgoutput",
"slot.name": "debezium_orders",
"publication.name": "dbz_orders_pub",
"topic.prefix": "shop",
"table.include.list": "public.orders,public.outbox",
"snapshot.mode": "initial",
"tombstones.on.delete": "true",
"heartbeat.interval.ms": "10000",
"key.converter": "io.confluent.connect.avro.AvroConverter",
"value.converter": "io.confluent.connect.avro.AvroConverter",
"key.converter.schema.registry.url": "http://schema-registry:8081",
"value.converter.schema.registry.url": "http://schema-registry:8081"
}
}
What each decision buys you:
plugin.name: pgoutputuses the logical decoding plugin that ships inside Postgres itself. There is nothing to compile or install on the database host. Older guides referencewal2jsonordecoderbufs; on any modern Postgres,pgoutputis the default answer.slot.nameandpublication.nameare explicit so you can see them inpg_replication_slotsandpg_publicationand alert on them. Let Debezium auto-generate these and you will not know what to monitor.snapshot.mode: initialmeans the first time the connector starts, it reads the entire current contents of the included tables and emits them asreadevents, then switches to streaming from the WAL. This is how a brand-new consumer gets the full picture instead of only changes from now on. For a table too large to snapshot in one shot, Debezium supports incremental snapshots that run alongside streaming without a long lock.heartbeat.interval.msis not optional in practice. If your captured tables are quiet but the rest of the database is busy, the slot’s confirmed LSN never advances and WAL piles up on an idle connector. The heartbeat forces periodic acknowledgements so the slot keeps moving. Skipping this is the single most common way people fill a disk.tombstones.on.deleteemits a null-valued record after a delete so log-compacted topics actually drop the key.
A single change event looks like this (value, trimmed):
{
"op": "u",
"ts_ms": 1795000000000,
"before": { "id": 42, "status": "placed", "total": 1999 },
"after": { "id": 42, "status": "shipped", "total": 1999 },
"source": { "lsn": 309812736, "txId": 55012, "table": "orders" }
}
op is c create, u update, d delete, r snapshot read. You get the full before and after image, the transaction id, and the LSN. That before block is something application-level publishing almost never bothers to send, and it is what makes CDC consumers able to compute diffs without re-reading the source.
Schema evolution without breaking every consumer
The question that kills naive CDC setups: what happens when someone runs ALTER TABLE orders ADD COLUMN coupon_code TEXT? If your events are raw JSON blobs, the answer is “every consumer finds out at runtime, and the brittle ones crash.” That is not a hypothetical, it is Tuesday.
This is why the Avro converter in the config matters more than it looks. With Avro plus a Schema Registry, every message carries a schema id, not the schema itself, and the registry enforces a compatibility rule before a new schema is allowed to register at all. Set the subject to BACKWARD compatibility and the registry will reject any change that would break existing consumers.
| Change | Backward compatible | What happens |
|---|---|---|
| Add a column with a default | Yes | Old consumers ignore the new field; new ones read it |
| Add a column with no default | No | Rejected at registry; deploy fails fast, not at runtime |
| Drop a column | Yes (backward) | Old consumers still find the fields they need |
| Rename a column | No | Registry rejects; forces an explicit add-then-drop migration |
| Narrow a type (int to smallint) | No | Rejected; would overflow existing readers |
The important shift is where the failure moves to. Without a registry, a bad schema change is discovered in production by whichever consumer deserializes next. With a registry on BACKWARD mode, the incompatible change is rejected when the connector tries to register it, which surfaces as a connector failure in your deploy pipeline, where you want it. You trade a 3am page for a red build.
Avro is also just smaller on the wire than JSON, because the field names live once in the registry instead of in every message. On a high-volume orders stream that is a real bandwidth and storage win, but the compatibility enforcement is the reason to do it.
The outbox event router: CDC done right
Here is the trap people walk into after they get Debezium working: they point it straight at the orders table and call the resulting stream their public event contract. Now the shape of a Kafka message is whatever the table columns happen to be. Rename total to total_amount for a perfectly good internal reason and every downstream service breaks, because you accidentally made your storage schema your API.
The fix is to not capture business tables at all. Capture an outbox table whose only job is to hold events in the shape you want to publish, written in the same transaction as the business change.
def place_order(cart):
with db.session.begin(): # one transaction
order = Order(items=cart.items, total=cart.total, status="placed")
db.session.add(order)
db.session.flush() # get order.id
db.session.add(Outbox(
aggregate_type="order",
aggregate_id=str(order.id),
event_type="OrderPlaced",
payload={"order_id": order.id, "total": float(order.total)},
))
# commit: the order row and the outbox row are atomic
return order
Because the business row and the outbox row commit together, there is no gap and no lost event - the dual-write problem is gone by construction, not patched by a reconciliation poll. Then Debezium captures the outbox table from the WAL, and Debezium’s built-in outbox event router (a single-message transform) reshapes those raw insert events into clean, per-aggregate topics:
{
"transforms": "outbox",
"transforms.outbox.type":
"io.debezium.transforms.outbox.EventRouter",
"transforms.outbox.route.by.field": "aggregate_type",
"transforms.outbox.route.topic.replacement": "shop.events.${routedByValue}",
"transforms.outbox.table.field.event.key": "aggregate_id",
"transforms.outbox.table.field.event.payload": "payload"
}
place_order() --commit--> Postgres WAL (orders row + outbox row, atomic)
|
Debezium reads replication slot
|
EventRouter SMT reshapes by aggregate_type
|
+---------------+----------------+
v v
topic shop.events.order topic shop.events.customer
Three things fall out of this that you get almost for free:
- The event schema is the
payloadfield, which you control explicitly. Refactor theorderstable however you like; the contract does not move. aggregate_idbecomes the Kafka message key, so every event for one order lands in the same partition and is consumed in commit order.OrderShippedcannot overtakeOrderPlaced.- The write path has zero Kafka calls. The handler cannot half-succeed, cannot block on the broker, and cannot emit a phantom event. It writes two rows and commits.
This is the combination that lets you finally delete the reconciliation poll. The outbox guarantees the event is recorded atomically with the state change; CDC guarantees the recorded event reaches Kafka. Neither half leaks.
At-least-once, so consumers must be idempotent
Debezium delivers at least once, not exactly once. If the connector crashes after handing a batch to Kafka but before it confirms the flushed LSN, it restarts from the last confirmed position and re-emits those events. A Kafka consumer rebalance redelivers. This is correct and unavoidable; the slot bookmark is coarse on purpose.
So duplicates are guaranteed, and a consumer that is not idempotent has simply moved the corruption one hop downstream. The clean guard is a processed-events table checked in the same transaction as the side effect.
def handle_order_placed(event):
with db.session.begin():
seen = db.session.execute(
text("SELECT 1 FROM processed_events WHERE event_id = :id"),
{"id": event["order_id"]},
).first()
if seen:
return # duplicate, skip
charge_card(event["order_id"], event["total"])
db.session.execute(
text("INSERT INTO processed_events (event_id) VALUES (:id)"),
{"id": event["order_id"]},
)
Key the dedup on a business id, not the raw Kafka offset. Offsets do not survive a topic re-partition; the aggregate id does.
The failure mode that will actually page you
Everything above is the happy path. The thing that takes down production is the replication slot, so give it its own attention. When a Debezium connector stops - crashed task, a consumer that stopped confirming, a Connect worker that got OOM-killed - the slot stays put. Its confirmed LSN does not advance. Postgres refuses to recycle any WAL newer than that LSN, because a replication client still needs it. WAL accumulates on the primary’s disk. If nobody notices, the primary runs out of disk and the whole database goes down, taking every application with it, not just the CDC pipeline.
Monitor the slot directly and alert well before it matters:
SELECT slot_name,
active,
pg_size_pretty(
pg_wal_lsn_diff(pg_current_wal_lsn(), confirmed_flush_lsn)
) AS retained_wal
FROM pg_replication_slots;
Alert on active = false on any slot you expect to be live, and on retained_wal crossing a threshold measured against free disk, not an absolute number. A slot that is a few hundred MB behind is normal under load; one that climbs steadily and never falls is a dead connector quietly eating your disk. This is the one Debezium number that is non-negotiable to watch.
The second trap is dropping a connector and forgetting its slot. Deleting the connector does not delete the slot. The orphaned slot keeps pinning WAL forever. When you decommission a connector, SELECT pg_drop_replication_slot('slot_name') by hand, or you have left the gun loaded with no one holding it.
What works, what does not, what to do
What genuinely works: CDC moves event publishing out of application code and into the database’s own durability machinery, so “it is in the database but not in Kafka” stops being a reachable state. The before/after images, commit ordering, and full snapshot-then-stream are things you would never bother to build by hand, and you get them by deploying one connector. Paired with an outbox table, it closes the dual-write hole completely and lets you delete the reconciliation poll that was papering over it.
What does not come for free: you are now running Kafka Connect and a Schema Registry, which is real operational surface. The replication slot is a genuine footgun that has taken down primaries. CDC gives you at-least-once, so every consumer still has to be idempotent, and the outbox table still needs a reaper or it grows without bound. None of this is hard, but pretending it is zero-cost is how teams get surprised.
What to actually do. If you already have Postgres and Kafka and your events are published from application code, you have the dual-write problem whether or not you have seen the divergence yet - the reconciliation poll is the proof you have. Add an outbox table and write to it in the same transaction as your business change; that alone eliminates lost and phantom events. Then put Debezium on the outbox with the event router SMT, Avro plus a registry on BACKWARD compatibility, and a hard alert on replication slot lag before you route a single real consumer through it. Do those three things and the 30-second poll becomes dead code you get to delete - which is the whole point.
Comments