The Outbox Pattern Is the Only Safe Way to Publish Events After a Database Write

The most dangerous line of code in a microservice is not the one that crashes. It is this one:

order = db.save(order)          # write to Postgres
kafka.publish("orders", order)  # tell the world

It looks correct. It passes every test you write. It works in staging for months. Then one Tuesday your billing service is missing 0.3% of orders, nobody can explain it, and the audit takes three days because there is no error anywhere. The database has the order. Kafka does not. No exception was thrown. No alert fired. The two systems just quietly disagreed, and you found out because a customer did not get charged.

This is the dual-write problem, and it is not a bug you can fix by being careful. It is a property of writing to two systems that do not share a transaction. This post walks through exactly how it breaks, why retries and ordering tricks do not save you, and the two designs that actually do: the transactional outbox and change data capture.

The naive version and why it looks fine

Here is the standard shape of a service that owns some state and needs to tell other services when that state changes.

def place_order(cart):
    order = Order(items=cart.items, total=cart.total, status="placed")
    db.session.add(order)
    db.session.commit()                 # (A) Postgres commit

    kafka.produce("orders", {           # (B) Kafka publish
        "type": "OrderPlaced",
        "order_id": order.id,
        "total": order.total,
    })
    return order

Downstream, the billing service consumes OrderPlaced and charges the card. The shipping service consumes it and reserves inventory. The analytics service counts it. Clean, decoupled, event-driven.

The problem is the gap between (A) and (B). Those are two independent operations against two independent systems. There is no transaction that spans both. Anything that can go wrong between them will eventually go wrong, because at scale “one in a million” happens several times a day.

The four ways it actually fails

Walk through the failure modes one at a time, because each one eliminates a different “just do X” fix.

Failure 1: the process dies between A and B. Commit succeeds. Before the Kafka call returns, the pod gets OOM-killed, the node is drained, or someone ships a deploy. The order exists in Postgres forever. The event was never published. Billing never charges. There is no error to catch because the process that would have caught it is gone.

Failure 2: Kafka is momentarily unavailable. The commit succeeds, then the broker is mid-rebalance or the network blips and the produce call times out. Now you have a choice and both options are wrong:

• Retry the publish. But your process might die during the retry loop (see Failure 1), or the retry itself succeeds after you already returned an error to the user, who retried the whole order.
• Give up and return an error. But the order is already committed. The user sees a failure for an order that actually exists.

Failure 3: you flip the order and write to Kafka first. “Publish then commit” so you never lose an event. Now the opposite happens: you publish OrderPlaced, then the Postgres commit fails on a constraint violation or deadlock. Billing charges a card for an order that does not exist in your database. This is worse than losing events, because now you are emitting lies.

Failure 4: everything succeeds but out of order. Two concurrent updates to the same order commit in one order in Postgres and publish in the opposite order to Kafka, because the publishes race. Downstream sees OrderShipped before OrderPlaced. Consumers that assume causal order silently corrupt their state.

The table below is the uncomfortable summary.

Two-phase commit across Postgres and Kafka technically closes the gap, but in practice it is off the table: Kafka has no usable XA participant, distributed transactions add a coordinator that blocks progress when any participant is slow, and locking a database row for the duration of a broker round trip destroys throughput. Nobody runs 2PC between a database and a message broker on the hot path. The realistic answers are the last two rows.

The core idea: make the event part of the transaction

Every one of those failures comes from the same root cause. The state change and the intent to publish an event live in two systems, and only one of them is transactional. The fix is to move the event into the one place that already gives you atomicity: the database itself.

Instead of publishing to Kafka inside the request, you write the event into an outbox table in the same transaction as the business change. Either both rows commit or neither does. There is no gap.

   Request                          Postgres (one transaction)
   -------                          --------------------------
   place_order()  ----commit---->   orders     row inserted
                                    outbox     row inserted
                                        |
                                        |  (asynchronous, separate process)
                                        v
                              Relay reads unpublished outbox rows
                                        |
                                        v
                                     Kafka topic "orders"

The write path becomes boring, which is exactly what you want:

def place_order(cart):
    with db.session.begin():                     # single transaction
        order = Order(items=cart.items, total=cart.total, status="placed")
        db.session.add(order)
        db.session.flush()                        # get order.id

        db.session.add(Outbox(
            aggregate_type="order",
            aggregate_id=order.id,
            event_type="OrderPlaced",
            payload=json.dumps({
                "order_id": order.id,
                "total": float(order.total),
            }),
        ))
    # commit here: order and outbox row are atomic
    return order

The outbox table is deliberately simple:

CREATE TABLE outbox (
    id             BIGSERIAL PRIMARY KEY,
    aggregate_type TEXT        NOT NULL,
    aggregate_id   TEXT        NOT NULL,
    event_type     TEXT        NOT NULL,
    payload        JSONB       NOT NULL,
    created_at     TIMESTAMPTZ NOT NULL DEFAULT now(),
    published_at   TIMESTAMPTZ
);

-- only unpublished rows, kept small and fast to scan
CREATE INDEX idx_outbox_unpublished
    ON outbox (id) WHERE published_at IS NULL;

Now the request handler never talks to Kafka at all. It cannot half-succeed. If the transaction commits, the event is durably recorded. If it rolls back, there is no event. Failures 1 through 3 are gone by construction, because there is only one system and it is transactional.

Getting the event out: the relay

A separate process, the relay (or message relay, or publisher), reads unpublished rows and pushes them to Kafka. This is where the “at least once” delivery guarantee comes from, and where people get the details wrong.

def relay_loop(batch_size=100):
    while True:
        with db.session.begin():
            rows = db.session.execute(
                text("""
                    SELECT id, event_type, payload
                    FROM outbox
                    WHERE published_at IS NULL
                    ORDER BY id
                    LIMIT :n
                    FOR UPDATE SKIP LOCKED
                """), {"n": batch_size}
            ).fetchall()

            for row in rows:
                kafka.produce(
                    topic="orders",
                    key=str(row.id),              # ordering key, see below
                    value=row.payload,
                )
            kafka.flush()                          # wait for broker acks

            ids = [r.id for r in rows]
            db.session.execute(
                text("UPDATE outbox SET published_at = now() WHERE id = ANY(:ids)"),
                {"ids": ids},
            )

        if not rows:
            time.sleep(0.2)

Three details carry the entire correctness argument:

FOR UPDATE SKIP LOCKED. This lets you run multiple relay instances safely. Each grabs a disjoint batch of rows without blocking on the others. Without SKIP LOCKED, relays serialize on the same rows and you get one slow worker instead of many fast ones.

Publish before marking published. The kafka.flush() blocks until the broker acknowledges. Only then do you set published_at. If the relay crashes after the flush but before the update, those rows get republished on the next run. That is fine, because the guarantee is at-least-once, not exactly-once. Downstream consumers must be idempotent regardless.

Marking is in its own transaction, or the same one. If you mark first and publish second, a crash loses the event, and you are right back to the dual-write problem you were trying to escape. Always publish, confirm, then mark.

The consumer side is not optional

The outbox gives you at-least-once delivery. That means duplicates are guaranteed, not hypothetical. A relay that crashes after flushing republishes. A Kafka rebalance redelivers. If your consumer is not idempotent, you have moved the corruption downstream instead of eliminating it.

The clean way is an inbox table keyed by event id, checked in the same transaction as the side effect.

def handle_order_placed(event):
    with db.session.begin():
        already = db.session.execute(
            text("SELECT 1 FROM processed_events WHERE event_id = :id"),
            {"id": event["order_id"]},
        ).first()
        if already:
            return                                 # duplicate, skip

        charge_card(event["order_id"], event["total"])
        db.session.execute(
            text("INSERT INTO processed_events (event_id) VALUES (:id)"),
            {"id": event["order_id"]},
        )

Use a business-meaningful idempotency key when you have one. A raw offset does not survive topic re-partitioning; the aggregate id and event type do.

Ordering

Downstream consumers often need events for a single entity in the order they happened. OrderShipped must not arrive before OrderPlaced. Kafka only guarantees order within a partition, and partition is chosen by key. So key by the aggregate, not by the outbox row id:

kafka.produce(topic="orders", key=str(row.aggregate_id), value=row.payload)

Now every event for a given order lands in the same partition and is consumed in insertion order. The relay’s ORDER BY id preserves the order events were written, and the shared partition preserves it end to end. Cross-aggregate ordering is not guaranteed, and you almost never actually need it. If you think you do, you probably have an aggregate boundary in the wrong place.

Change data capture: the same idea without the relay

The transactional outbox has one operational cost you cannot ignore: you own the relay. You run it, scale it, monitor its lag, and page someone when it falls behind. Change data capture removes that piece by reading the database’s own write-ahead log.

With CDC, a tool like Debezium tails the Postgres logical replication stream. Every committed row change becomes an event. You still write to the outbox table in the same transaction, but nothing polls it. Debezium sees the insert in the WAL and produces to Kafka for you.

  place_order()  --commit-->  Postgres WAL
                                   |
                          Debezium reads replication slot
                                   |
                                   v
                              Kafka topic

Why keep the outbox table at all if CDC can capture changes to the orders table directly? Because capturing the business table couples your Kafka schema to your database schema. Rename a column and every consumer breaks. The outbox table is a contract: you decide exactly what shape the event has, independent of how the row is stored. This is the “outbox event router” pattern, and it is the reason serious CDC setups still write an outbox row rather than streaming raw table changes.

Here is the honest comparison.

The subtle trap with CDC is the replication slot. If Debezium falls behind or stops, Postgres cannot recycle WAL segments the slot still needs, and disk fills on your primary. An unmonitored dead slot has taken down production databases. A polling relay fails more gently: it just lags, and lag is easy to alert on.

What breaks, in both designs

Neither approach is free, and pretending otherwise is how people get burned.

The outbox table grows without bound. Every event sits there forever unless you delete it. Run a reaper that deletes rows where published_at < now() - interval '3 days', or partition by day and drop old partitions. If you skip this, the partial index eventually stops being small and the relay slows down.

At-least-once is not exactly-once. You will deliver duplicates. If any consumer is not idempotent, the outbox has not saved you, it has just hidden the seam. Budget for the inbox table.

Relay lag is real latency. With a polling relay, events are delayed by up to your poll interval plus batch time. For most systems this is fine. For anything that feels the difference between 50ms and 2s, tune the interval down or move to CDC.

Poison messages. One malformed payload that a consumer cannot process can stall an entire partition. Send it to a dead letter topic after N failures instead of retrying forever.

What to actually do

If you are writing to a database and then telling another system about it, you already have the dual-write problem, whether or not you have noticed the divergence yet. You have noticed it if you have ever run a reconciliation job to find records that “should have” produced an event and did not.

Start with the transactional outbox. It is a table, a single-transaction write, and a small polling loop. You can add it to an existing service in an afternoon, it needs no new infrastructure, and it eliminates lost and phantom events completely. Make every consumer idempotent from day one, because at-least-once means duplicates are not a maybe.

Move to CDC when relay latency starts to matter, when you have many services all needing outbox relays and you would rather run one Debezium cluster, or when the polling load on your primary becomes visible. Keep the outbox table even then, so your event schema stays decoupled from your storage schema, and put a hard alert on replication slot lag before it fills a disk.

The one thing you should never do is the naive version, commit then publish, and hope the gap does not bite. It will. The only question is whether you find out from a monitoring dashboard or from a customer who was never charged.