Reliable Event-Driven Microservices with the Transactional Outbox Pattern

A service can commit an order to PostgreSQL and still fail to publish the corresponding event. If the process crashes after the database commit but before the Kafka publish, the order exists but downstream services never hear about it. Reverse the order and a different failure appears: Kafka receives an event for a transaction that later rolls back.

This is the dual-write problem. It is one of the most common reliability gaps in event-driven systems—and retrying the two writes does not make them atomic.

The transactional outbox pattern closes that gap by recording the business change and the event to be published in the same database transaction. A separate relay then forwards committed outbox rows to a message broker. With PostgreSQL, Debezium and Kafka, the relay can read the PostgreSQL write-ahead log instead of repeatedly polling the table.

This guide builds the pattern step by step, explains the failure modes it does and does not solve, and shows how to make consumers idempotent. The central guarantee is deliberately narrow: the database transaction atomically records both the business state and the intent to publish. Delivery to consumers is still typically at least once, so duplicates must be expected and handled.

1. Why two writes are not one transaction

Imagine an order service that stores an order and publishes OrderCreated to Kafka:

order = repository.create_order(command)
producer.send("orders", OrderCreated.from_order(order))

These operations belong to different transactional systems. A PostgreSQL commit cannot atomically commit a Kafka record unless you introduce a distributed transaction protocol supported by every participant and accept its operational costs. A normal application-level sequence has a crash window.

Failure pointDatabase stateBroker stateConsequence Before database commitOrder rolled backNo eventConsistent After database commit, before publishOrder existsNo eventLost notification After publish, before database commitOrder rolled backEvent existsPhantom notification Broker accepted event, producer timed outOrder existsEvent may existRetry may duplicate it

A retry loop can reduce transient failures, but it cannot reliably determine whether a timed-out broker request was committed. The system needs a durable record of publication intent that participates in the same transaction as the business update.

2. The transactional outbox architecture

The service writes two rows in one PostgreSQL transaction: the business row and an outbox row. If either insert fails, both roll back. Once committed, a relay reads the outbox and publishes each event to Kafka.

Command
  |
  v
Order service
  |
  | BEGIN
  |   INSERT order
  |   INSERT outbox event
  | COMMIT
  v
PostgreSQL
  |
  | WAL / logical decoding
  v
Debezium PostgreSQL connector
  |
  | Outbox Event Router SMT
  v
Kafka topic
  |
  +------> Inventory consumer
  |
  +------> Notification consumer

The key boundary is the PostgreSQL commit. A rolled-back transaction produces neither a committed order nor a committed outbox event. A committed transaction leaves both available for the relay, even if the application process exits immediately afterward.

Debezium's Outbox Event Router is a Single Message Transform (SMT) designed to route outbox records. The PostgreSQL connector captures committed row-level changes through logical decoding and streams them through Kafka Connect.

3. Design the outbox table

A useful starting schema is:

CREATE TABLE outbox_events (
    id              UUID PRIMARY KEY,
    aggregatetype   VARCHAR(100) NOT NULL,
    aggregateid     VARCHAR(200) NOT NULL,
    type            VARCHAR(200) NOT NULL,
    payload         JSONB NOT NULL,
    created_at      TIMESTAMPTZ NOT NULL DEFAULT now()
);

CREATE INDEX outbox_events_created_at_idx
    ON outbox_events (created_at);

• id is a globally unique event identifier. Consumers can use it for deduplication.
• aggregatetype identifies the domain aggregate, such as Order.
• aggregateid is the aggregate key. Debezium uses it as the Kafka record key by default, helping keep events for one aggregate in the same partition.
• type names the event, such as OrderCreated.
• payload contains the event contract, not an arbitrary dump of the entire database row.

Keep event payloads intentional and versionable. Include the identifiers and facts consumers need; avoid leaking credentials, payment data, or unrelated personal information. Treat event schemas as public contracts between independently deployed services.

4. Write the business row and event atomically

Here is a simplified Python example using a DB-API-style connection. The exact connection and repository interfaces will differ between psycopg versions and application frameworks, but the transaction boundary is the important part.

from uuid import uuid4
import json

def create_order(conn, customer_id: str, total_cents: int) -> str:
    order_id = str(uuid4())
    event_id = str(uuid4())

    event = {
        "event_id": event_id,
        "event_type": "OrderCreated",
        "order_id": order_id,
        "customer_id": customer_id,
        "total_cents": total_cents,
    }

    # Both inserts use the same connection and transaction.
    with conn.transaction():
        conn.execute(
            """
            INSERT INTO orders (id, customer_id, total_cents, status)
            VALUES (%s, %s, %s, 'created')
            """,
            (order_id, customer_id, total_cents),
        )

        conn.execute(
            """
            INSERT INTO outbox_events
                (id, aggregatetype, aggregateid, type, payload)
            VALUES (%s, 'Order', %s, 'OrderCreated', %s::jsonb)
            """,
            (event_id, order_id, json.dumps(event)),
        )

    return order_id

Do not publish to Kafka inside this transaction and do not commit the order before inserting the outbox row. The database transaction should contain only database work. The relay handles transport independently.

In production, also validate the command, enforce business invariants in the transaction, and use an idempotency key for client retries where duplicate order creation would be harmful. The outbox solves reliable event intent; it does not automatically make the original HTTP command idempotent.

5. Route outbox changes with Debezium

Debezium's PostgreSQL connector reads changes from PostgreSQL logical decoding. Configure PostgreSQL for logical replication and create a replication publication/slot according to the connector's deployment model. The database role needs the appropriate replication and table-read privileges. Consult the connector documentation and your managed PostgreSQL provider's guidance before changing production settings.

A simplified Kafka Connect connector configuration looks like this (adjust connector class, slot, publication, database credentials, topic prefix, and converter settings for your deployment):

{
  "name": "orders-outbox-connector",
  "config": {
    "connector.class": "io.debezium.connector.postgresql.PostgresConnector",
    "database.hostname": "${file:/opt/connect-secrets/db.properties:hostname}",
    "database.port": "5432",
    "database.user": "${file:/opt/connect-secrets/db.properties:username}",
    "database.password": "${file:/opt/connect-secrets/db.properties:password}",
    "database.dbname": "orders",
    "topic.prefix": "commerce",
    "plugin.name": "pgoutput",
    "slot.name": "orders_outbox_slot",
    "publication.name": "orders_outbox_publication",
    "table.include.list": "public.outbox_events",
    "tombstones.on.delete": "false",
    "transforms": "outbox",
    "transforms.outbox.type": "io.debezium.transforms.outbox.EventRouter"
  }
}

This is an illustrative configuration, not a drop-in production manifest. Connector options and required PostgreSQL privileges vary with Debezium and PostgreSQL versions. Store credentials in a secret provider, not in a checked-in connector file. Restrict capture to the outbox table where possible and verify the connector's snapshot and publication behavior before rollout.

With the default Outbox Event Router conventions, the aggregate type contributes to the topic name (for example, outbox.event.Order), the aggregate ID becomes the Kafka key, the payload becomes the message value, and the event ID is included in a header. These defaults can be changed using the SMT configuration options.

6. Understand the delivery guarantee: duplicates are normal

The outbox makes the database write and event record atomic. It does not create a single end-to-end transaction across PostgreSQL, Kafka, and every consumer.

Consider a relay that publishes an event successfully but crashes before its progress is durably recorded. On restart, the event can be delivered again. PostgreSQL logical decoding can also replay recent changes after a crash because replication-slot progress is persisted at checkpoints. PostgreSQL explicitly documents that consumers must tolerate repeated changes in this situation in its logical decoding and replication slot documentation.

Design for at-least-once delivery with idempotent processing. Do not promise exactly-once business effects merely because Kafka producer idempotence or transactions are enabled. Those features address specific Kafka producer/consumer processing boundaries; they do not atomically include an unrelated PostgreSQL transaction and arbitrary external side effects.

7. Make consumers idempotent

A consumer should record the event ID in the same local database transaction as the business effect. A uniqueness constraint turns duplicate deliveries into harmless no-ops.

CREATE TABLE processed_events (
    consumer_name  TEXT NOT NULL,
    event_id       UUID NOT NULL,
    processed_at   TIMESTAMPTZ NOT NULL DEFAULT now(),
    PRIMARY KEY (consumer_name, event_id)
);

def handle_order_created(conn, event: dict) -> None:
    event_id = event["event_id"]

    with conn.transaction():
        inserted = conn.execute(
            """
            INSERT INTO processed_events (consumer_name, event_id)
            VALUES (%s, %s)
            ON CONFLICT DO NOTHING
            RETURNING event_id
            """,
            ("inventory-service", event_id),
        ).fetchone()

        if inserted is None:
            # This consumer has already committed this event's effect.
            return

        reserve_inventory(
            conn,
            order_id=event["order_id"],
            items=event["items"],
        )

The deduplication record and the effect must commit together. If inventory reservation commits but the processed-event marker does not, a retry can reserve inventory twice. If the marker commits first and the effect fails, the event can be lost from the consumer's perspective. One transaction avoids that split for effects stored in the same database.

If handling an event must call another external system—send an email, charge a card, or invoke a third-party API—you need that boundary's own idempotency mechanism or another durable handoff. A database inbox alone cannot make an arbitrary network side effect atomic.

8. Operational concerns that decide whether this works in production

Replication-slot lag and WAL retention

A logical replication slot retains WAL that has not yet been consumed. If Debezium is stopped or falls behind, retained WAL can grow and put pressure on database storage. Monitor replication-slot lag, retained WAL bytes, connector status, and disk headroom. Define alerts and a recovery procedure before enabling CDC on a production database.

Outbox retention and cleanup

Do not delete outbox rows merely because they are old. With CDC, the connector reads the WAL stream; table cleanup and WAL retention are related but separate concerns. Choose a retention policy based on audit requirements, replay needs, backup strategy, and the connector's progress. Partitioning by creation time can make large-table cleanup more manageable, but validate how partition maintenance interacts with the connector and schema.

Ordering and partition keys

Using aggregateid as the Kafka key places events for one aggregate on the same partition, preserving the broker's order within that partition. It does not guarantee global ordering across all aggregates, nor can it fix events produced in the wrong order by application logic. Include aggregate version numbers when consumers need to detect stale or out-of-order updates.

Schema evolution

Version event contracts deliberately. Prefer additive changes that old consumers can tolerate. For breaking changes, publish a new event version or use a schema registry and a compatibility policy. Avoid coupling consumers to internal database table layouts.

Poison events and recovery

Define what happens when a consumer cannot process a validly delivered event: bounded retries, dead-letter handling where appropriate, alerting, and an operator replay procedure. A dead-letter queue is not a substitute for understanding why processing failed. Preserve the event ID and original metadata so replay remains traceable.

Observability

Track the age of the oldest unprocessed event, connector health, replication-slot lag, Kafka publish/consume rates, consumer retry counts, duplicate detection counts, and dead-letter volume. Put the event ID and aggregate ID in structured logs and distributed traces, while avoiding sensitive payload logging.

9. When should you use an outbox?

The pattern is a strong fit when a service owns a relational database and must reliably publish domain events after committing state: orders, payments, inventory, account lifecycle changes, or workflow transitions. CDC is especially useful when polling latency or query load is a concern and the team can operate Kafka Connect and Debezium.

A simpler polling relay can be a better first step for a smaller system. It can claim rows in batches using a locking strategy such as FOR UPDATE SKIP LOCKED, publish them, and mark them as dispatched. That approach requires careful lease, retry, and duplicate handling, but it avoids operating a CDC platform. The right choice depends on throughput, latency, operational maturity, and the need to replay changes.

The outbox is not free. It adds a table, a relay, retention work, monitoring, and a consumer idempotency requirement. It is usually worth that complexity when a lost event would leave business workflows inconsistent and reconciliation is expensive.

10. A practical implementation checklist

• Write the business change and outbox event in one database transaction.
• Give every event a stable unique ID and every aggregate a stable key.
• Define explicit, versioned event payloads; do not publish raw database rows.
• Choose polling or CDC based on operational constraints, not fashion.
• For Debezium, configure and monitor logical replication, publication, slot, connector offsets, and WAL retention.
• Assume duplicate delivery and make each consumer's local effect idempotent.
• Use a separate idempotency strategy for external side effects.
• Plan retention, replay, poison-event handling, schema evolution, and observability.
• Test crashes between every meaningful boundary: database commit, relay publish, offset persistence, consumer effect, and acknowledgment.

Conclusion

The transactional outbox pattern does not make PostgreSQL and Kafka one distributed transaction. It removes the dangerous dual write from the request path by committing business state and publication intent together. Debezium can then stream the committed outbox changes from PostgreSQL's WAL to Kafka, while idempotent consumers protect business effects against redelivery.

That combination is practical and robust when its guarantees are understood: atomic outbox creation, asynchronous at-least-once delivery, and idempotent processing at each consumer boundary. The implementation succeeds not just because the connector is running, but because retention, lag, replay, schema changes, and failure recovery are part of the design from day one.

Further reading

• Debezium: Outbox Event Router
• Debezium: PostgreSQL connector
• PostgreSQL: Logical decoding concepts and replication slots
• PostgreSQL: Logical decoding