Dual Writes and the Outbox Pattern

Your daily reconciliation job flags a problem: dozens of orders exist in your database, but no “order-created” event for them ever reached Kafka. The email service never sent confirmations. The inventory service never reserved stock. After some digging, you find the cause: Kafka had a bad half hour yesterday and rejected every publish. The orders had already been committed by then. The API returned errors, so some customers retried and now have two orders and zero confirmations.

The Dual-Write Problem

Your order service needs to do two things:

1. Save the order to the database
2. Publish an “order-created” event to Kafka

The straightforward approach looks like this:

func CreateOrder(ctx context.Context, order Order) error {
    if err := db.SaveOrder(ctx, order); err != nil {
        return err
    }

    if err := kafka.Publish(ctx, "order-created", order); err != nil {
        // The order is already committed. Returning an error won't undo it,
        // and the caller will probably retry and create a second order.
        // Swallowing the error loses the event. Deleting the order is
        // another dual write with the same problem.
        return err
    }

    return nil
}

Most of the time this works fine. But then your service crashes between those two operations. Or the network blips. Or Kafka is temporarily unavailable. Now you have an order in your database, but the event never went out.

You might think: “Just publish to Kafka first, then save to the database.” But that has the opposite problem: if the database write fails, you’ve just told the world about an order that doesn’t exist.

The root issue is that you’re trying to make two separate systems act as one. Your database and Kafka don’t share a transaction in this setup. Kafka has transactions of its own, but they only span Kafka topics.

A Simple Solution

The transactional outbox pattern (the name comes from Chris Richardson’s pattern catalogue) takes a different approach: instead of publishing directly to Kafka, you write the event to your database.

1. Save the order to the database
2. In the same transaction, save the event to an “outbox” table
3. A separate process reads from the outbox and publishes to Kafka

  flowchart LR
    Service[Order Service] -->|"Single Transaction"| DB[(Database)]
    DB --> Orders[orders]
    DB --> Outbox[outbox]
    Publisher[Publisher] -->|"Poll / mark published"| Outbox
    Publisher -->|"Publish"| Kafka[Kafka]

Your database already knows how to make several writes atomic. Write the order and the outbox row in the same transaction and they either both land or neither does. Publishing to Kafka becomes a separate step, done later by its own process. If that step fails, the row is still sitting in the outbox to retry from.

How It Works

The Outbox Table

You need a table to store events alongside your business data. In PostgreSQL, for example:

CREATE TABLE outbox (
    id VARCHAR PRIMARY KEY,
    aggregate_type VARCHAR NOT NULL,  -- the entity type, e.g. "order", "user", "payment"
    aggregate_id VARCHAR NOT NULL,    -- the entity's ID, e.g. "order-123"
    event_type VARCHAR NOT NULL,      -- what happened, e.g. "order-created", "order-cancelled"
    payload JSONB NOT NULL,           -- the event data
    created_at TIMESTAMPTZ NOT NULL,  -- approximate order, not a sequence
    published_at TIMESTAMPTZ          -- null = not yet published, timestamp = published
);

-- The publisher only ever asks for unpublished rows, so index exactly those.
CREATE INDEX idx_outbox_unpublished ON outbox (created_at)
WHERE published_at IS NULL;

Write Both in One Transaction

When creating an order, you write to both tables atomically:

func (s *OrderService) CreateOrder(ctx context.Context, order Order) error {
    tx, err := s.db.BeginTransaction(ctx)
    if err != nil {
        return err
    }
    defer tx.Rollback(ctx)

    if err := tx.InsertOrder(ctx, order); err != nil {
        return err
    }

    event := OutboxEvent{
        ID:            uuid.New().String(),
        AggregateType: "order",
        AggregateID:   order.ID,
        EventType:     "order-created",
        Payload:       marshalJSON(order),
        CreatedAt:     time.Now(),
    }
    if err := tx.InsertOutboxEvent(ctx, event); err != nil {
        return err
    }

    return tx.Commit(ctx)
}

Publish from the Outbox

A background worker polls for unpublished events and sends them to Kafka:

func (p *OutboxPublisher) publishPending(ctx context.Context) error {
    events, err := p.db.GetUnpublishedEvents(ctx, 100)
    if err != nil {
        return err
    }

    for _, event := range events {
        if err := p.kafka.Publish(ctx, event); err != nil {
            log.Printf("Failed to publish %s: %v", event.ID, err)
            continue
        }

        if err := p.db.MarkEventPublished(ctx, event.ID, time.Now()); err != nil {
            log.Printf("Failed to mark %s as published: %v", event.ID, err)
            // It may be published again on the next poll.
        }
    }

    return nil
}

I usually poll every one to five seconds. That adds a small delay, but you get reliability in return.

Two things this sketch leaves out. Run exactly one publisher, and remember that a rolling deploy briefly runs two. If you need more than one, claim rows with SELECT ... FOR UPDATE SKIP LOCKED or a lease column. And count attempts: as written, continue lets later events overtake a failed one, and a payload that can never be published sits at the head of every batch forever. Stop the batch on the first failure if order matters, and park events that keep failing.

Polling is the simplest to run, but it needs the index, a cleanup job for published rows, and that single publisher. Debezium’s outbox event router reads the database’s write-ahead log instead, which removes those chores (you can even delete the outbox row in the same transaction that inserts it), and adds Kafka Connect and a replication slot to operate.

Duplicates, and When to Skip It

With a durable outbox and a publisher that keeps retrying, this pattern supports at-least-once publishing to Kafka. An event can go out twice when the publisher crashes after sending but before marking, when marking fails, or when two publisher instances pick up the same rows. Consumers need to handle duplicates: track the event IDs you’ve already processed, or write the handler so that running it twice gives the same result as running it once.

Skip the outbox when a downstream can rebuild from the source of truth: analytics, a search index, a cache. Publishing directly and logging failures is enough there. Use it when a lost event leaves a customer or a ledger wrong and nothing reconciles it.

Schema details, ordering, cleanup and a complete publisher are in my learning note on the pattern.

The transactional outbox is a simple idea: if you can’t make two systems atomic, don’t try. Write everything to one system and let a separate process handle the rest. The next time the reconciliation job runs, it has nothing to report.