Idempotency Keys and the Transactional Outbox

Here's a scenario that has caused real money to go missing at more than one company. A mobile client sends POST /payments. The server charges the card, writes to the database, and sends back 201 Created. On the way back, the phone switches from Wi-Fi to cellular and the response is lost. The client never saw a success, so it does the reasonable thing and retries.

Without protection, the customer has now been charged twice.

The uncomfortable truth about distributed systems is that a request that failed and a request whose response was lost look exactly the same to the caller. Retries are therefore unavoidable, and every operation that changes state has to be safe to repeat. This post covers the two patterns I use most for that: idempotency keys at the API edge, and the transactional outbox for events leaving the service.

Part 1: Idempotency keys

An operation is idempotent if doing it twice has the same effect as doing it once. GET and PUT are idempotent by definition in HTTP. POST isn't, so we have to make it idempotent ourselves.

The idea, made popular by payment APIs like Stripe's, is simple. The client generates a unique key (a UUID works) for each logical operation and sends it in a header:

POST /payments HTTP/1.1
Idempotency-Key: 5f1c9a1e-8d2b-4c7e-9a4f-2b7d0c3e6a11
Content-Type: application/json

{"amount": 4999, "currency": "USD", "source": "card_123"}

Every retry of the same payment reuses the same key. The server's job is to guarantee that a given key is processed at most once, and that every retry gets back the original response.

Storing keys

A table with a unique constraint does most of the work:

CREATE TABLE idempotency_keys (
    key           TEXT        NOT NULL,
    client_id     TEXT        NOT NULL,
    request_hash  TEXT        NOT NULL,
    status        TEXT        NOT NULL DEFAULT 'in_progress',
    response_code INT,
    response_body JSONB,
    created_at    TIMESTAMPTZ NOT NULL DEFAULT now(),
    PRIMARY KEY (client_id, key)
);

Scoping keys by client_id stops one client's keys from colliding with another's. request_hash lets you detect a client reusing a key with a different body, which is a client bug. The right response to that is an error, not quietly replaying the old response.

The flow

Claim the key. Try to insert it:
```
INSERT INTO idempotency_keys (key, client_id, request_hash)
VALUES ($1, $2, $3)
ON CONFLICT (client_id, key) DO NOTHING;
```
If a row was inserted, this is a new request, so go to step 2. If nothing was inserted, the key already exists. Load that row: if it's completed, return the stored response. If it's still in_progress, another attempt is running right now, so return 409 Conflict and let the client retry shortly.
Do the work. Charge the card, write the payment, whatever the operation is.
Store the result. Update the row to completed with the response code and body, then return that response.

Ideally, steps 2 and 3 happen in the same database transaction as the business write, so that "payment recorded" and "key completed" can never disagree.

The hard part: side effects outside your database

A card charge happens at a payment provider, not in your database, so it can't take part in your transaction. If your service crashes after charging the card but before recording the result, the key is stuck at in_progress and you don't know whether the charge happened.

There are two ways to deal with that:

Pass the idempotency down the chain. Most payment providers accept their own idempotency key. Use a key derived from yours, so a retry of your operation becomes a safe retry of theirs.
Recover stuck keys. A background job finds keys that have been in_progress too long, asks the provider what actually happened, and finishes or rolls back.

Keys should expire eventually, after something like 24 hours. That's long enough to cover any sensible retry window without keeping every key forever.

Part 2: The dual-write problem

The second problem shows up as soon as your service publishes events. After creating a payment, you want to publish PaymentCreated to Kafka or a similar broker so other services can react. The naive code:

if err := repo.CreatePayment(ctx, tx, p); err != nil {
	return err
}
if err := tx.Commit(); err != nil {
	return err
}
// What if we crash right here?
return broker.Publish(ctx, "payments", PaymentCreated{ID: p.ID})

This is a dual write: two systems updated without a shared transaction. If the process dies between commit and publish, the payment exists but nobody downstream ever hears about it. Swap the order and you can publish an event for a payment that then fails to commit. No ordering of the two calls fixes it.

Part 3: The transactional outbox

The fix is to stop writing to two systems. Write the event into your own database, in the same transaction as the business data, and let a separate process move it to the broker.

CREATE TABLE outbox (
    id           BIGSERIAL   PRIMARY KEY,
    aggregate_id TEXT        NOT NULL,
    event_type   TEXT        NOT NULL,
    payload      JSONB       NOT NULL,
    created_at   TIMESTAMPTZ NOT NULL DEFAULT now(),
    published_at TIMESTAMPTZ
);

tx, err := db.BeginTx(ctx, nil)
if err != nil {
	return err
}
defer tx.Rollback() // no-op after a successful Commit

if err := repo.CreatePayment(ctx, tx, p); err != nil {
	return err
}
if err := outbox.Add(ctx, tx, "PaymentCreated", p.ID, payload); err != nil {
	return err
}
return tx.Commit()

Now either both the payment and its event are saved, or neither is. The atomicity comes from the database, which is very good at it.

The relay

A relay process reads unpublished rows, publishes them, and marks them done:

SELECT id, event_type, payload
FROM outbox
WHERE published_at IS NULL
ORDER BY id
LIMIT 100
FOR UPDATE SKIP LOCKED;

FOR UPDATE SKIP LOCKED lets you run several relay instances safely. Each one locks the rows it picks up, and the others skip past those locks instead of waiting on them. Do the select, publish and UPDATE outbox SET published_at = now() WHERE id = ... inside one transaction per batch.

The alternative to polling is change data capture: a tool like Debezium reads the database's write-ahead log and streams new outbox rows to the broker. It adds infrastructure but removes both the polling and the relay code. I start with polling and switch to CDC when the latency or load actually justifies it.

Part 4: At-least-once, so consumers must be idempotent too

Now for the catch. The relay can crash after publishing but before marking the row as published. When it restarts, it publishes the same event again. The outbox gives you at-least-once delivery, not exactly-once.

That's not a flaw in the outbox. It's the best any system can do across a network. "Exactly-once" in practice means at-least-once delivery combined with idempotent processing. So the consumer needs the same discipline as the API:

CREATE TABLE processed_events (
    event_id     TEXT PRIMARY KEY,
    processed_at TIMESTAMPTZ NOT NULL DEFAULT now()
);

In the consumer's transaction, insert the event ID with ON CONFLICT DO NOTHING. If no row was inserted, you've already handled the event, so skip it. Otherwise do the work and commit both together. Include a stable event ID in every outbox payload so this is possible. The outbox row's id works well.

The whole picture

Put together, a request flows through the system like this:

The client sends a POST with an idempotency key. Retries reuse it.
The API claims the key, does the work, and writes the business row, the outbox event and the completed key in one transaction.
The relay publishes outbox rows at least once.
Consumers dedupe by event ID and process each event effectively once.

Each step assumes the step before might repeat itself, and each one is built to handle that. That's what makes it reliable: not preventing duplicates, which you can't, but making them harmless.

Want to talk distributed systems? Drop me a message on the contact page.