Idempotency Keys Are How You Make Retries Safe in Distributed Systems

The network will lie to you. A client sends POST /charges, the server charges the card, and then the response is lost to a dropped connection, a load balancer timeout, or a client that gave up at 30 seconds. The client has no idea whether the charge happened. So it retries. Now you have charged the customer twice, and no amount of careful transaction code on the server prevented it, because the duplicate came from outside your database.

This is the core problem. In any distributed system, the caller cannot tell the difference between “my request failed” and “my request succeeded but the response was lost.” The only safe assumption is that any request can and will be sent more than once. Idempotency keys are how you make that safe. Get them slightly wrong and they give you false confidence while still charging people twice.

Idempotent versus safe versus retriable

People conflate three things. A method is safe if it has no side effects (GET). A method is idempotent if sending it N times has the same effect as sending it once. PUT /users/42 {name: "Ada"} is naturally idempotent: set the name to Ada however many times, the result is the same. DELETE is idempotent too.

The problem is POST. Creating a charge, placing an order, transferring money - these are not naturally idempotent. Two POST /charges calls mean two charges. That is correct HTTP semantics and exactly what you do not want when the second call is a retry of the first.

An idempotency key makes a non-idempotent operation idempotent by giving the retry a way to say “this is the same logical operation as before, do not do it again, just tell me what happened the first time.” The client generates a unique key (a UUID is fine), sends it in a header, and reuses the exact same key on every retry of that one logical request.

POST /v1/charges HTTP/1.1
Idempotency-Key: 9f1c2b4a-8d3e-4f21-a0b7-1e6c5d4a3b2f
Content-Type: application/json

{"amount": 4999, "currency": "usd", "source": "card_xxx"}

The naive implementation, and where it breaks

The first version everyone writes looks like this. Check if the key exists. If it does, return early. If not, do the work.

def charge(key, payload):
    if store.exists(key):
        return store.get_response(key)      # already done
    result = process_charge(payload)         # do the work
    store.put(key, result)                   # remember it
    return result

This is broken in at least four ways, and every one of them shows up in production.

Race condition on concurrent retries. A client times out and retries while the first request is still running. Both requests call store.exists(key), both see nothing, both call process_charge. You just did the work twice. Retries are not spread out politely; they often arrive within milliseconds of each other because the same timeout fired on the same client.

Crash between doing the work and saving the key. process_charge succeeds, then the process dies before store.put. The card was charged but no key was recorded. The retry finds nothing and charges again. The window is small, but at scale small windows happen thousands of times a day.

No request fingerprint. A client reuses a key it already used, but with a different payload - maybe a bug, maybe a caller that recycles keys. The naive code returns the old response and silently ignores the new, different request. Or worse, if you only check existence loosely, it processes the new payload under an old key. Either way the behavior is wrong and invisible.

No notion of “in progress.” There is no state for “I have seen this key but have not finished yet.” So a retry that arrives mid-flight has no correct answer to return, and the code above has no way to represent that.

The design that actually works

The fix is to treat the idempotency key as a row in your database with a lifecycle, protected by a uniqueness constraint, and to record the intent before doing the work rather than after.

The key insight: the uniqueness constraint in the database is what makes this race-free, not your application-level check. Application checks have a time-of-check-to-time-of-use gap. A unique index does not.

CREATE TABLE idempotency_keys (
    key             TEXT PRIMARY KEY,
    request_hash    TEXT NOT NULL,        -- fingerprint of the request
    status          TEXT NOT NULL,        -- 'in_progress' | 'completed'
    response_code   INT,
    response_body   JSONB,
    resource_id     TEXT,                 -- e.g. the charge id created
    created_at      TIMESTAMPTZ NOT NULL DEFAULT now(),
    locked_at       TIMESTAMPTZ,
    expires_at      TIMESTAMPTZ NOT NULL
);

The flow becomes: claim the key atomically, and let the database reject the duplicate.

def charge(key, payload):
    request_hash = sha256_canonical(payload)

    # 1. Claim the key. The unique constraint arbitrates the race.
    try:
        db.execute(
            "INSERT INTO idempotency_keys (key, request_hash, status, expires_at) "
            "VALUES (%s, %s, 'in_progress', now() + interval '24 hours')",
            [key, request_hash],
        )
    except UniqueViolation:
        return handle_existing_key(key, request_hash)

    # 2. We won the race. Do the work exactly once.
    try:
        result = process_charge(payload)
    except Exception:
        # Release so a genuine retry can proceed, or mark failed. See below.
        db.execute("DELETE FROM idempotency_keys WHERE key = %s AND status = 'in_progress'", [key])
        raise

    # 3. Persist the outcome in the SAME transaction as the side effect.
    db.execute(
        "UPDATE idempotency_keys SET status='completed', response_code=%s, "
        "response_body=%s, resource_id=%s WHERE key=%s",
        [200, result.body, result.charge_id, key],
    )
    return result

The handle_existing_key path is where correctness lives.

def handle_existing_key(key, request_hash):
    row = db.get("SELECT * FROM idempotency_keys WHERE key = %s", [key])

    # Same key, different request body: the client is misusing the key.
    if row.request_hash != request_hash:
        raise HttpError(422, "Idempotency key reused with a different payload")

    if row.status == 'completed':
        return Response(row.response_code, row.response_body)   # replay the original

    # status == 'in_progress': the first request has not finished yet.
    raise HttpError(409, "A request with this key is still being processed. Retry shortly.")

Three behaviors fall out of this that the naive version could not express:

• Concurrent duplicate: loses the INSERT race, sees in_progress, gets a 409. It does not do the work. The client backs off and retries, eventually getting the replayed 200.
• Late duplicate: sees completed, gets the exact original response replayed, including the same charge_id.
• Key reuse with a different body: gets a 422 instead of a silently wrong result.

Why the side effect and the record must share a transaction

Step 3 above is the part people skip, and it reintroduces the crash window. If process_charge writes to your own ledger, then the update to idempotency_keys should be in the same database transaction as the ledger write. Commit both or neither. Then a crash can never leave “money moved but key not recorded.”

The hard case is when the side effect is an external call you do not control - charging a card at Stripe, sending a wire, calling a partner API. You cannot make an external HTTP call part of your Postgres transaction. This is where people discover that idempotency is not one key but a chain of them.

The pattern that works: pre-write the intent, then make the external call idempotent too. Before calling the external provider, commit a row saying “I am about to charge with provider key K.” Use a deterministic K derived from your own record, and pass that as the provider’s own idempotency key. Now if you crash after the provider charged but before you recorded success, recovery reads the pre-written intent, sees key K, and retries the provider call with the same K. The provider dedupes on its side and returns the original charge instead of making a new one.

# Deterministic provider key so a retry after a crash reuses it.
provider_key = f"charge-{internal_charge_id}"
db.execute("UPDATE idempotency_keys SET resource_id=%s WHERE key=%s",
           [internal_charge_id, key])                 # pre-write, commit

resp = stripe.Charge.create(                            # external, non-transactional
    amount=payload["amount"], currency=payload["currency"],
    source=payload["source"],
    idempotency_key=provider_key,                       # SAME on every retry
)

Every trustworthy payment API supports an idempotency key on writes for exactly this reason. If a downstream service does not, you cannot make the boundary exactly-once, and you must fall back to reconciliation: record what you attempted, and later compare against the provider’s statement to detect and repair drift.

At-least-once delivery is not exactly-once processing

This is the misconception that makes people think they are done when they are not. Message queues, webhook senders, retrying HTTP clients, and service meshes almost universally give you at-least-once delivery. They guarantee a message arrives, and to guarantee that in the face of lost acks, they will sometimes deliver it more than once. There is no queue that gives you true exactly-once delivery over an unreliable network - it is provably impossible, because the sender can never be certain its delivery was received.

What you can build is exactly-once processing, also called effectively-once. You accept that duplicates will arrive, and you make processing a duplicate a no-op. The mechanism is deduplication on a key, which is the same idempotency machinery applied at the consumer.

at-least-once delivery  +  idempotent consumer  =  effectively-once processing

So-called “exactly-once” features in systems like Kafka are really this: at-least-once delivery plus a deduplication mechanism (producer IDs and sequence numbers, transactional offsets) that discards the duplicates. The network still delivers more than once. The system just refuses to act twice. Once you internalize that no infrastructure removes duplicates from the wire, you stop looking for the magic setting and start writing idempotent consumers. That is the whole game.

  ┌──────────┐   at-least-once    ┌────────────────────┐
  │ Producer │ ─────────────────> │ Queue / Webhook /  │
  └──────────┘   (may duplicate)  │ Retrying client    │
                                   └─────────┬──────────┘
                                             │  same message
                                             │  possibly twice
                                             v
                                   ┌────────────────────┐
                                   │ Consumer           │
                                   │  dedup on key ────────> seen? drop.
                                   │  else process once │
                                   └────────────────────┘

For consumers, the “key” is often carried by the message itself. A payment event carries the payment id. An order event carries the order id plus an event sequence. Dedupe on that. If the producer does not give you a stable id, generate one deterministically from the message content, but be careful: content hashing means two legitimately identical messages (the same user clicking “like” twice on purpose) collapse into one, which may or may not be what you want.

The edge cases that break naive key designs

The happy path is easy. These are the cases that separate a real implementation from one that looks right in a demo.

A few of these deserve detail.

Stuck in_progress. If a process crashes after claiming a key but before completing, that row sits in in_progress forever and every retry gets a 409 forever. Treat the claim as a lease. Store locked_at, and let a request that finds an in_progress row older than, say, 60 seconds reclaim it, on the assumption the original worker is dead. This must itself be atomic - a conditional update guarded by locked_at - or two reclaimers race.

Failure is not the same as in-progress. If the operation genuinely failed (card declined, validation error), you have a choice. You can store the failure as a completed outcome so the retry replays the same error deterministically, or you can free the key so the client can genuinely try again. For deterministic errors (declined card, bad input) replaying is correct. For transient errors (a timeout talking to the provider, where you do not know if it went through) freeing the key is dangerous, because the “failed” call might have actually succeeded. In that gray zone, do not free the key blindly; pre-write intent and reconcile, as above.

Expiry that is too short. Keys cannot live forever, or the table grows without bound. But if you expire a key in one hour and a client’s retry logic backs off and retries after two hours, the key is gone and the retry is treated as a brand new request. Set expiry comfortably longer than the longest possible client retry window. Twenty-four hours is a common, safe default; some payment providers keep keys for 24 hours to a few days. Reap expired rows with a background job or a partitioned table you drop by day.

One key per operation. The most common client-side bug is reusing a single idempotency key for multiple distinct operations - for example, generating it once at app startup, or once per user session. The second real operation gets deduplicated against the first and silently vanishes. Document loudly that a key identifies one logical request, and that clients must generate a fresh key per operation and reuse it only across retries of that same operation.

Scope, and what the key covers

A subtle correctness question: what is the key scoped to? An idempotency key should be unique within an account or API consumer, not globally guessable, and it should be tied to the operation it was created for. If key abc from customer A could collide with key abc from customer B, one customer’s retry could return another’s data. Make the effective key (account_id, idempotency_key). Most APIs also namespace by endpoint so the same key on POST /charges and POST /refunds never collide.

Do not let clients treat the key as a request cache across different operations. It is a deduplication token for one operation, not a general-purpose “give me the last response” mechanism.

What works, what does not, what to actually do

What works: the combination of a client-generated key, a database uniqueness constraint that arbitrates the race, a request fingerprint, an explicit in_progress state with a lease, and storing the outcome in the same transaction as the side effect. That set genuinely turns unreliable at-least-once plumbing into effectively-once behavior for operations you fully control.

What does not work: an application-level “check then act” without a unique constraint (races), saving the key after the side effect instead of with it (crash window), skipping the request fingerprint (silent wrong replays), and assuming your queue’s “exactly-once” flag means duplicates never reach your code (they do). And no amount of idempotency machinery makes an irreversible external call safe on its own; that always needs a deterministic downstream key plus reconciliation.

What to actually do: if you have any endpoint that moves money or mutates critical state, require an Idempotency-Key header on it today, back it with a unique constraint, store a fingerprint, model in_progress as a leased state, and expire keys well past your clients’ retry windows. Make every queue consumer dedupe on a stable message id from day one, because at-least-once means duplicates are not a possibility you might hit - they are a certainty you have not noticed yet. The teams that get burned are not the ones without idempotency keys. They are the ones who added the header, skipped the request fingerprint and the transaction, and believed they were done.