Design a Wire Transfer API - System Design

A wire transfer looks like the wallet transfer’s twin, and interviewers love it because it is not. In a wallet, “move 500 from Alice to Bob” is one UPDATE inside one ACID transaction because both accounts live in your database. A wire moves money between two different banks. The debit lives in your ledger; the credit lives in a bank you cannot see, reach transactionally, or roll back. There is no shared transaction. There is an external rail (Fedwire, RTGS, SWIFT, ACH) that is slow, at-least-once, and - once it settles - irreversible. You cannot ROLLBACK a wire. You can only send a second, compensating wire and hope.

That single fact reshapes the whole design. The wallet problem is “make one ACID transaction correct.” The wire problem is “orchestrate an irreversible, multi-hour, cross-institution money movement that must be exactly-once, must pass sanctions and fraud screening before it leaves, must survive any crash without losing or duplicating a payment, and must be provable to a regulator years later.” The core invariant: money must leave exactly once, only after it is cleared to leave, and every state it passed through must be permanently recorded. Let me build it properly.

Functional Requirements (FR)

In scope:

• Initiate a wire. A customer (or an internal system) requests a transfer of a specific amount and currency from a source account to a beneficiary at another bank, identified by routing number / IBAN / SWIFT BIC.
• Debit-hold then settle. Reserve funds from the source account immediately (so they cannot be spent twice), send the payment instruction over the rail, and only mark the money truly gone once the rail confirms settlement.
• Compliance screening in the critical path. Every wire is screened against sanctions lists (OFAC, UN, EU), AML rules, and a fraud model before it is released. A hit holds or blocks the wire; it does not leave first and ask questions later.
• Idempotent initiation. A client that submits the same wire twice (timeout, retry, double-click) sends money once.
• Status tracking and lifecycle. The customer can see where a wire is: initiated, screening, sent, settled, returned, or failed. Terminal states are final.
• Returns and recalls. Handle the beneficiary bank returning a wire (bad account) or a recall request. Money comes back as a new inbound transfer, never as an edit of the original.
• Regulatory audit trail. Every state change, every screening decision, every operator action is recorded immutably and retained for years, reproducible on demand.

Explicitly out of scope (state it so you own the scope):

• The rail internals. I integrate with Fedwire / SWIFT / an RTGS gateway as a black box with a message API and a settlement confirmation. I do not design the central bank’s settlement engine.
• The KYC / onboarding and the sanctions-list data pipeline. I consume a screening decision and a sanctions dataset as inputs; I do not build the list-ingestion or identity-verification systems.
• FX conversion. Cross-currency wires need an FX rate and a second ledger; I name it but design a single-currency corridor here.
• Customer-facing UI and notifications. I expose an API and emit events; the app and email/SMS are downstream consumers.

The decision that drives everything: the external leg is irreversible and non-transactional, so the whole wire is a persisted state machine driven by a saga, not a database transaction. Correctness comes from ordering (screen, then reserve, then send, then confirm), idempotency at every boundary, and an immutable event log - not from ACID spanning two banks, which is impossible.

Non-Functional Requirements (NFR)

• Scale: 1M transfers/day. That is only ~12/sec average, ~60-120/sec at peak (business hours, month-end payroll runs, quarter-end). This is a low-throughput, high-value system. The hard part is never QPS; it is correctness, latency of the compliance path, and durability.
• Latency: the synchronous part the customer waits on (validate, reserve, screen, accept) should return p99 under ~2s. The rail settlement itself is minutes to hours and is inherently async; we return “accepted, settling” and update status later.
• Availability: 99.99% on initiation and status. But money movement degrades safely: if the rail is down we queue and hold, we never drop or silently fail a wire.
• Consistency: strong for the debit hold and ledger. A source account can never be double-debited or go negative from a race. The wire state machine is strictly ordered and monotonic. We do not trade consistency for availability on the money path.
• Durability: absolute, zero data loss. Once a wire is accepted, it survives any node loss. Every accepted instruction and every state transition is synchronously replicated and never hard-deleted. A lost “sent” record could mean a duplicate wire or an unaccounted-for one - both are incidents.
• Auditability: every wire is reconstructable end to end, years later: who initiated it, what the screening decided and why, when it was sent, the rail reference, when it settled. This is a legal requirement (BSA, recordkeeping rules), not a nice-to-have.

The tension: money movement over an irreversible external rail wants perfect ordering and durability, which fight throughput and latency - but throughput here is tiny, so we spend it all on correctness. We keep a small strongly-consistent core (the reserve + state transition), gate release on screening, drive the rail with an idempotent saga, and record everything to an append-only log.

Back-of-the-Envelope Estimation (BoE)

Real numbers, because they tell us this is a correctness problem, not a scale problem.

Write volume (initiations):

1,000,000 transfers/day
= 1,000,000 / 86,400 sec
≈ 11.6 initiations/sec average
Peak (business hours are ~8h of the day, plus month-end bursts):
  daily volume lands in ~8h => 1M / 28,800 ≈ 35/sec business-hours avg
  month-end / payroll burst ~3x => ~100/sec peak

Each wire generates many state transitions and events (initiated, screening_started, screening_cleared, reserved, sent, ack, settled…), say ~8-12 events per wire.

1M wires * ~10 events = 10M state-transition events/day
≈ 115 events/sec average, ~350/sec peak

Still trivial throughput. A single well-provisioned SQL primary handles this with room to spare. The number that matters is not QPS; it is that each of those 10M events is durable, ordered, and immutable.

Ledger + audit storage per year:

Per wire, the durable footprint is the transfer row, its ledger entries (double-entry: debit source, credit a clearing/nostro account), and its event-log rows.

transfer row              ~500 bytes (accounts, amounts, beneficiary, refs, state)
ledger entries (~4)       ~4 * 150  = 600 bytes
event log (~10)           ~10 * 250 = 2,500 bytes
screening records (~2)    ~2 * 400  = 800 bytes
------------------------------------------------
≈ 4.4 KB per wire, round to ~5 KB with indexes/overhead

1M wires/day * 5 KB ≈ 5 GB/day
* 365 ≈ 1.8 TB/year

Wire records are retained 5-7 years by regulation and never deleted, so plan for ~10-13 TB over the window. Tiny by modern standards; the story is retention and immutability (WORM cold storage), not sharding for space.

Screening load:

Every wire hits sanctions + AML + fraud screening once (plus re-screens on held items).

~1M screening calls/day + re-screens ≈ 1.2M/day ≈ 14/sec avg, ~120/sec peak

The sanctions list itself is small (hundreds of thousands of entries, tens of MB) and fits in memory on every screening node for fast fuzzy matching. The cost per call is CPU (name matching), not IO.

Idempotency key store:

Keep keys long enough to cover retries and rail settlement windows - say 7 days
(a wire can be in flight for a day+, and a retried recall references the same key).
1M/day * 7 days ≈ 7M keys.
Per key: key(40) + status(1) + transfer_id(16) + response(~200) ≈ ~260 bytes
7M * 260 ≈ 1.8 GB

The headline: raw scale is nothing (~12 writes/sec). The engineering is entirely about correctness under an irreversible external rail: exactly-once sending, screening before release, a monotonic state machine, and an immutable audit trail with zero data loss. That is where the design spends everything.

High-Level Design (HLD)

The spine: a Transfer API validates and dedupes on an idempotency key, a Ledger Service places a strongly-consistent debit hold, a Compliance/Screening Service decides release inside the critical path, an Orchestrator (saga) drives the irreversible rail via a Rail Gateway and advances a persisted state machine, and an append-only Audit Log records every transition. Settlement confirmations and returns arrive async and feed reconciliation.

                    ┌──────────────────────────────────┐
                    │  Client (customer app / system)   │
                    └────────────────┬─────────────────┘
                                     │ POST /transfers (Idempotency-Key)
                    ┌────────────────▼─────────────────┐
                    │          Transfer API              │
                    │  authN/Z + validate + idempotency  │
                    │  Stateless.                         │
                    └───────┬────────────────────┬───────┘
              idempotency   │                    │  reads/status
                 check      │                    └──────────────┐
              ┌─────────────▼──────┐                  ┌─────────▼────────┐
              │ Idempotency Store  │                  │  Status Read Model│
              │ (Redis + DB unique)│                  │  (per-user)       │
              └─────────────┬──────┘                  └───────────────────┘
                            │ new request
                    ┌───────▼──────────────────────────┐
                    │       Ledger Service               │
                    │  ONE ACID txn: place DEBIT HOLD    │
                    │  guard balance-available >= amount │
                    │  write transfer row = INITIATED    │
                    │  + outbox event                    │
                    └───────┬────────────────────────────┘
                            │ transfer.initiated (outbox -> bus)
                    ┌───────▼──────────────────────────┐
                    │        Orchestrator (Saga)         │
                    │  drives state machine, step by step│
                    └──┬──────────────┬─────────────┬────┘
             screen    │              │ send        │ record
          ┌────────────▼───┐   ┌──────▼───────┐  ┌──▼──────────────┐
          │  Compliance /   │   │ Rail Gateway │  │  Audit Log       │
          │  Screening Svc   │   │ (Fedwire/    │  │ (append-only,    │
          │  sanctions+AML+  │   │  SWIFT/RTGS) │  │  immutable, WORM)│
          │  fraud           │   │ idempotent   │  └──────────────────┘
          └────────┬─────────┘   └──────┬───────┘
          hit ->   │ hold/block         │ ISO 20022 / MT103 msg
          ops queue│              ┌─────▼──────────────────────┐
                   │              │  External Rail / Correspondent│
                   │              │  bank network (irreversible)  │
                   │              └─────┬──────────────────────┘
                   │                    │ ack + settlement confirm + returns
                   │              ┌─────▼──────────────────────────────┐
                   │              │  Confirmation Consumer               │
                   │              │  advances SETTLED / RETURNED / FAILED │
                   │              └─────┬──────────────────────────────┘
                   │                    │
                   │              ┌─────▼──────────────────────────────┐
                   └─────────────►│  Reconciliation (nostro/vostro vs   │
                                  │  rail statement) + break management  │
                                  └──────────────────────────────────────┘

Synchronous path (what the customer waits on):

1. Client calls POST /transfers with {source_account, beneficiary, amount, currency} and a client-generated Idempotency-Key.
2. Transfer API authenticates, validates the beneficiary format (IBAN/routing/BIC checksum), and hits the idempotency layer: a known key returns the stored original response and does nothing else.
3. On a new key, the Ledger Service places a debit hold in one ACID transaction: guard available_balance >= amount, move the amount from available to held, write the transfers row in state INITIATED, and insert an outbox event. Commit. The customer’s money is now reserved but not gone.
4. API returns 202 Accepted with the transfer_id and state SCREENING. The wire is now the Orchestrator’s problem.

Asynchronous path (the saga drives the rest):

1. The Orchestrator consumes transfer.initiated and runs the state machine. First step: call Compliance/Screening. A clear result advances to RELEASED; a hit moves to HELD (into an ops review queue) or BLOCKED (terminal, funds released back). Screening happens before anything leaves.
2. On RELEASED, the Orchestrator tells the Rail Gateway to send the payment message (ISO 20022 pacs.008 / MT103) with a deterministic idempotency key so the rail sends once. State -> SENT, holding the rail reference.
3. The rail acknowledges receipt (ACK), then later confirms settlement. The Confirmation Consumer advances SENT -> SETTLED (money truly gone; the hold becomes a real debit) or handles RETURNED / REJECTED by posting a compensating credit back to the source.
4. Every transition writes to the append-only Audit Log, and Reconciliation matches our ledger’s clearing/nostro entries against the rail’s end-of-day statement, raising a break for any mismatch.

The structural insight: there is no cross-bank transaction, so the wire is a durable, monotonic state machine plus a saga. The only strongly-consistent step is the local debit hold. Everything else - screening, sending, settlement - is an ordered sequence of idempotent steps, each recorded immutably, each recoverable from the persisted state after any crash.

Component Deep Dive

The hard parts, naive-first then evolved: (1) exactly-once sending over an irreversible rail, (2) strong consistency of the debit hold and the state machine, (3) compliance screening in the critical path, (4) the immutable regulatory audit trail and reconciliation.

1. Exactly-once sending over an irreversible rail

The job: the payment instruction must leave for the beneficiary bank exactly once, even though the Orchestrator can crash mid-send, the queue is at-least-once, and the rail call can fail ambiguously. And unlike a card charge, a wire cannot be reversed - a duplicate is real money gone that you must claw back by hand.

Naive approach: consume the event, call the rail, mark it sent

def on_initiated(evt):
    rail.send(evt.beneficiary, evt.amount)   # external call
    db.set_state(evt.transfer_id, 'SENT')
    # commit offset

Where it breaks:

• Crash between send and record. The rail accepts the message, then the Orchestrator dies before writing SENT or committing the queue offset. On restart the event is redelivered, the wire is sent again. Two irreversible wires. The customer’s counterparty is paid twice and you are now writing a recall request and eating the loss if it fails.
• Ambiguous rail failure. You send; the connection drops before the ack. Did it go through? Retrying may double-send; not retrying may lose the wire. The error alone cannot tell you.
• Send-before-reserve ordering bug. If sending happens before the funds are locked, a race can send a wire the account cannot cover.

At-least-once delivery plus a non-idempotent, irreversible external call is the worst outcome in the system: duplicated real money you cannot undo.

Evolved approach: reserve-first ordering + deterministic rail idempotency key + a persisted send state with a pre-write

Compose exactly-once from ordering, a provider idempotency key, and a durable “intent to send” record.

1. Order is fixed and enforced by the state machine. A wire can only be sent from state RELEASED, which is only reachable from INITIATED (hold placed) via SCREENING (cleared). Sending can never precede reserving or screening because the state machine forbids the transition. The Orchestrator reads the persisted state, not an in-memory assumption, so a restart resumes from exactly where it was.

2. Pre-write the send intent, then send, then confirm. Before calling the rail, the Orchestrator writes SENDING with the exact rail_idempotency_key it will use, in a committed transaction. Only then does it call the rail. This means after any crash, recovery sees SENDING with a known key and knows precisely which key to retry - never “did I send something?”

3. Deterministic idempotency key toward the rail. Real rails and gateways accept a client-supplied reference / idempotency key (Fedwire IMAD/OMAD, an end-to-end ID in ISO 20022, a gateway idempotency header). We derive it deterministically from the transfer: key = transfer_id + ":send", and pass the same key on every attempt. The rail (or the gateway in front of it) then guarantees one send regardless of retries; a retry after an ambiguous failure either completes the original or returns its already-known result.

def send_wire(transfer):
    if transfer.state in ('SENT', 'SETTLED'):
        return                                   # already sent, no-op
    if transfer.state != 'RELEASED':
        raise IllegalTransition                  # state machine guard

    key = f"{transfer.id}:send"
    db.set_state(transfer.id, 'SENDING', rail_idempotency_key=key)  # pre-write, commit

    resp = rail.send(
        instruction=build_pacs008(transfer),
        idempotency_key=key)                      # SAME key on every retry

    if resp.status == 'ACCEPTED':
        db.set_state(transfer.id, 'SENT', rail_ref=resp.rail_ref)
    elif resp.permanent_reject:
        with db.transaction():
            ledger.release_hold(transfer)         # give funds back
            db.set_state(transfer.id, 'REJECTED')
    else:
        raise RetryableError                      # transient: re-drive with backoff

4. The outbox for the trigger event. The transfer.initiated event that wakes the Orchestrator is written in the same ACID transaction as the debit hold (outbox pattern), so the money-reserve and the “go process this” signal commit together. A relay publishes the outbox to the bus at-least-once; the Orchestrator dedupes on transfer_id. There is no window where funds are held but nothing drives the wire, nor one where a wire is driven without funds held.

5. Recovery sweeper. A periodic job scans for wires stuck in SENDING (crash mid-send) or SENT past the expected settlement window and re-drives them - safely, because the rail idempotency key makes a re-send a no-op if the first landed, and reconciliation confirms which actually settled.

Why this is right: you cannot get exactly-once from an irreversible rail on trust, so you build it - fixed ordering makes send unreachable until funds are reserved and screening cleared, the pre-written SENDING state makes recovery deterministic, and the deterministic rail key makes the external send happen once no matter how many times you call. At-least-once plumbing over an irreversible rail becomes exactly-once money movement.

2. Strong consistency: the debit hold and the monotonic state machine

The job: the source account must never be double-debited or overdrawn from a race, and the wire must move through its lifecycle in one direction only, with each transition durable, even under concurrent operators, retries, and crashes.

Naive approach: debit the balance up front, track state with a status column you overwrite

def initiate(transfer):
    acct.balance -= transfer.amount    # read-modify-write
    save(acct)
    transfer.status = 'SENT'           # later overwritten to SETTLED, etc.
    save(transfer)

Where it breaks:

• Lost updates / overdraft under concurrency. Two wires from the same account read balance = 100 at once, both subtract 80, both write 20. The account sent 160 it did not have. A read-modify-write with no guard prints money out the door.
• Debiting before screening / settlement. If you subtract the balance up front and the wire is later blocked or returned, you must remember to add it back - and if the process crashes in between, the customer’s money is simply gone from their available balance with nothing sent.
• Overwriting status loses the trail and allows illegal transitions. A retried callback can flip SETTLED back to SENT, or BLOCKED to RELEASED. Nothing enforces that state only moves forward, and the history of how it got there is erased on each overwrite.

A mutable balance with no hold and an overwritable status is race-prone, non-atomic, and unauditable.

Evolved approach: a two-bucket balance with an ACID hold + an append-only, monotonic state machine

Split the balance into available and held. A wire does not debit; it reserves. In one ACID transaction, guarded against races:

BEGIN;
  -- guard overdraft AND concurrent races in one conditional update:
  UPDATE accounts
     SET available = available - :amount,
         held      = held      + :amount,
         version   = version   + 1
   WHERE account_id = :src
     AND available >= :amount;          -- 0 rows => insufficient funds => ROLLBACK

  INSERT INTO ledger_entries (tx_id, account_id, direction, amount, kind) VALUES
    (:tx, :src,            'DEBIT',  :amount, 'HOLD'),
    (:tx, :clearing_acct,  'CREDIT', :amount, 'HOLD');   -- double-entry, sums to zero

  INSERT INTO transfers (transfer_id, ..., state) VALUES (:tx, ..., 'INITIATED');
  INSERT INTO outbox (transfer_id, topic, payload) VALUES (:tx, 'transfer.initiated', ...);
COMMIT;

• The conditional UPDATE ... WHERE available >= amount is the concurrency guard. Row locks serialize concurrent wires on the same account; the second either still passes or fails and rolls back. No overdraft, no lost update, no external lock service.
• The money moves from available to held, not out. It is committed only when the wire settles (hold becomes a real DEBIT against a nostro/clearing account) and released back to available if the wire is blocked, rejected, or returned. Double-entry keeps the books balanced at every step.

The state machine is append-only and monotonic. State transitions are validated against an allowed-transition table and recorded as immutable rows (event-sourced), with the current state as a derived column updated in the same transaction.

INITIATED -> SCREENING -> RELEASED -> SENDING -> SENT -> SETTLED   (happy path)
     |            |            |                    |
     |            v            v                    v
     |          HELD         BLOCKED             RETURNED / REJECTED
     |            |            (funds released)   (compensating credit, funds released)
     |            v
     |         RELEASED (operator clears) or BLOCKED (operator confirms)
     +------> CANCELLED (only before RELEASED)

• A transition is applied with an optimistic guard: UPDATE transfers SET state=:new, version=version+1 WHERE transfer_id=:id AND state=:expected AND version=:v. If zero rows change, someone else moved it; the caller re-reads and decides. This makes concurrent operators and retried callbacks safe - a duplicate SETTLED callback finds the row already SETTLED and is a no-op.
• Terminal states are final. SETTLED, BLOCKED, RETURNED, REJECTED, CANCELLED allow no outgoing transition. A wire cannot un-settle; a return is a new inbound transfer that references the original, not a mutation of it.
• Because every transition is an immutable row, the full history is inherent: you can replay exactly how a wire reached its state and when.

Why this is right: the hold makes the money safe and reversible locally (the one place reversal is possible) while the wire is still in your control, the conditional update makes overdraft and races impossible, and the monotonic, append-only state machine makes the lifecycle deterministic, idempotent under retries, and fully auditable. Strong consistency lives exactly where it must - the local reserve and the state transition - and nowhere it cannot (across banks).

3. Compliance screening in the critical path

The job: every wire must be screened against sanctions lists (OFAC SDN, UN, EU), AML rules (structuring, velocity, high-risk corridors), and a fraud model, and it must be screened before it leaves. A sanctioned payment that settles is a felony-grade regulatory failure; you cannot send first and screen later.

Naive approach: screen asynchronously after sending, or skip it under load

def initiate(transfer):
    reserve(transfer)
    rail.send(transfer)              # money leaves
    queue.async(screen, transfer)    # check later

Where it breaks:

• The money is already gone. If screening later flags a match to a sanctioned entity, the wire has settled irreversibly. You have facilitated a prohibited transaction. There is no undo; there is a fine and a consent order.
• No hold state. With screening off the critical path, there is no “held for review” lifecycle, no queue for compliance officers, and no record that a decision was made - which is itself a recordkeeping violation.
• Load-shedding compliance. Skipping screening when busy is not a degradation, it is a breach. Compliance is not optional under load.

Screening after the irreversible send defeats the entire purpose of screening.

Evolved approach: synchronous gate, tiered checks, held-state review queue, immutable decisions

Screening is a mandatory gate between INITIATED and RELEASED. The wire cannot advance to SENDING until it clears.

Tiered, fast-first checks so the common (clean) case is quick and only ambiguous ones get expensive:

• Tier 1 sanctions matching is fuzzy, not exact. Names transliterate (“Mohammed” vs “Muhammad”), addresses vary, so we use normalized fuzzy matching (Jaro-Winkler / token-based) with a tuned threshold. A strong match blocks; a weak match holds for a human. False positives are acceptable (a held clean wire is delayed); a false negative (a sanctioned wire released) is not - the threshold is set to favor holding.
• HELD -> review queue. A hold puts the wire in a compliance-officer queue with the match details. The officer clears (-> RELEASED) or confirms (-> BLOCKED, funds released back to available). Every decision - automated or human - is written to the immutable audit log with the rule/list version, the match score, the officer id, and the rationale. That record is what a regulator inspects.
• Screening is idempotent and re-runnable. The Orchestrator calls screening with the transfer_id; a retry returns the same decision. Long-held wires are re-screened if the sanctions list updates before they release, because the list that mattered is the one in effect at release time.
• Determinism and versioning. We store which list version and rule set version produced the decision. If a wire is later reviewed, we can reproduce exactly what the system saw. “We screened against yesterday’s list” is a defensible, provable statement only if the version is recorded.

Why this is right: putting screening on the critical path, before the irreversible send, is the only ordering that is legally correct - you cannot recall a sanctioned wire. Tiering keeps the clean majority fast; the held-state queue and immutable, versioned decisions give compliance officers a workflow and give regulators a reproducible trail. The cost is latency on the initiation path (seconds, occasionally a manual hold), which at 12 wires/sec we can absolutely afford.

4. The immutable regulatory audit trail and reconciliation

The job: reconstruct any wire completely, years later - every state, every decision, every operator action - and prove daily that our ledger matches what the rail actually settled. Zero data loss.

Naive approach: log to the transfers table and trust it

Keep a status and an updated_at on the transfer row; rely on database backups for history.

Where it breaks:

• Overwrites erase history. A mutable status column shows only the current state, not the path. “When was it screened, by whom, against which list?” is unanswerable.
• No tamper-evidence. A row can be updated or deleted by anyone with DB access; there is no proof the record was not altered after the fact, which is exactly what an auditor tests.
• Missed external events. If a settlement confirmation webhook is lost, our ledger says SENT forever while the rail says settled. Money is in a state we do not know about, and nothing catches it.

“Trust the transfers table” fails both requirements: it is neither immutable nor verified against reality.

Evolved approach: append-only event log (WORM) + hash chaining + double-sided reconciliation

Append-only event log. Every state transition, screening decision, and operator action is an immutable event row: (event_id, transfer_id, seq, type, actor, payload, prev_hash, hash, created_at). Nothing is ever updated or deleted. The current transfer state is a projection of this log; the log is the source of truth for history.

• Hash chaining for tamper-evidence. Each event stores hash = H(prev_hash || payload), chaining events per transfer (and periodically anchoring the chain head). Any after-the-fact edit breaks the chain and is detectable. This is what turns “we have logs” into “we can prove the logs were not altered.”
• WORM retention. The log is written to write-once-read-many cold storage (object storage with a compliance lock / legal hold) for the 5-7 year retention window. It is never mutable and never purged early.
• Separation of duties. Application code appends; nobody has update/delete on the log. Access to read it is itself logged.

Double-sided reconciliation proves the ledger matches the rail:

• Source A - our ledger. All entries against the nostro/clearing accounts for the period.
• Source B - the rail statement. The end-of-day settlement file (MT940/camt.053) the correspondent bank / rail produces, listing every wire it actually settled and any fees.
• Match on the shared reference (rail_ref / end-to-end ID). Each line is one of:

• Suspense (control) account. Every break posts a balancing entry to a suspense account so the ledger stays balanced while a human investigates; the suspense balance is a live measure of unreconciled money and should trend to zero. A rising suspense balance pages someone.
• Break management workflow. Each unmatched line becomes an owned break (open / investigating / resolved); nothing closes without a ledger action, so the audit trail stays complete.
• Internal integrity checks run alongside: global SUM(all ledger entries) == 0 and per-account available + held == SUM(entries). These catch our own bugs independent of the rail.

Why this is right: idempotency and the state machine make us internally correct, but only an immutable, tamper-evident log proves how each wire happened, and only reconciliation against the rail’s own statement proves the money actually moved as recorded. Together they satisfy “reconstructable years later” and “zero unaccounted money” - the two things a wire system is legally required to guarantee.

API Design & Data Schema

API

All mutating calls require an Idempotency-Key.

POST /api/v1/transfers
Headers: Authorization: Bearer <token>
         Idempotency-Key: <client-generated UUID, reused on retries>
Body:
{
  "source_account": "acc_123",
  "beneficiary": {
    "name": "Acme GmbH",
    "iban": "DE89370400440532013000",
    "bic":  "COBADEFFXXX",
    "bank_name": "Commerzbank"
  },
  "amount":   250000,           // integer minor units (cents) = 2500.00
  "currency": "USD",
  "reference": "invoice 4471"
}
Response 202:
{
  "transfer_id": "wt_9f3a...",
  "state": "SCREENING",
  "amount": 250000,
  "currency": "USD",
  "estimated_settlement": "2026-07-18T16:00:00Z",
  "created_at": "2026-07-18T09:04:00Z"
}
Errors:
  400  invalid payload / bad IBAN or BIC checksum
  402  insufficient available funds
  401  bad token
  409  idempotency key still in progress (retry shortly)
  422  blocked by compliance (sanctions/AML)

Initiation returns 202, not 201: we have accepted, reserved funds, and started screening, but the wire has not settled - its final state arrives asynchronously. Reads and lifecycle:

GET  /api/v1/transfers/{id}                 -> full state, timeline, rail_ref
GET  /api/v1/transfers/{id}/events          -> immutable event log (audit view)
GET  /api/v1/accounts/{id}/transfers        -> paginated history (read model)
POST /api/v1/transfers/{id}/cancel          -> only valid before RELEASED
POST /api/v1/webhooks/{rail}                -> rail ack / settlement / return callbacks
POST /api/v1/transfers/{id}/recall          -> best-effort recall request (new saga)

Data stores

Different access patterns, different stores. Be explicit.

1. Ledger, accounts, transfers, state machine - SQL, ACID, synchronously replicated. This is where strong consistency, the available >= amount guard, the two-bucket hold, and the same-transaction outbox live. A relational database (PostgreSQL, or distributed SQL like CockroachDB/Spanner for HA) is the correct, non-negotiable choice. At ~12 writes/sec we do not even need sharding for throughput; we run a primary with synchronous standby replicas and automatic failover for durability. NoSQL’s eventual consistency would reintroduce the overdraft races we designed out.

Table: accounts
  account_id     BIGINT   PRIMARY KEY
  customer_id    BIGINT   INDEX
  currency       CHAR(3)
  available      BIGINT   -- minor units, spendable
  held           BIGINT   -- minor units, reserved by in-flight wires
  version        BIGINT   -- optimistic concurrency
  status         STRING   -- active|frozen|closed
  Invariant: available >= 0 and held >= 0

Table: transfers                        -- one row per wire, state projection
  transfer_id       UUID    PRIMARY KEY
  source_account    BIGINT  INDEX
  beneficiary       JSON    -- name, iban, bic, bank
  amount            BIGINT  -- minor units
  currency          CHAR(3)
  state             STRING  -- INITIATED|SCREENING|HELD|RELEASED|SENDING|SENT|
                            --   SETTLED|BLOCKED|RETURNED|REJECTED|CANCELLED
  version           BIGINT  -- optimistic guard for transitions
  rail_ref          STRING  NULL  -- IMAD/OMAD / end-to-end id
  rail_idem_key     STRING  NULL  -- deterministic key we pass to the rail
  screening_ref     UUID    NULL
  created_at        TIMESTAMP
  updated_at        TIMESTAMP

Table: ledger_entries                   -- APPEND ONLY, immutable
  entry_id     UUID     PRIMARY KEY
  transfer_id  UUID     INDEX
  account_id   BIGINT   INDEX
  direction    CHAR(1)  -- D | C
  kind         STRING   -- HOLD | SETTLE | RELEASE | REVERSAL | ADJUSTMENT
  amount       BIGINT   -- minor units, positive; direction gives sign
  currency     CHAR(3)
  created_at   TIMESTAMP
  Invariant: per transfer_id, SUM(credits) - SUM(debits) = 0

Table: transfer_events                  -- APPEND ONLY, hash-chained audit log
  event_id     UUID     PRIMARY KEY
  transfer_id  UUID     INDEX
  seq          INT      -- per-transfer monotonic sequence
  type         STRING   -- state change, screening decision, operator action
  actor        STRING   -- system | officer:<id> | rail
  payload      JSON     -- old->new state, match score, list version, rationale
  prev_hash    BYTES
  hash         BYTES    -- H(prev_hash || payload) : tamper evidence
  created_at   TIMESTAMP
  UNIQUE(transfer_id, seq)

Table: idempotency_keys
  idem_key     STRING   PRIMARY KEY   -- UNIQUE; the real dedup guarantee
  transfer_id  UUID     NULL
  status       STRING   -- IN_PROGRESS | DONE
  response     JSON     -- stored original response for replay
  created_at   TIMESTAMP
  TTL/archive: 7 days hot

Table: outbox
  outbox_id    BIGINT   PRIMARY KEY
  transfer_id  UUID
  topic        STRING
  payload      JSON
  sent         BOOL     INDEX (partial: WHERE sent=false)

2. Status / history read model - NoSQL wide-column (Cassandra/DynamoDB), by account. Customer-facing lists are read by account_id in reverse-chronological pages, never joined, and tolerate eventual consistency. Built by a projector off the event bus so status reads never touch the transactional primary.

3. Sanctions list + screening cache - in-memory (per screening node) + a versioned store. The list (tens of MB) is loaded into memory for fuzzy matching; the versioned copy in object storage lets us record and reproduce exactly which list version made each decision.

4. Audit cold storage - WORM object storage. The transfer_events log is continuously exported to write-once-read-many storage with a compliance lock for the 5-7 year retention window.

5. Reconciliation store - SQL. Rail statement lines, match results, and the breaks workflow. Relational because it is joined against the ledger and needs consistency for break state.

Why SQL for the core, NoSQL for the projections: the wire’s money path needs multi-row ACID, row-level locking for the overdraft guard, monotonic state transitions, and same-transaction idempotency+outbox - exactly what relational databases exist for, and at this volume one primary handles it. History and status reads are key-scoped, read-heavy, and eventual-consistency-tolerant, so they go to a wide-column store. Match the guarantee to the data: absolute for money and audit, relaxed for the read projections of it.

Bottlenecks & Scaling

Where it breaks first, in order, and the fix. Note that at ~12 writes/sec, most “bottlenecks” here are correctness and availability failures, not throughput ceilings.

1. Duplicate irreversible sends (breaks first, worst impact). A retry or crash sends a second wire that cannot be undone. Fix: client Idempotency-Key with a DB unique-constraint backstop and same-transaction result storage; a deterministic rail idempotency key passed on every send attempt so the rail sends once; the pre-written SENDING state so recovery is deterministic; consumers dedupe rail callbacks on transfer_id. This is the single most important mechanism in the system.

2. Sending before funds are reserved or screening clears. An ordering bug sends money the account cannot cover or that should have been blocked. Fix: the monotonic state machine makes SENDING reachable only from RELEASED, which requires a committed hold (INITIATED) and a cleared screen (SCREENING -> RELEASED). Order is enforced by data, not by code discipline.

3. Overdraft / double-debit under concurrency. Two wires from one account race on the balance. Fix: two-bucket balance (available/held) with a conditional UPDATE ... WHERE available >= amount and row locks; optimistic version guards on every state transition so concurrent operators and retried callbacks serialize safely.

4. The dual-write problem (DB commit vs bus publish). Funds reserved but the driving event lost leaves a wire held with nothing progressing it. Fix: the outbox pattern - the transfer.initiated event is inserted in the same ACID transaction as the hold; a relay publishes at-least-once; a recovery sweeper re-drives wires stuck in any non-terminal state.

5. Rail outage or ambiguous rail responses. The rail is down or answers ambiguously; the wire is in limbo. Fix: queue and hold (state RELEASED/SENDING persists) rather than dropping; typed retries with exponential backoff and jitter; a circuit breaker and, where a corridor has one, a secondary correspondent/rail; a DLQ for wires that exhaust retries, worked by ops. Availability degrades to “accepted, settling later,” never to “lost.”

6. Compliance screening as a latency and correctness chokepoint. Screening is synchronous and mandatory; a slow or unavailable screening service stalls every wire. Fix: tiered checks (fast in-memory sanctions first), the sanctions list replicated in memory on every screening node (horizontally scalable, no shared IO), and fail-closed semantics - if screening is unavailable, wires hold rather than release unscreened. Holding a clean wire is acceptable; releasing an unscreened one is not.

7. Ledger drift from the rail (missed confirmations, returns, fees). Internal correctness cannot detect external divergence. Fix: daily and intraday reconciliation against the rail’s settlement statement, a suspense account that keeps the ledger balanced while breaks are worked, and an owned break-management workflow. A rising suspense balance pages someone.

8. Audit integrity and retention. History must be immutable, tamper-evident, and retained for years with zero loss. Fix: append-only, hash-chained event log; WORM cold storage with compliance lock and legal hold; separation of duties (no update/delete on the log); continuous export so nothing lives only on the hot primary.

9. Single points of failure / durability. Any node loss must lose no accepted wire. Fix: SQL with synchronous replication and automatic failover (no committed transaction lost); bus and read stores replicated; stateless API and Orchestrator horizontally scaled; the outbox, DLQ, and sweeper guarantee nothing in flight vanishes silently.

Shard keys / partitioning, stated plainly: at this volume the SQL core does not need sharding for throughput - it needs synchronous replication for durability. If volume grew 100x, shard accounts and ledger_entries by account_id (an account’s entries co-located, writes spread across accounts), transfers/transfer_events by transfer_id, and the read model by account_id for single-partition reverse-chronological reads. Never shard by time - it concentrates all current writes on the newest partition. Kafka topics partition by transfer_id for per-wire ordering.

Wrap-Up

The trade-offs that define this design:

• No cross-bank transaction, so a saga over a monotonic state machine. The two legs live in different institutions with no shared ACID scope. We accept that reality and make correctness come from ordering, idempotency, and a persisted, append-only state machine instead of a database transaction we cannot have.
• Reserve locally, send irreversibly, confirm asynchronously. The only strongly-consistent step is the local two-bucket debit hold. Everything after - screening, sending, settlement - is an ordered sequence of idempotent, recoverable steps, because the money leaving the bank cannot be rolled back.
• Screen before release, always. Sanctions/AML/fraud screening sits on the critical path before the irreversible send, and fails closed. We trade some initiation latency (and the occasional manual hold) for never facilitating a prohibited or fraudulent wire.
• Exactly-once against an irreversible rail. A client idempotency key with a DB unique-constraint backstop, a deterministic rail key on every send, the pre-written SENDING state, and the outbox together turn at-least-once plumbing into exactly-once, undoable-only-by-compensation money movement.
• Prove it, do not trust it. An immutable, hash-chained, WORM-retained event log makes every wire reconstructable and tamper-evident; daily reconciliation against the rail’s own statement proves the money moved as recorded. Auditability and zero data loss are guarantees, not aspirations.

One-line summary: a low-throughput, high-stakes Transfer API places a strongly-consistent local debit hold, screens every wire against sanctions/AML/fraud before release, then drives an irreversible external rail through an idempotent saga over a monotonic, append-only state machine - sending exactly once with a deterministic rail key, recording every transition to a tamper-evident WORM audit log, and reconciling daily against the rail - so money leaves once, only when cleared, and is always provable years later.