“Move 500 rupees from Alice to Bob” sounds like an UPDATE statement. Then you think about it. What if the request times out and the client retries - did the first one go through? What if the debit succeeds and the process crashes before the credit? What if two withdrawals from the same wallet run at the same instant and both read the old balance? What if the bank tells you a transfer succeeded but your database says it failed? What if a support agent needs to prove, three years later, exactly where a specific rupee went? The one-line update is actually a distributed system with an idempotency layer, an immutable ledger, a state machine per transaction, an integration with unreliable external banks and gateways, and a reconciliation job that catches the cases where all of that still drifted.

The core of this problem is that money is not data you can afford to lose, duplicate, or invent. A double-charge is a refund and an angry user. A lost credit is money that vanished. An inconsistent balance is fraud waiting to happen. Every mechanism in this design exists to make money movement idempotent, atomic, auditable, and eventually provably correct against the outside world. Let me build it properly.

Functional Requirements (FR)

In scope:

  • Hold a balance. Each user has one or more wallet accounts with a balance in a specific currency. The balance is always derivable and always non-negative for a normal wallet.
  • Top up (add money). Pull funds in from an external source - a card, UPI, net banking - via a payment gateway, and credit the wallet on confirmation.
  • Peer transfer. Move money from one wallet to another atomically: the sender is debited exactly what the receiver is credited, or neither happens.
  • Pay a merchant / withdraw. Debit the wallet and push money out to an external account (merchant settlement or bank withdrawal) through a provider.
  • Idempotent money movement. Every mutating operation is safe to retry. A client that sends the same transfer twice (timeout + retry) moves money once.
  • Transaction history and balance query. A user can see every transaction and the running balance. Internally we can reconstruct any balance from the ledger.
  • Reconciliation. A daily job that proves our ledger matches what banks and gateways actually did, and flags every discrepancy.

Explicitly out of scope (say it so you own the scope):

  • KYC, onboarding, and the fraud/risk engine. I assume a risk decision is an input (a transaction is approved or declined by a separate service); I do not build the fraud model.
  • The card network / issuer internals. I integrate with a payment gateway as a black box; I do not design the Visa/UPI rails themselves.
  • Foreign exchange and multi-currency conversion. Wallets are single-currency here; FX is a whole separate ledger problem I would name but not design in this pass.
  • Interest, lending, and statements/taxation. The wallet holds money; it does not compute interest or generate tax documents.

The decision that drives everything: the balance is never the source of truth - the ledger is. A wallet balance is a cached projection of an append-only, double-entry ledger. Every design choice - idempotency, atomicity, reconciliation - follows from treating the ledger as immutable financial fact and the balance as a fast, rebuildable read model.

Non-Functional Requirements (NFR)

  • Scale: a large consumer wallet. ~100M registered users, ~30M DAU. Peak ~10K payment transactions/sec (festival sale, salary day, IPL final). Reads (balance + history) dominate at ~100K/sec.
  • Latency: a payment authorization the user is waiting on should return p99 under ~500ms for the internal path (excluding the external bank round trip, which we make async where possible). Balance reads p99 under ~50ms.
  • Availability: 99.99% on the read/authorize path. A user must be able to check balance and initiate a payment almost always. Money movement can degrade to “accepted, settling” but must never silently fail.
  • Consistency: strong for money. A balance must never go negative from a race; a transfer must be atomic; the ledger must always balance (sum of debits = sum of credits). This is the one system where we do NOT trade consistency for availability on the write path. History/analytics can be eventually consistent.
  • Durability: absolute. Once a transaction is committed, it survives any single node loss. Financial data is replicated synchronously and never hard-deleted; corrections are new entries, not edits.
  • Auditability: every money movement is traceable end to end, years later, immutably. Regulators and disputes demand it. The ledger is append-only for exactly this reason.

The tension to state up front: money wants strong consistency and full durability, which fight throughput and availability. We resolve it by keeping the strongly-consistent core small (the ledger write and the balance guard), making everything around it - reads, history, notifications, external settlement - async and horizontally scalable, and using idempotency + reconciliation to stay correct across the inevitable retries and external failures.

Back-of-the-Envelope Estimation (BoE)

Real numbers with the arithmetic, because they dictate the architecture.

Transaction write volume:

30M DAU, each doing ~3 money-moving transactions/day
(a top-up, a couple of payments).
30M * 3 = 90M transactions/day
= 90,000,000 / 86,400 sec
≈ 1,040 transactions/sec average
Peak (salary day / sale, ~10x avg) ≈ 10,000 transactions/sec write

Each money-moving transaction is at least two ledger entries (double-entry: one debit, one credit), often four when an external account and a fee account are involved.

Peak ledger-entry writes ≈ 10,000 tx/sec * 2-4 entries ≈ 20,000-40,000 entries/sec

Read volume (balance + history):

Reads outnumber writes heavily - every app open checks balance.
30M DAU * ~10 balance/history views/day = 300M reads/day
≈ 3,500 reads/sec average, peak ~100,000 reads/sec

That 100:1 read:write ratio is why balance is a cached projection, not a live SUM over the ledger.

Ledger storage per year:

Per ledger entry:

entry_id         16 bytes (UUID)
transaction_id   16 bytes
account_id       8 bytes
direction        1 byte  (debit/credit)
amount           8 bytes (integer minor units - paise, never float)
currency         3 bytes
balance_after    8 bytes (optional running balance snapshot)
created_at       8 bytes
metadata_ref     ~40 bytes
----------------------------------------------
≈ 108 bytes, round to ~200 bytes with indexes/overhead

90M tx/day * 3 entries avg = 270M entries/day
270M * 200 bytes ≈ 54 GB/day
* 365 ≈ 19.7 TB/year

Ledger data is never deleted (regulatory retention is typically 7-10 years), so plan for ~140-200 TB over the retention window. Hot data (recent ~90 days) stays on fast storage; older partitions roll to cheaper cold storage but remain queryable. This is a sharding-and-partitioning story, not a “fits in one box” story.

Idempotency key store:

Keep idempotency keys for a window long enough to cover client retries -
say 24-48h hot, then archive.
Peak-ish 10K writes/sec * 86,400 * 2 days ≈ 1.7B keys at true peak,
steady more like 1,040/sec * 86,400 * 2 ≈ 180M keys.
Per key: key(~40) + response_hash(~40) + status(1) + tx_id(16) ≈ ~120 bytes
180M * 120 bytes ≈ 21 GB hot

~20-60 GB of fast storage for idempotency, TTL’d, backed by a hard unique constraint in the durable store so a cache gap can never cause a double-spend.

Balance cache:

100M accounts * (account_id 8 + balance 8 + version 8 + currency 3) ≈ 27 bytes
Round to ~60 bytes with overhead: 100M * 60 ≈ 6 GB

Tiny. The active-user working set is far smaller, so a few Redis nodes hold every hot balance.

The headline: raw write throughput (~10K tx/sec) is modest. The hard parts are correctness under retries (idempotency), atomicity of the two-sided ledger write, exactly-once processing across queues and external calls, and proving the whole thing matches reality (reconciliation). That is where the design earns its keep.

High-Level Design (HLD)

The spine is a Payment API that authenticates and validates, an Idempotency layer that collapses retries, a strongly-consistent Ledger Service that writes the double-entry transaction atomically and updates the balance under a guard, and an asynchronous settlement path that talks to external banks/gateways and is reconciled daily. Reads are served from a cached balance and a separate history read model.

                    ┌──────────────────────────────┐
                    │   Client (app / merchant)     │
                    └───────────────┬──────────────┘
                                    │ POST /transfers  (Idempotency-Key)
                    ┌───────────────▼──────────────┐
                    │        Payment API            │
                    │  authN/Z + validate + risk    │
                    │  check. Stateless.             │
                    └───────┬───────────────┬───────┘
             idempotency    │               │  reads
                check       │               └──────────────┐
              ┌─────────────▼──────┐              ┌─────────▼─────────┐
              │ Idempotency Store  │              │  Balance Cache     │
              │ (Redis + DB unique)│              │  (Redis)           │
              └─────────────┬──────┘              └───────────────────┘
                            │ new request
                    ┌───────▼───────────────────────┐
                    │        Ledger Service           │
                    │  ONE ACID transaction:          │
                    │  1 guard: balance >= amount     │
                    │  2 insert debit + credit entries│
                    │  3 update balances (versioned)  │
                    │  4 write tx row (state machine) │
                    └───┬───────────────────────┬─────┘
                writes  │                       │ emits event
              ┌─────────▼──────────┐    ┌───────▼──────────────┐
              │  Ledger DB          │    │  Event Bus (Kafka)   │
              │ (SQL, sharded,      │    │  transaction.posted  │
              │  synchronous repl)  │    └───┬──────────┬───────┘
              └─────────────────────┘        │          │
                                    ┌─────────▼───┐  ┌───▼────────────────┐
                                    │ Settlement   │  │ History Projector  │
                                    │ Worker       │  │ (builds read model)│
                                    │ (calls bank/ │  └───┬────────────────┘
                                    │  gateway)    │      │
                                    └───┬──────────┘   ┌──▼───────────────┐
                                        │ provider API │ History Store     │
                                ┌───────▼─────────┐    │ (NoSQL, by user)  │
                                │ Payment Gateway  │    └──────────────────┘
                                │ / Bank rails     │
                                └───────┬─────────┘
                                        │ status webhooks + daily statement file
                                ┌───────▼──────────────────────────────────┐
                                │  Reconciliation Service                    │
                                │  compares ledger vs provider statement,    │
                                │  posts corrections, flags breaks           │
                                └───────────────────────────────────────────┘

Write flow (a peer transfer, synchronous where the user waits):

  1. Client calls POST /transfers with {from_account, to_account, amount, currency} and a client-generated Idempotency-Key header.
  2. Payment API authenticates, validates, and runs the risk check (approved/declined comes from the risk service). It then hits the Idempotency layer: if this key was already processed, it returns the stored original response and does nothing else. This is what makes retries safe.
  3. On a new key, control passes to the Ledger Service, which does the money movement inside one ACID database transaction: acquire a guard on the source account (row lock / conditional update), check balance >= amount, insert a debit entry and a matching credit entry (double-entry: they must sum to zero), update both account balances with an optimistic version bump, and write the transaction row in state POSTED. Commit. If anything fails, the whole transaction rolls back and money did not move.
  4. The commit atomically records the idempotency key -> result mapping (same DB, same transaction) so the retry-safety is durable, not just cached.
  5. The Ledger Service returns success to the client and emits a transaction.posted event on the bus.

Async flow (settlement + read models):

  1. The Settlement Worker consumes transaction.posted for transactions that touch the outside world (top-ups, withdrawals, merchant payouts). It calls the payment gateway / bank, tracks the external state, and posts a follow-up ledger entry when the external leg confirms or fails. This is the exactly-once-across-an-external-system problem, handled in the deep dive.
  2. The History Projector consumes the same events and builds the per-user history read model, so history reads never touch the hot ledger shards.
  3. Provider status webhooks update the transaction state machine (PENDING -> SETTLED / FAILED). The Reconciliation Service ingests the provider’s end-of-day statement file and compares it line by line against our ledger, posting correcting entries and raising alerts for anything that does not match.

The key structural insight: the strongly-consistent core is exactly one ACID transaction - the ledger write. Everything cheap and high-volume (reads, history, notifications) is pushed to async projections, and everything unreliable (external banks) is pushed behind a settlement worker and made correct after the fact by reconciliation. The small ACID core is what we scale carefully; the rest scales the easy way.

Component Deep Dive

The hard parts, naive-first then evolved: (1) idempotent money movement, (2) the double-entry ledger and atomic balance update, (3) exactly-once processing across queues and external providers, (4) reconciliation.

1. Idempotent money movement

The job: a client that sends the same money-moving request more than once (because of a timeout, a retry, a flaky network, a double-tap) must move money exactly once.

Naive approach: just process every request that arrives

The obvious first cut trusts the network. The client posts a transfer; the server processes it. If the client does not get a response, it retries.

def transfer(from, to, amount):
    debit(from, amount)
    credit(to, amount)
    return 200

Where it breaks:

  • The server processes the transfer, debits and credits, then the response is lost on the way back (timeout). The client, seeing no success, retries. The server processes it again - money moves twice. The user is charged twice.
  • A double-tap in the UI sends two requests; both are valid-looking; both go through.
  • A Kafka redelivery or a load-balancer retry does the same thing one layer down.

There is no way for the server to tell a genuine second transfer from a retry of the first. Retries are a fact of every network, so “process everything” guarantees double-spends.

Evolved approach: client-supplied idempotency key, stored result, hard unique constraint

Every mutating request carries a client-generated Idempotency-Key (a UUID the client picks once per logical operation and reuses on every retry of that operation). The server makes the operation idempotent on that key:

  1. Reserve the key. Before doing any work, insert a row (idempotency_key, status=IN_PROGRESS) with a unique constraint on the key. If the insert fails on the unique constraint, this key is already known - go to step 3.
  2. Do the work and store the result. Run the ledger transaction. In the same DB transaction, update the idempotency row to status=DONE with the response payload (or a hash + the resulting transaction_id). Commit atomically. Return the response.
  3. Replay on a known key. If the key already existed: if DONE, return the stored original response verbatim - do not touch money. If IN_PROGRESS, the first request is still running (or crashed); return 409 / please retry, or block briefly, so we never run two concurrently for the same key.
def transfer(req):
    try:
        db.insert_idempotency(req.key, status='IN_PROGRESS')  # UNIQUE(key)
    except UniqueViolation:
        row = db.get_idempotency(req.key)
        if row.status == 'DONE':
            return row.stored_response          # replay, no money moves
        else:
            return 409  # in flight; client retries later
    with db.transaction():
        tx = ledger.post_double_entry(req)      # the actual money movement
        db.update_idempotency(req.key, status='DONE', response=tx.response)
    return tx.response

Why this is right:

  • The unique constraint is the real guarantee. Two concurrent retries race to insert the same key; the database lets exactly one win. The loser reads the winner’s result. No application-level locking gymnastics.
  • Same-transaction storage of the result means the “we did this” fact and the money movement commit together. There is no window where money moved but the key was not recorded (which would let a retry double-spend) or the key was recorded but money did not move (which would swallow a legitimate operation).
  • The scope of the key is one logical operation. The client must reuse the same key across retries of the same intent and use a fresh key for a genuinely new transfer. This is the contract we document loudly.
  • A Redis fast path in front (SET key NX) absorbs the vast majority of duplicate hits cheaply, but it is only an optimization; the DB unique constraint is the truth, so a Redis eviction or gap can never cause a double-spend.

Idempotency turns “at-least-once delivery from the client” into “exactly-once money movement,” which is the correctness bar for the whole system.

2. The double-entry ledger and atomic balance update

The job: record money movement so it is always internally consistent (every debit has an equal credit, balances never go negative from a race, and any balance is provable from history), while serving 100K balance reads/sec.

Naive approach: a balance column you increment and decrement

Store a single balance on each account and mutate it directly.

def transfer(from, to, amount):
    from.balance = from.balance - amount   # read-modify-write
    to.balance   = to.balance   + amount
    save(from); save(to)

Where it breaks:

  • Lost updates / negative balances under concurrency. Two withdrawals from the same account read balance = 100 at the same instant, both subtract 80, both write 20 - the account paid out 160 it did not have. A read-modify-write with no guard is a classic race, and here it prints money.
  • No audit trail. The balance is a number with no history. When a user disputes a charge or a regulator asks “where did this 500 go,” there is nothing to show. You cannot reconstruct anything.
  • Partial failure. If the debit saves and the process dies before the credit, money vanished. Two independent updates are not atomic.
  • No integrity check. Nothing enforces that money is conserved. A bug that credits without debiting silently invents money and nobody notices until the books do not add up.

A mutable balance with no ledger is unauditable, race-prone, and non-atomic - three fatal properties for money.

Evolved approach: append-only double-entry ledger + balance as a guarded projection

Adopt double-entry accounting, the 500-year-old invariant that makes money provably conserved: every transaction is a set of entries whose debits and credits sum to zero. Money is never created or destroyed, only moved between accounts.

  • Ledger entries are immutable and append-only. You never update or delete an entry. A transfer of 500 from Alice to Bob is two entries: debit Alice 500, credit Bob 500. Their sum is zero. A mistake is fixed by posting a reversing transaction, not by editing history.
  • Balance is a projection, not the source of truth. The authoritative balance is SUM(credits) - SUM(debits) over an account’s entries. Because summing 20K entries on every read is absurd at 100K reads/sec, we keep a materialized balance column that is updated in the same ACID transaction as the entry inserts, plus periodic snapshot rows so a rebuild does not scan from the beginning of time.

The whole money movement is one database transaction:

BEGIN;
  -- guard against overdraft AND concurrent races in one shot:
  UPDATE accounts
     SET balance = balance - :amount, version = version + 1
   WHERE account_id = :from
     AND balance >= :amount;          -- conditional: 0 rows => insufficient funds
  -- if 0 rows affected -> ROLLBACK, return "insufficient funds"

  UPDATE accounts
     SET balance = balance + :amount, version = version + 1
   WHERE account_id = :to;

  INSERT INTO ledger_entries (tx_id, account_id, direction, amount) VALUES
    (:tx, :from, 'DEBIT',  :amount),
    (:tx, :to,   'CREDIT', :amount);   -- must sum to zero

  INSERT INTO transactions (tx_id, type, state, ...) VALUES (:tx, 'TRANSFER', 'POSTED');
COMMIT;

Why this is right:

  • The conditional UPDATE ... WHERE balance >= amount is the concurrency guard. The database applies row locks so two concurrent debits serialize; the second one either still passes the check or fails it and rolls back. The balance can never go negative and there is no lost update. No separate lock service needed - the row lock and the WHERE do it.
  • Atomicity is free because debit, credit, entries, and transaction row are all in one ACID transaction. Partial states are impossible: either the whole transfer commits or none of it does.
  • Auditability is inherent. Every rupee’s path is a chain of immutable entries. A dispute or an audit is a query, not an archaeology dig.
  • The balancing invariant is checkable. A background job (and reconciliation) asserts SUM(all entries) == 0 globally and materialized_balance == SUM(entries) per account. If it ever fails, we have a bug and we know immediately, before it compounds.
  • Reads scale separately. Balance reads hit the cached materialized balance (Redis, invalidated on write via the version); history reads hit the async history projection. The hot ledger shards are protected from the 100:1 read flood.

For a top-up or withdrawal where the other side is an external bank, the “other” ledger account is a system account (e.g. bank_settlement or gateway_clearing). Double-entry still holds: crediting a user’s wallet on top-up debits the gateway_clearing system account, so the books always balance and the system accounts are exactly what reconciliation checks against the provider.

Money as an integer count of minor units (paise), never a float - floats lose pennies and pennies are the whole game.

3. Exactly-once processing across queues and external providers

The job: the settlement leg that talks to an external bank/gateway must charge/pay the outside world exactly once, even though queues are at-least-once and external calls can fail ambiguously.

Naive approach: consume the event, call the bank, hope

The Settlement Worker consumes transaction.posted, calls the gateway, done.

def on_event(evt):
    gateway.charge(evt.card, evt.amount)   # external call
    # commit offset

Where it breaks:

  • At-least-once queues. The worker charges the card, then crashes before committing its Kafka offset. On restart the event is redelivered and the card is charged again. Queues do not do exactly-once; assuming they do double-charges the customer.
  • Ambiguous external failures. You call the gateway; the connection drops before you read the response. Did the charge go through? Retrying might double-charge; not retrying might lose the top-up. The error alone cannot tell you.
  • No record of the external attempt. With only fire-and-hope, there is no state saying “we already tried this against the provider,” so every retry is blind.

At-least-once delivery plus a non-idempotent external call equals double-charges - the single worst outcome in a payment system.

Evolved approach: idempotency toward the provider + a transaction state machine + the outbox

Make exactly-once the composition of three things: an idempotency key passed to the provider, a persisted state machine per transaction, and an outbox so we never lose the intent.

1. State machine per transaction. The external leg is not one action; it is a tracked lifecycle. The transaction row carries a state, and the worker only advances it on confirmed transitions.

INITIATED -> PENDING_GATEWAY -> (SETTLED | FAILED) -> [REVERSED]

Before calling the gateway, the worker writes PENDING_GATEWAY with the provider idempotency key it will use. It only writes SETTLED/FAILED when the provider confirms. A crash mid-flight leaves the row in PENDING_GATEWAY, which tells recovery exactly what to reconcile - not “did anything happen.”

2. Idempotency key toward the provider. Every serious gateway (Stripe, Razorpay, bank APIs) accepts an idempotency key. We generate a deterministic key per transaction leg (e.g. tx_id + ":settle") and pass it on every attempt, including retries after an ambiguous failure. The provider then guarantees it charges once no matter how many times we call. This is what makes the ambiguous “connection dropped” case safe: retry with the same key; the provider either completes the original or reports its already-known result.

def settle(tx):
    if tx.state == 'SETTLED': return           # already done, no-op
    db.set_state(tx.id, 'PENDING_GATEWAY')
    result = gateway.charge(
        amount=tx.amount, source=tx.source,
        idempotency_key=f"{tx.id}:settle")      # SAME key on every retry
    if result.status == 'success':
        with db.transaction():
            ledger.post_settlement_entry(tx)    # move gateway_clearing -> settled
            db.set_state(tx.id, 'SETTLED')
    elif result.permanent_failure:
        with db.transaction():
            ledger.post_reversal(tx)            # give the money back
            db.set_state(tx.id, 'FAILED')
    else:
        raise RetryableError                    # transient: re-queue with backoff

3. The outbox pattern for the event itself. How does transaction.posted reliably reach Kafka if the DB commit and the Kafka publish are two systems (dual-write problem)? If we commit to the DB then publish to Kafka and crash in between, the settlement never happens - money is stuck. The fix: in the same ACID transaction as the ledger write, insert a row into an outbox table. A separate relay process reads the outbox and publishes to Kafka, marking rows sent. The DB commit is the single source of truth; the event is guaranteed to be published at-least-once because it is durably queued in the same transaction that moved the money. Consumers dedupe on tx_id, giving effectively exactly-once.

BEGIN;
  ... ledger entries + balance update ...
  INSERT INTO outbox (tx_id, topic, payload) VALUES (:tx, 'transaction.posted', ...);
COMMIT;
-- relay: SELECT unsent FROM outbox -> publish to Kafka -> mark sent

Typed retries and a DLQ. The worker classifies provider responses: transient (5xx, timeout, throttle) -> re-queue with exponential backoff and jitter; ambiguous (dropped mid-call) -> retry safely because of the provider idempotency key; permanent (declined, invalid account) -> post a reversal and mark FAILED. After N transient attempts the leg goes to a Dead Letter Queue for a human, because a stuck payment is a real-money problem, not a log line.

Why this is right: exactly-once against an external system is impossible to get from the queue alone, so we build it from pieces that each hold - the provider idempotency key makes the external charge once, the state machine makes recovery deterministic, and the outbox makes the event durable with the money. Together they turn an at-least-once pipeline over an unreliable bank into effectively-once money movement.

4. Reconciliation

The job: even with everything above, our ledger and the outside world can drift - a webhook is missed, a gateway silently succeeds a call we recorded as failed, a bank reverses a settlement. Reconciliation is the daily proof that our books match reality, and the process that fixes them when they do not.

Naive approach: trust our own ledger

Assume that if our code is correct and idempotent, our ledger must match the bank. Do nothing.

Where it breaks:

  • Missed webhooks. The gateway’s “settled” callback never arrived (their retry gave up, our endpoint had a blip). Our ledger says PENDING, the bank says paid. Real money is in a state we do not know about.
  • Provider-side reversals and chargebacks happen entirely outside our flow - the bank claws money back days later. Nothing in our synchronous path ever learns about it.
  • Fees and rounding. The gateway takes a fee we did not model exactly; over millions of transactions the difference is real money.
  • Bugs. Any latent bug that violates the balancing invariant is invisible until someone counts the money.

“Trust the ledger” means the first time you learn your books are wrong is when a regulator or an auditor tells you, which is the most expensive possible way to find out.

Evolved approach: automated double-sided reconciliation with a control ledger and break management

Reconciliation is a daily (and intraday) batch job that compares two independent records of the same money and resolves every difference.

  • Source A - our ledger. All entries against the system/clearing accounts for the period (gateway_clearing, bank_settlement).
  • Source B - the provider statement. The end-of-day settlement file / statement API the gateway and bank produce, listing every transaction they actually processed and its fee.
  • Match. Join the two on a shared key (the provider transaction id we stored, or the idempotency key we passed). Three outcomes per line:
Case Meaning Action
Matched our entry == provider line (amount, status) mark reconciled, no action
Missing in ledger provider processed money we have no entry for investigate; post a catch-up entry (e.g. a settlement we missed the webhook for)
Missing at provider our ledger shows a settlement the provider never did the external leg never really happened; reverse our entry or re-drive settlement
Amount/fee mismatch same tx, different amount post a fee/adjustment entry to the difference account; alert if large
  • The control (suspense) account. Discrepancies are not silently absorbed. Each break posts a balancing entry to a dedicated suspense account so the ledger stays balanced while a human investigates, and the suspense account’s balance is a live measure of unreconciled money - it should trend to zero. A growing suspense balance is an alarm.
  • Break management workflow. Every unmatched line becomes a break with a status (open, investigating, resolved) and an owner. Auto-resolvable breaks (a late webhook that we can now confirm) are closed by the job; the rest go to an ops queue. Nothing is closed without a corresponding ledger action, so the audit trail is complete.
  • Internal integrity checks run alongside: assert global SUM(all ledger entries) == 0 and per-account materialized_balance == SUM(entries). These catch our own bugs independent of the provider.
For each provider_line in statement:
    entry = ledger.find_by_provider_id(provider_line.id)
    if entry is None:
        open_break('MISSING_IN_LEDGER', provider_line)      # we owe an entry
    elif entry.amount != provider_line.amount:
        post_adjustment(entry, provider_line)               # fee/rounding delta
        if delta_large: open_break('AMOUNT_MISMATCH', ...)
    else:
        mark_reconciled(entry)
For each ledger_entry not seen in statement:
    open_break('MISSING_AT_PROVIDER', ledger_entry)         # phantom settlement

Why this is right: idempotency and the state machine make us internally correct, but they cannot detect a divergence from an external system we do not control. Reconciliation is the only thing that does. It turns “we hope the books are right” into “we prove the books are right every day, and every discrepancy is a tracked, owned break with a balancing entry.” For a payment system this is not optional polish - it is the difference between a system you can operate and audit and one that quietly rots until it fails a regulator.

API Design & Data Schema

API

The money-movement endpoints (all mutating calls require an Idempotency-Key):

POST /api/v1/transfers
Headers: Authorization: Bearer <token>
         Idempotency-Key: <client-generated UUID, reused on retries>
Body:
{
  "from_account": "acc_alice",
  "to_account":   "acc_bob",
  "amount":       50000,               // integer minor units (paise) = 500.00
  "currency":     "INR",
  "note":         "dinner"
}
Response 201:
{
  "transaction_id": "tx_9f3...",
  "state": "POSTED",
  "amount": 50000,
  "from_balance_after": 120000,
  "created_at": "2026-07-06T10:04:00Z"
}
Errors:
  400  invalid payload
  402  insufficient funds
  401  bad token
  409  idempotency key still in progress (retry shortly)
  422  risk declined

POST /api/v1/topups        {source: card/upi ref, amount, currency, Idempotency-Key}
                           -> 202 {transaction_id, state: "PENDING_GATEWAY"}
POST /api/v1/withdrawals   {to_bank_account, amount, currency, Idempotency-Key}
                           -> 202 {transaction_id, state: "PENDING_GATEWAY"}

Reads and status:

GET  /api/v1/accounts/{id}/balance          -> {balance, currency, as_of}
GET  /api/v1/accounts/{id}/transactions     -> paginated history (from read model)
GET  /api/v1/transactions/{tx_id}           -> full state + ledger entries
POST /api/v1/webhooks/{provider}            -> gateway status callbacks (settled/failed/chargeback)

Top-ups and withdrawals return 202, not 201 - we accepted and posted the internal legs, but the external settlement is async and its final state arrives via webhook and reconciliation. Transfers between two internal wallets are fully synchronous and return 201 because the whole thing committed in one ledger transaction.

Data stores

Different access patterns, different stores. Be explicit.

1. Ledger + accounts + transactions - SQL, ACID, sharded, synchronously replicated. This is the one place the whole design depends on strong consistency, row locks, and multi-row atomic transactions. A relational database (PostgreSQL / a distributed SQL like CockroachDB or Spanner) is the correct and non-negotiable choice here - the double-entry write, the balance >= amount guard, and the same-transaction idempotency + outbox all rely on ACID. NoSQL’s eventual consistency would reintroduce exactly the races we designed out.

Table: accounts
  account_id     BIGINT   PRIMARY KEY
  user_id        BIGINT   INDEX
  currency       CHAR(3)
  balance        BIGINT   -- minor units; materialized projection
  version        BIGINT   -- optimistic concurrency
  status         STRING   -- active|frozen|closed
  Shard by: account_id

Table: ledger_entries                 -- APPEND ONLY, immutable
  entry_id       UUID     PRIMARY KEY
  tx_id          UUID     INDEX
  account_id     BIGINT   INDEX
  direction      CHAR(1)  -- D | C
  amount         BIGINT   -- minor units, always positive; direction gives sign
  currency       CHAR(3)
  created_at     TIMESTAMP
  Shard by: account_id    -- an account's entries live together for balance rebuild
  Invariant: per tx_id, SUM(credits) - SUM(debits) = 0

Table: transactions                   -- one row per logical operation (state machine)
  tx_id          UUID     PRIMARY KEY
  type           STRING   -- transfer|topup|withdrawal|reversal|adjustment
  state          STRING   -- INITIATED|POSTED|PENDING_GATEWAY|SETTLED|FAILED|REVERSED
  provider_ref   STRING   NULL     -- gateway/bank transaction id
  provider_idem  STRING   NULL     -- key we passed to the provider
  amount         BIGINT
  created_at     TIMESTAMP
  updated_at     TIMESTAMP
  Shard by: tx_id

Table: idempotency_keys
  idem_key       STRING   PRIMARY KEY   -- UNIQUE; the real dedup guarantee
  tx_id          UUID     NULL
  status         STRING   -- IN_PROGRESS | DONE
  response       JSON     -- stored original response for replay
  created_at     TIMESTAMP
  TTL/archive: 48h hot

Table: outbox
  outbox_id      BIGINT   PRIMARY KEY
  tx_id          UUID
  topic          STRING
  payload        JSON
  sent           BOOL     INDEX (partial: WHERE sent=false)
  created_at     TIMESTAMP

2. Balance cache - Redis. account_id -> {balance, version}. Serves the 100K/sec read flood; updated/invalidated on each ledger commit using the version so a stale read is detectable. On a miss, read the materialized balance from SQL. It is a cache, never the source of truth.

3. Transaction history read model - NoSQL wide-column (Cassandra / DynamoDB), by user. History is high-volume, append-mostly, read by user_id in reverse-chronological pages, never joined, and can be eventually consistent. Built by the History Projector off the event bus. Keeping it out of the SQL ledger protects the hot transactional shards from history scans.

Table: user_history
  user_id        BIGINT   PARTITION KEY
  tx_id          UUID     CLUSTERING KEY (created_at DESC)
  type, amount, counterparty, state, created_at
  Shard by: user_id

4. Reconciliation store - SQL. Provider statement lines, match results, and the breaks workflow (status, owner, resolving entry). Relational because it is queried, joined against the ledger, and needs consistency for the breaks state.

Why SQL for the core, NoSQL for history: the ledger write needs multi-row ACID, row-level locking for the overdraft guard, and same-transaction idempotency+outbox - that is precisely what relational databases exist for, and it is worth giving up some horizontal-scale convenience to get it (we shard the SQL by account_id to scale writes). History and balance reads have no such needs - they are read-heavy, key-scoped, and tolerant of eventual consistency - so they go to a cache and a wide-column store that scale reads cheaply. Match the consistency guarantee to the data: absolute for money, relaxed for the projections of it.

Bottlenecks & Scaling

Where it breaks first, in order, and the fix for each.

1. Double-spend under client retries and queue redelivery (breaks first, worst impact). The failure users actually feel and that costs real money. Fix: client Idempotency-Key with a DB unique-constraint backstop and same-transaction result storage; a Redis fast path in front for cheap dedup. Toward external providers, pass a deterministic idempotency key on every attempt so the gateway charges once. Consumers dedupe events on tx_id.

2. Hot account / write contention on a popular wallet. A merchant wallet or a system clearing account receives thousands of credits/sec; row locks on that single row serialize everything and throughput collapses. Fix: balance sharding for hot accounts - split a hot account’s balance into N sub-balance rows (account_id, shard_0..N), route each write to a random shard to spread lock contention, and sum the shards for a read. The double-entry invariant still holds per shard. For merchant settlement, batch/aggregate incoming credits and post them in coarser ledger entries. Never let all writes to one logical account hit one row.

3. Ledger write throughput and the ACID core. The strongly-consistent SQL core is the scaling ceiling; you cannot just add read replicas for writes. Fix: shard the ledger by account_id so writes spread across shards. A transfer that spans two shards needs a two-phase / distributed transaction (or a Saga: reserve on the debit shard, then credit, with compensation on failure) - keep these rarer by co-locating a user’s related accounts. Reads never hit the primary: they come from the balance cache and the history read model. Keep the ACID transaction tiny (a handful of rows) so lock hold time is minimal.

4. Read flood (100:1 read:write) hitting the ledger. Summing entries per read, or every balance check touching SQL, would crush the primaries. Fix: materialized balance + Redis cache for balance; a separate NoSQL history read model for transaction lists. The hot SQL shards see only writes and cache-miss reads.

5. The dual-write problem (DB commit vs Kafka publish). Committing money then failing to publish the event leaves settlement stuck. Fix: the outbox pattern - the event is inserted in the same ACID transaction as the ledger write, and a relay publishes it at-least-once. The DB is the single source of truth; the event can never be lost relative to the money.

6. External provider outage / ambiguous failures. The bank/gateway is down or answers ambiguously; the user’s top-up or withdrawal is in limbo. Fix: the state machine (PENDING_GATEWAY) makes in-flight legs recoverable; typed retries with exponential backoff and jitter; a circuit breaker and provider failover where a secondary exists; a DLQ for legs that exhaust retries. Reconciliation catches anything the async path missed.

7. Ledger drift from the outside world (missed webhooks, chargebacks, fees). Internal correctness cannot detect external divergence. Fix: daily and intraday reconciliation against provider statements, a suspense account that keeps the ledger balanced while breaks are worked, and a break-management workflow so every discrepancy is owned and resolved with a balancing entry. A rising suspense balance is a paging alarm.

8. Storage growth (~20TB/year, never deleted). Regulatory retention forbids purging, so the ledger only grows. Fix: time-partition the ledger; keep recent (~90d) partitions on fast storage, roll older ones to cheaper cold storage that is still queryable. Balance snapshot rows cap how far a rebuild must scan. Archive idempotency keys aggressively (48h hot) since their retry window is short.

9. Single points of failure. Any single stateful node loss must not stop money or lose it. Fix: SQL ledger with synchronous replication and automatic failover (no committed transaction lost on a node death); Kafka and Redis replicated; stateless API/workers horizontally scaled behind load balancers; the DLQ and outbox mean nothing in flight vanishes silently.

Shard keys, stated plainly: ledger_entries, accounts -> account_id (an account’s entries co-located for rebuild; writes spread across accounts). transactions, idempotency_keys -> tx_id / key. user_history -> user_id (single-partition reverse-chronological reads). Kafka topics -> partition by account_id (per-account ordering + even spread). Balance buckets for hot accounts -> (account_id, shard_n). Never shard the ledger by time - it concentrates all current writes on the newest partition, a permanent hot spot.

Wrap-Up

The trade-offs that define this design:

  • The ledger is truth; the balance is a cache. We store money as an append-only, double-entry ledger and treat the balance as a materialized projection. We trade the simplicity of a mutable balance column for auditability, provable conservation of money, and the ability to reconstruct any state - non-negotiable for money.
  • A tiny strongly-consistent core, everything else async. The only strongly-consistent operation is one ACID ledger transaction with a balance >= amount guard. Reads, history, settlement, and notifications are pushed to caches, projections, and workers. We keep the expensive consistency small so it stays fast and scalable.
  • Idempotency everywhere money moves. A client key with a DB unique-constraint backstop makes retries safe; a deterministic provider key makes external charges once; the outbox makes the event durable with the money. At-least-once plumbing plus idempotency equals exactly-once money movement.
  • Assume drift, then prove it away. Idempotency and the state machine keep us internally correct, but only daily reconciliation against provider statements - with a suspense account and owned breaks - proves the books match reality and fixes them when they do not.
  • SQL for the ledger, NoSQL for the projections. We match the storage guarantee to the data: ACID for money, eventual consistency for the read models of it, and we shard both by account_id/user_id to scale.

One-line summary: an idempotent Payment API fronts a small strongly-consistent double-entry ledger that moves money in one ACID transaction under an overdraft guard, emits events via an outbox to async settlement workers that charge external providers exactly-once with deterministic keys and a per-transaction state machine, and is proven correct every day by reconciliation against provider statements - so money moves once, is always conserved, and is always auditable, even across retries, crashes, and unreliable banks.