“Scan a QR, money leaves your HDFC account and lands in a shopkeeper’s SBI account in three seconds” sounds like one API call. It is not. HDFC and SBI are two different companies with two different databases that will never share a transaction. There is no UPDATE that touches both. In between sits a switch (NPCI) that neither holds the money nor owns either account, yet has to guarantee that the payer’s account is debited exactly when and only when the payee’s account is credited - across two banks, over a network, inside a 5-second timer, ten billion times a month, and it has to be provably correct afterwards when the regulator asks. The one thing you cannot do is the obvious thing: a single distributed transaction that locks both banks. Banks will not give you a lock. So the entire design is about moving money atomically-in-effect across systems that refuse to coordinate.

The core of this problem is that money must move between two independent banks with the guarantee “debit and credit both happen, or neither leaves a lasting effect,” using only asynchronous messages, a hard 5-second deadline, and after-the-fact reconciliation. A debit with no credit is money that vanished from a real person’s account. A credit with no debit is money the system printed. A stuck “did it go through?” state is a support call and a frozen balance. Every mechanism here - the switch as coordinator, idempotency by transaction id, the deemed-transaction rule, the reversal path, and net settlement - exists to make cross-bank money movement atomic in effect, bounded in time, and auditable. Let me build it properly.

Functional Requirements (FR)

In scope:

  • Address a payee without sharing a bank account number. A user is reachable by a Virtual Payment Address (VPA / “UPI ID” like alice@okhdfc), a scanned QR code, or a phone number, which resolves to a real account at a real bank. The account number never leaves the bank.
  • Link an account. A user links their bank account to a PSP app (PhonePe/GPay/BHIM) once, proving ownership via the bank, and sets a UPI PIN held by the bank, not the app.
  • Pay (push). Payer scans a QR or enters a VPA, enters amount and UPI PIN, and money moves from the payer’s bank account to the payee’s bank account. This is the hot path.
  • Collect (pull). A payee requests money from a payer’s VPA; the payer approves with a PIN. Same money movement, opposite initiator.
  • Move money across two different banks in one logical transaction with strong consistency: the payer’s debit and the payee’s credit either both take lasting effect or neither does.
  • Complete within the SLA. The user-facing result (success/failure) returns inside ~5 seconds, and an ambiguous timeout is resolved deterministically, never left dangling.
  • Settle and reconcile. Banks do not actually move real money on every transaction; they net their positions and settle in batches, and every transaction is reconciled so the books across all banks agree.

Explicitly out of scope (say it so you own the scope):

  • The bank’s own core banking ledger. Each bank debits/credits its own accounts inside its own core. I design the switch and the PSP, and I define the contract with the bank’s UPI adapter, but I do not build the bank’s core.
  • KYC, onboarding, and the fraud/risk engine. I assume a risk decision is an input (a transaction is allowed or blocked by a separate service). I do not build the fraud model, though I name where it hooks in.
  • UPI mandates, autopay, credit-line-on-UPI, international UPI. Real features, but I design the core P2P/P2M real-time transfer first and name the extensions.
  • The RTGS/NEFT rails underneath settlement. I net positions and hand a settlement file to the central bank’s account; the actual inter-bank fund transfer at the central bank is a separate rail I treat as a black box.

The decision that drives everything: the switch is a stateful coordinator that owns the transaction lifecycle, not the money. Banks hold the money and own their accounts; the switch owns the two-phase-ish protocol that makes a debit at bank A and a credit at bank B behave as one atomic transfer, and owns the timeout and reversal rules that keep it correct when a bank is slow or unreachable. Every hard requirement follows from that.

Non-Functional Requirements (NFR)

  • Scale: ~10B transactions/month. That is ~333M/day, ~3,900 tx/sec average, with peaks (festival sale, salary day, month end) at 4-8x average, so design for ~25,000-30,000 tx/sec at the switch. ~300M+ users, ~500M+ linked accounts and VPAs.
  • Latency: the user-facing round trip (PIN entered -> success screen) should be p99 under 5 seconds end to end, which includes two round trips to two different banks’ cores. The switch’s own added latency should be tens of milliseconds; the banks are the slow part, so the SLA budget is mostly “how long we wait for a bank.”
  • Availability: 99.99%+ for the switch. But UPI availability is a chain: the switch, the payer PSP, the payer’s bank, and the payee’s bank must all be up for a given transaction. So the design must degrade gracefully per bank - one bank being down fails only transactions touching that bank, never the whole network - and publish per-bank health so PSPs can route or warn.
  • Consistency: strong for money. A debit without a matching credit, or a credit without a debit, is unacceptable and must be impossible to leave permanent. This is the one place we do not trade consistency for availability: when in doubt, we reverse to the safe state (money back to payer) rather than guess. Reads (transaction status, history) can be eventually consistent.
  • Durability: absolute. Every transaction’s state transitions are journaled durably and synchronously replicated before they are acted on. Nothing accepted is ever lost; corrections are new records, never edits.
  • Auditability and compliance: every transaction is traceable end to end for years (regulatory retention 7-10 years), every message signed and non-repudiable, and the whole system reconciled daily so the regulator’s question “did this 500 rupees actually move” always has a proven answer.

The tension to state up front: strong consistency across two banks wants a distributed lock or a 2PC coordinator that both banks obey, but banks will not hold locks for you and can time out or vanish mid-transaction. We resolve it by making the switch a coordinator that drives a debit-then-credit sequence with a hard deadline, a deterministic timeout rule (the “deemed” transaction), and an automatic reversal path - so the system is always driven to a correct terminal state (SUCCESS or fully-reversed FAILURE) even when a bank is slow or unreachable, and net settlement plus reconciliation prove it after the fact.

Back-of-the-Envelope Estimation (BoE)

Real numbers with the arithmetic, because they dictate the architecture.

Transaction throughput:

10,000,000,000 transactions / month
/ 30 days              = 333,000,000 / day
/ 86,400 sec           ≈ 3,858 tx/sec average at the switch
Peak (4-8x average, festival / salary day) ≈ 25,000-30,000 tx/sec

Message fan-out per transaction. One user “pay” is NOT one message. A single pay transaction is a small orchestration:

1  PSP -> Switch:            ReqPay (the transaction request)
2  Switch -> Remitter bank:  ReqDebit (debit payer, verify PIN)
3  Remitter bank -> Switch:  RespDebit
4  Switch -> Beneficiary bank:ReqCredit (credit payee)
5  Beneficiary bank -> Switch:RespCredit
6  Switch -> PSP:            RespPay (final result)
   (+ async status/receipt notifications)
So ~6 switch-touching messages per transaction, sometimes more
(VPA resolution, reversals).
Peak message rate ≈ 30,000 tx/sec * 6 ≈ 180,000 messages/sec at the switch.

The switch is not doing 30K ops/sec; it is doing ~180K message hops/sec, each of which mutates a durable transaction state. That message multiplier, not the headline tx count, sizes the switch.

Storage per year. Per transaction we persist the full lifecycle - request, both bank legs, responses, signatures, final state:

txn_id            32 bytes
payer_vpa         ~30 bytes
payee_vpa         ~30 bytes
amount            8 bytes (integer paise, never float)
payer_bank, payee_bank, refs  ~40 bytes
state + timestamps (per transition, ~5 transitions * 16)  ~80 bytes
signatures / message blobs (kept for non-repudiation)   ~600 bytes
----------------------------------------------------------------
≈ 800 bytes-1 KB per transaction, round to 1 KB with indexes/overhead.

333M tx/day * 1 KB ≈ 333 GB/day
* 365 ≈ 120 TB/year

Retained 7-10 years for compliance, so plan ~1 PB over the window; recent ~90 days hot, older partitioned to cold-but-queryable storage. This is a partitioning and cold-storage story, not a “fits in one box” story.

VPA / mapper store. The address book that resolves alice@okhdfc to (bank, account token):

500M VPAs * (vpa ~40 + account_token 32 + bank_id 4 + status 1 + meta ~20)
 ≈ 500M * ~100 bytes ≈ 50 GB

Tiny and read-heavy - every transaction resolves the payee VPA at least once. A perfect fit for a sharded key-value store with an aggressive cache.

Bandwidth. Each message ~1-2 KB signed. 180K messages/sec * ~1.5 KB ≈ 270 MB/sec ≈ ~2.2 Gbps steady at peak across the switch, spread over many banks’ links. Modest; the hard part is latency and correctness, not raw bytes.

The headline: raw throughput (~30K tx/sec, ~180K msgs/sec) is large but not exotic. The genuinely hard parts are moving money atomically across two banks that share no transaction, doing it inside a 5-second deadline, resolving the inevitable timeouts deterministically without ever stranding money, and proving the whole thing reconciles across every bank daily. That is where the design earns its keep.

High-Level Design (HLD)

The spine: PSP apps (PhonePe et al.) that capture intent and talk to their Payer/Payee PSP backends; a central UPI Switch (the NPCI role) that is the stateful coordinator of every transaction and the only party that talks to all banks; a Mapper that resolves VPAs to bank accounts; Bank UPI adapters at each bank that translate switch messages into core-banking debits/credits and verify the UPI PIN; and an asynchronous Settlement & Reconciliation subsystem that nets positions and proves the books match.

   Payer (PhonePe app)                          Payee (shop QR / VPA)
        │  scan QR -> VPA + amount, enter UPI PIN
        ▼
 ┌──────────────────────────┐
 │   Payer PSP backend       │  session, device binding, build ReqPay,
 │   (PhonePe)               │  encrypt PIN block (bank public key)
 └───────────┬──────────────┘
             │ ReqPay {txn_id, payer_vpa, payee_vpa, amount, encPIN}
             ▼
 ┌───────────────────────────────────────────────────────────────┐
 │                     UPI SWITCH  (NPCI)                          │
 │  - the ONLY coordinator; owns the transaction state machine     │
 │  - stateless request handlers + durable Txn Store               │
 │  - resolve payee VPA via Mapper                                 │
 │  - drive: debit remitter -> credit beneficiary -> respond       │
 │  - enforce 5s deadline, deemed rule, reversals                  │
 └───┬───────────────┬────────────────────┬──────────────┬────────┘
     │ resolve VPA   │ ReqDebit            │ ReqCredit    │ emits events
     ▼               ▼                     ▼              ▼
 ┌────────┐   ┌──────────────┐      ┌──────────────┐  ┌──────────────┐
 │ Mapper │   │ Remitter Bank│      │ Beneficiary  │  │ Event log     │
 │ VPA -> │   │ UPI adapter  │      │ Bank adapter │  │ (Kafka)       │
 │ acct   │   │  verify PIN  │      │  credit acct │  └──────┬───────┘
 │ (KV)   │   │  debit acct  │      │              │         │
 └────────┘   └──────┬───────┘      └──────┬───────┘  ┌──────▼───────────┐
                     │ core banking        │ core     │ Settlement &     │
                ┌────▼──────┐         ┌─────▼─────┐    │ Reconciliation   │
                │ Bank A    │         │ Bank B    │    │ - net positions  │
                │ core CBS  │         │ core CBS  │    │ - EOD recon file │
                └───────────┘         └───────────┘    │ - UDIR disputes  │
                                                       └──────┬───────────┘
                                                              │ net file
                                                       ┌──────▼───────────┐
                                                       │ Central bank RTGS │
                                                       │ settlement account│
                                                       └──────────────────┘

Pay flow (push, the hot path, the switch is the coordinator):

  1. Payer scans the payee’s QR (which encodes the payee VPA + optional amount) or types a VPA, enters the amount and their UPI PIN. The PSP app encrypts the PIN into a PIN block using the bank’s public key - the PSP and switch never see the plaintext PIN. It builds a ReqPay with a client-generated txn_id (the idempotency key for the whole transaction).
  2. Payer PSP sends signed ReqPay to the Switch. The switch validates the signature, resolves the payee VPA -> (beneficiary bank, account token) via the Mapper, writes a transaction row in state INITIATED, and durably persists it. This row is the coordinator’s source of truth for the rest of the flow.
  3. The switch sends ReqDebit to the remitter (payer’s) bank adapter. The bank verifies the encrypted PIN against its records, checks funds and risk, debits the payer’s account inside its own core in one local ACID transaction, and returns RespDebit(SUCCESS). The switch moves the transaction to DEBITED.
  4. On a successful debit, the switch sends ReqCredit to the beneficiary (payee’s) bank adapter, which credits the payee’s account in its own core and returns RespCredit(SUCCESS). The switch moves the transaction to SUCCESS.
  5. The switch returns RespPay(SUCCESS) up to the payer PSP, which shows the success screen and pushes a receipt notification to both users. The whole thing has to finish inside ~5 seconds.

The unhappy paths the coordinator must own:

  • Debit fails (wrong PIN, insufficient funds, risk block) -> transaction goes to FAILED, no money moved, respond failure. Clean.
  • Debit succeeds but credit fails or times out -> money left the payer but did not reach the payee. The switch must drive a reversal (RevDebit) to put money back in the payer’s account, ending in FAILED (reversed). This is the dangerous case and the reason the switch is stateful.
  • Any leg times out past the deadline -> the transaction is marked deemed and resolved deterministically (deep dive 3), never left in-flight.

Async flow (settlement, reconciliation, notifications):

  1. Every state transition emits an event to the log. A notification service consumes it to update transaction status and send receipts, so status reads never touch the hot switch path.
  2. The Settlement service aggregates, per bank pair, all successful transactions in a cycle and computes each bank’s net position (net debtor/creditor). Only the netted amounts are actually moved between banks at the central bank, not every transaction.
  3. The Reconciliation service ingests each bank’s own transaction log and compares it line by line against the switch’s ledger, driving the automated dispute resolution (UDIR) flow for any mismatch.

The key structural insight: the switch is a coordinator that owns a transaction state machine and drives it to a terminal state under a deadline, but never holds the money. Banks do local ACID debits/credits in their own cores; the switch stitches those two local transactions into one atomic-in-effect transfer using ordering (debit before credit), a hard timeout, and a reversal path. No global lock, no 2PC that banks must obey - just a coordinator that is relentless about reaching SUCCESS or fully-reversed FAILURE, and reconciliation that proves it later.

Component Deep Dive

The hard parts, naive-first then evolved: (1) VPA addressing and resolution, (2) atomic money movement across two independent banks, (3) idempotency and the 5-second deadline / deemed transactions, (4) net settlement and reconciliation.

1. VPA addressing and QR resolution

The job: let a payer address a payee by alice@okhdfc, a QR, or a phone number - never by raw account number - and resolve that to the right bank and account, at 30K resolutions/sec, without leaking account numbers.

Naive approach: put the account number in the QR / VPA

The obvious first cut: the QR encodes the payee’s actual bank account number + IFSC, and the payer PSP sends money straight to it. A VPA is just a friendly alias stored in one central table vpa -> account_number.

QR encodes:  { account: "50100123456789", ifsc: "HDFC0001234", amount: 250 }
pay(): send debit/credit using the account number from the QR

Where it breaks:

  • It leaks account numbers. A printed QR on a shop counter now exposes the merchant’s full account details to anyone with a camera. Account numbers are sensitive; the whole point of a VPA is to keep them private.
  • It is unforgeable-only if signed, which a raw QR is not. Anyone can print a QR pointing at their own account and stick it over the shop’s - the payer has no way to know they are paying the wrong account. A static account-number QR has no integrity.
  • One central vpa -> account table is a hot, sensitive single point. Every transaction reads it; it holds every mapping in plaintext; a breach is catastrophic and it is a throughput bottleneck.
  • No indirection means re-linking is painful. If the merchant changes banks, every printed QR is now wrong, because it hard-coded the account.

Encoding the account into the address conflates addressing with the account itself, leaking data and removing the layer of indirection that makes the system safe and flexible.

Evolved approach: VPA as an indirection layer, tokenized accounts, signed QRs

Separate the address from the account. A VPA is <handle>@<psp> where the suffix (@okhdfc, @ybl, @axl) tells the switch which PSP owns the mapping, and the switch routes the resolution to that PSP’s mapper.

  • The Mapper stores VPA -> (bank_id, account_token), where the account_token is an opaque reference, not the raw account number. The raw number lives only inside the bank; the switch and PSPs carry a token. Resolving a VPA returns “beneficiary bank = HDFC, token = X”; the actual account is dereferenced only inside HDFC’s adapter.
  • Resolution is distributed by handle suffix. alice@okhdfc -> the switch asks the PSP/bank that owns okhdfc. This spreads load across banks/PSPs instead of one global table, and each bank owns (and secures) its own users’ mappings. The switch keeps a routing table of handle_suffix -> owner, which is tiny and cacheable.
  • QRs carry a VPA (and for merchants, a signed merchant identifier), not an account. A dynamic QR for a specific bill also carries an amount and a reference, and merchant QRs are signed so a payer PSP can verify the QR was issued by a registered merchant and was not tampered with. Static personal QRs encode just the VPA; the resolution still goes through the mapper, so re-linking a VPA to a new bank updates one mapping and every QR keeps working.
  • Aggressive caching. VPA -> (bank, token) is read every transaction and changes rarely, so it is cached hot (Redis) with a short TTL and invalidation on re-link. A cache hit avoids a network hop to the owning PSP and protects the 5-second budget.
VPA layout:   handle @ psp_suffix          e.g.  9876543210@ybl,  coffeeshop@okhdfc
Resolve(vpa):
  suffix   = vpa.split('@')[1]
  owner    = routing_table[suffix]          # which PSP/bank owns this handle space
  cached   = cache.get(vpa)                 # hot path: mostly a cache hit
  if cached: return cached
  mapping  = owner.resolve(vpa)             # -> {bank_id, account_token, name}
  cache.set(vpa, mapping, ttl=short)
  return mapping                            # NEVER the raw account number

Why this is right:

  • Account numbers never leave the bank. Addressing is done entirely in tokens and VPAs, so a QR or a mapper breach exposes references, not accounts.
  • The indirection layer decouples identity from account. Change banks, keep your VPA; the mapping updates in one place and every existing QR and saved payee still resolves. That is the whole value of a VPA over an account number.
  • Load spreads by handle owner instead of hammering one central table, and each bank secures its own users’ data, which is also the correct trust boundary.
  • Signed merchant QRs prevent the sticker-swap attack - the payer PSP verifies the QR’s signature and shows the verified merchant name before the user pays, so paying the wrong account requires forging a signature, not printing a sticker.

Addressing is not a lookup table; it is a security and flexibility layer, and getting it right is what lets the rest of the system stay both private and re-configurable.

2. Atomic money movement across two independent banks

The job: debit the payer at bank A and credit the payee at bank B so that both take lasting effect or neither does, when A and B share no database, no transaction, and no lock, and either can be slow or crash mid-flight.

Naive approach: one distributed transaction / two-phase commit across both banks

The textbook instinct: run a distributed transaction. The switch is the 2PC coordinator. Phase 1: ask bank A to prepare the debit and bank B to prepare the credit, both holding locks. Phase 2: if both voted yes, tell both to commit.

coordinator.begin(txn)
prepare_A = bankA.prepare_debit(payer, amount)     # holds a lock on payer account
prepare_B = bankB.prepare_credit(payee, amount)    # holds a lock on payee account
if prepare_A.ok and prepare_B.ok:
    bankA.commit(); bankB.commit()
else:
    bankA.abort();  bankB.abort()

Where it breaks:

  • Banks will not hold locks for an external coordinator. 2PC requires each participant to prepare and then hold a lock on the account until the coordinator says commit or abort. No real bank will let an outside switch pin a customer’s account balance for the duration of a network round trip. It is operationally and legally a non-starter across independent institutions.
  • Blocking on coordinator failure. Classic 2PC: if the coordinator crashes after prepare but before commit, participants are stuck holding locks indefinitely, unsure whether to commit or abort. At 30K tx/sec, a coordinator blip freezes accounts across the country.
  • The credit bank cannot un-credit cleanly. Even if you got 2PC, “prepare a credit” is awkward - the money is not really the payee’s until commit, so the payee sees a pending amount that might vanish. And banks post to real customer-visible balances, not a two-phase staging area.
  • Latency. Two prepares + two commits = four serial round trips to two banks, each of which can be slow. That blows the 5-second budget and holds locks the whole time.

2PC assumes cooperative participants that will block under a lock at a coordinator’s command. Independent banks are not that. The premise fails at the trust boundary.

Evolved approach: switch-coordinated debit-then-credit with reversal (a Saga with a deadline)

Give up on locking both banks at once. Instead, model the transfer as a sequence of two local, independently-committed transactions, ordered so that failure is always recoverable, driven by the switch as a persistent coordinator with a compensating action (reversal). This is a Saga, specialized for money and a hard deadline.

  • Order matters: debit first, then credit. The switch asks the remitter bank to debit the payer inside the bank’s own local ACID transaction and commit. The debit is real and committed at bank A - the money has left the payer. Only after a confirmed RespDebit(SUCCESS) does the switch ask the beneficiary bank to credit the payee, also as a committed local transaction at bank B. Debit-before-credit means the failure we can hit is “debited but not yet credited,” and that is fixable by reversal (give the money back). The opposite order - credit first - would risk crediting the payee and then failing the debit, which means the system printed money, and there is no one to claw it back from cleanly.
  • The switch persists the transaction state before each step. INITIATED -> DEBITED -> SUCCESS, each transition durably written before the next message goes out. The coordinator is stateful precisely so a switch crash mid-flight is recoverable: a recovery process reads any transaction stuck in DEBITED and drives it forward (retry credit) or back (reverse debit).
  • Credit failure triggers a compensating reversal. If the credit fails permanently or the beneficiary bank is unreachable past the deadline, the switch sends RevDebit to the remitter bank to credit the money back to the payer (a new, committed local transaction that compensates the debit), and the transaction ends in FAILED (reversed). The payer is made whole; the system is back to a consistent state with net-zero movement.
  • The credit is idempotent by txn_id. Because the switch may retry the credit after an ambiguous timeout, the beneficiary bank must apply the credit once per txn_id even if it receives ReqCredit twice. The bank stores the txn_id with a unique constraint (deep dive 3), so a retry after “did my RespCredit get lost?” credits once, not twice.
# Switch coordinator (simplified), each state persisted before the next call:
def pay(txn):
    persist(txn, INITIATED)
    resp_d = remitter_bank.debit(txn, deadline)         # local ACID at bank A
    if not resp_d.success:
        persist(txn, FAILED); return failure(resp_d)     # no money moved, clean
    persist(txn, DEBITED)                                # money has LEFT the payer

    resp_c = beneficiary_bank.credit(txn, deadline)      # local ACID at bank B
    if resp_c.success:
        persist(txn, SUCCESS); return success()
    else:
        # debited but not credited -> compensate
        remitter_bank.reverse(txn)                       # credit money BACK to payer
        persist(txn, FAILED_REVERSED); return failure()

Why this is right:

  • No cross-bank lock, no blocking 2PC. Each bank does a normal local committed transaction on its own schedule; the switch coordinates by ordering and compensation, not by pinning both accounts. This is the only shape that works across institutions that will not coordinate under a lock.
  • The only bad intermediate state is “debited, not credited,” and it is always recoverable by a reversal, because the debit is a real committed transaction that a real bank can compensate with an equal-and-opposite credit. We never end in “credited, not debited” because we never credit first.
  • Atomicity is achieved in effect, not in a single transaction. The user sees either SUCCESS (both legs done) or a failure where their money is back. There is no lasting state where money is missing or invented.
  • The stateful coordinator is what makes crash recovery deterministic. Every in-flight transaction has a persisted state, so a switch failover knows exactly which transactions to drive forward and which to reverse - “did anything happen” is never a question, the state says so.

This is the heart of the system: cross-bank atomicity as an ordered Saga (debit, then credit, else reverse) driven by a persistent, deadline-aware coordinator, using each bank’s own local ACID transaction as the unit that actually moves money.

3. Idempotency and the 5-second deadline (deemed transactions)

The job: every message can be lost, retried, or delayed, and there is a hard 5-second SLA. The system must move money exactly once per txn_id and must resolve every timeout to a definite terminal state, never leaving money in limbo.

Naive approach: fire the request, wait, retry on timeout

The switch sends ReqDebit, waits; if no response by the deadline, it retries. The bank processes whatever it receives. The PSP retries ReqPay if it does not hear back.

def debit(txn):
    resp = bank.send(ReqDebit(txn))     # no response by 5s?
    if timeout: resp = bank.send(ReqDebit(txn))   # retry - might debit AGAIN
    return resp

Where it breaks:

  • Retries double-debit. The bank debited the payer, but RespDebit was lost on the way back. The switch times out and retries ReqDebit; the bank, treating it as a new request, debits the payer a second time. The customer paid twice for one purchase. This is the single worst outcome.
  • The ambiguous timeout is unresolved. “No response in 5 seconds” does not tell you whether the debit happened. Retrying might double-debit; not retrying might mean the money left but the switch thinks it did not, and then it credits nobody or reverses a debit that never occurred. The timeout alone carries no information about the real state.
  • The user is stuck on a spinner. With no deterministic rule, a slow bank leaves the transaction “processing” for minutes, the user re-taps pay (another txn_id? the same?), and now there are duplicate transactions and a frozen balance.

At-least-once delivery plus a non-idempotent bank operation plus a hard deadline equals double-debits and stranded money - exactly what a payments network cannot have.

Evolved approach: txn_id idempotency at every hop + the “deemed” timeout rule + reversal

Make the txn_id (client-generated, unique per logical payment) the idempotency key end to end, and turn “timeout” from an ambiguous event into a deterministic rule with a guaranteed reconciling action.

1. Idempotency by txn_id at every party. The PSP generates one txn_id per pay attempt and reuses it on every retry of that same attempt. The switch and both banks each key their state on txn_id with a unique constraint:

  • The switch, on receiving a ReqPay for a txn_id it already has, returns the stored current state, it does not start a second transaction.
  • The remitter bank, on a second ReqDebit for a txn_id it already debited, returns the stored result and does not debit again. The unique constraint on txn_id is the real guarantee - two concurrent retries race to insert, exactly one wins, the other reads the winner’s result.
  • Same for the beneficiary bank on ReqCredit, and for the reversal on RevDebit (reverse once).
def bank_debit(req):
    try:
        db.insert(txn_id=req.txn_id, status='IN_PROGRESS')   # UNIQUE(txn_id)
    except UniqueViolation:
        row = db.get(req.txn_id)
        return row.stored_response          # replay; NEVER debit twice
    with db.transaction():                  # local ACID
        debit_account(req.payer, req.amount)
        db.update(req.txn_id, status='DONE', response=make_resp())
    return the response

2. The deadline is a hard, network-wide constant. The switch enforces a per-transaction deadline (say ~30s for the back office, but the user-facing target is 5s). Every leg carries the deadline; a bank must not act on a request whose deadline has passed and must respond within it or be treated as timed out.

3. The “deemed” rule turns ambiguity into a decision. When the switch does not get a definitive response within the deadline, it does not guess and it does not blindly retry a mutating call. It marks the transaction DEEMED (state unknown-at-this-hop) and applies a deterministic resolution:

  • Timeout on the debit response: the switch does not know if the payer was debited. It cannot credit the payee (that could print money). It marks the transaction for reversal/verification: it queries the remitter bank’s status for that txn_id (a safe, non-mutating status check the idempotency store answers exactly), and if the debit did happen, drives a reversal so the money returns to the payer. Terminal state: FAILED (reversed) or, if the debit never happened, FAILED. Either way, the payer is not out any money.
  • Timeout on the credit response: the payer is already debited (state is DEBITED). The switch queries the beneficiary bank for the txn_id: if the credit landed, mark SUCCESS; if not, reverse the debit and mark FAILED (reversed). Because both banks are idempotent on txn_id, the status query and any retry are safe.
  • Bias to the safe state: when a deemed transaction cannot be confirmed as fully SUCCESS, the safe resolution is always money back to the payer. A false failure (money returned that could have succeeded) is recoverable and cheap; a false success (money missing or invented) is not.

4. Deemed transactions are reconciled, not left to the user. A deemed transaction that could not be resolved synchronously within the deadline is handed to the reconciliation / UDIR path (deep dive 4), which finalizes it from both banks’ books, typically within minutes to hours, and the user sees a definite status and any reversal, automatically. The user is never asked to figure out whether their money moved.

Deadline reached, no definitive response:
  mark txn DEEMED
  status = bank.query_status(txn_id)     # SAFE: non-mutating, idempotency store answers
  if leg == DEBIT:
      if status == DEBITED: switch.reverse_debit(txn_id)  -> FAILED_REVERSED
      else:                 -> FAILED
  if leg == CREDIT:
      if status == CREDITED: -> SUCCESS
      else: switch.reverse_debit(txn_id)  -> FAILED_REVERSED
  if still unknown: hand to reconciliation (finalize from EOD books)

Why this is right:

  • Exactly-once money movement under retries comes from the txn_id unique constraint at every party - the network is at-least-once, but debit, credit, and reversal each apply once.
  • The ambiguous timeout becomes a decision, not a gamble. Instead of blindly retrying a mutating call, the switch does a safe status query and drives to the correct terminal state, always biased to returning money on doubt.
  • No stranded transactions. Every transaction reaches SUCCESS or a fully-reversed FAILURE, synchronously if possible, via reconciliation if not - so a user is never left staring at “processing” with money missing.
  • The 5-second SLA is a user-facing promise, backed by a back-office guarantee. If the synchronous path cannot confirm in time, the user gets a definite (possibly “pending, will resolve”) status and the deemed machinery finishes it correctly, rather than the system lying either way.

Idempotency plus the deemed rule is what makes a hard real-time deadline compatible with strong money consistency: you can time out the user without ever timing out the correctness.

4. Net settlement and reconciliation across banks

The job: banks do not shove real money between each other on every one of 333M daily transactions; they net and settle in cycles. And even with everything above, the switch’s view and each bank’s books can drift, so every transaction must be reconciled and every dispute resolved automatically.

Naive approach: settle each transaction instantly, trust the switch’s ledger

Move real inter-bank money per transaction (gross settlement), and assume that if the switch says SUCCESS, both banks agree, so no reconciliation is needed.

Where it breaks:

  • Gross settlement per transaction is impossibly expensive. 333M real inter-bank fund transfers a day at the central bank would swamp the RTGS rail and cost a fortune in settlement operations. Real networks net: sum up who owes whom over a cycle and move only the difference.
  • The switch’s ledger and a bank’s core will drift. A RespCredit that was lost after the bank committed leaves the switch thinking DEEMED/FAILED while the bank actually credited. A reversal that the switch sent but the bank dropped. A deemed transaction resolved one way at the switch and another at the bank. Over hundreds of millions of transactions, some fraction always diverges, and each divergence is real customer money in the wrong state.
  • Disputes have no home. A user says “debited but not received.” With no reconciliation, there is nothing that authoritatively decides what actually happened across two banks, so it becomes a manual, days-long support fight.

“Settle everything instantly and trust our own ledger” is both operationally impossible and blind to the inevitable drift between independent systems.

Evolved approach: netted settlement cycles + automated line-by-line reconciliation (UDIR)

Separate clearing (deciding who owes whom) from settlement (actually moving central-bank money), net across a cycle, and reconcile every transaction against both banks’ books with an automated dispute machine.

  • Net multilateral settlement in cycles. In each settlement cycle (UPI runs multiple per day), the switch aggregates every SUCCESS transaction and computes each bank’s net position: sum(credits into bank) - sum(debits out of bank). A bank that received more than it sent is a net creditor; the reverse is a net debtor. Only the net amount per bank is moved, once per cycle, through the banks’ pre-funded settlement accounts at the central bank. 333M transactions collapse into a few dozen net transfers.
For a cycle, per bank:
  net(bank) = Σ amount(txn where beneficiary_bank == bank and state==SUCCESS)
            - Σ amount(txn where remitter_bank    == bank and state==SUCCESS)
  Σ over all banks of net(bank) == 0        # money is conserved; the file must balance
Settlement file: one net debit/credit per bank -> central bank settlement accounts.
  • The switch holds the authoritative clearing ledger, but it is proven, not trusted. Each bank sends its own transaction log for the cycle; reconciliation joins the two on txn_id.
Case Meaning Action
Matched switch SUCCESS == bank debited/credited, same amount reconciled, include in net settlement
Bank credited, switch says failed lost RespCredit; money reached payee but switch reversed/failed UDIR: confirm and either finalize SUCCESS or claw back the credit; adjust settlement
Switch says success, bank never credited credit lost/dropped at bank UDIR: re-drive credit or reverse the debit; make the payer or payee whole
Debit at bank, no matching switch txn orphan debit reverse at the bank; the payer must not lose money
Amount mismatch same txn, different amount flag, adjust, alarm if large
  • Automated dispute resolution (UDIR). Every mismatch and every unresolved deemed transaction becomes a dispute record with a status (open -> investigating -> resolved) driven by rules: a debited-but-not-credited transaction past the SLA auto-triggers a reversal to the payer (this is the “if not reversed within T, auto-reverse” rule real UPI enforces), and a credited-but-marked-failed transaction is confirmed and finalized. Most breaks resolve automatically from the two logs; only genuine ambiguities go to a human ops queue. Nothing is closed without a balancing action in the ledger.
  • A suspense/control account keeps the books balanced while a break is worked, and its balance is a live alarm - it should trend to zero. The switch also runs internal integrity checks: the settlement file must sum to zero across banks, and per-transaction debit_amount == credit_amount.

Why this is right:

  • Netting makes settlement physically feasible. Clearing happens per transaction (in the switch’s ledger); real money moves per bank per cycle. This is the only way 10B transactions a month settle without melting the underlying rail.
  • Reconciliation is the only thing that detects drift from systems we do not control. Idempotency and the deemed rule keep us internally correct, but a lost response or a bank-side failure can only be caught by comparing our ledger against the bank’s own books - and then fixed with a real, owned, balancing action.
  • Automated dispute resolution turns “debited but not received” from a days-long fight into a rule. The auto-reversal SLA guarantees a wronged payer gets money back within a bounded time, without a human deciding case by case.
  • The books are proven daily, per bank. For a regulated national payment rail, this is not polish - it is the difference between a network the central bank will license and one that quietly accumulates missing money until it fails an audit.

API Design & Data Schema

API

The user-facing money-movement calls (every mutating call carries the network txn_id, which is the idempotency key):

POST /api/v1/pay                     (PSP -> Switch)
Headers: X-Signature: <PSP request signature>
Body:
{
  "txn_id":      "UPI-2026-08-03-9f3a2c...",   // client-generated, unique per attempt
  "payer_vpa":   "alice@okhdfc",
  "payee_vpa":   "coffeeshop@ybl",
  "amount":      25000,                          // integer paise = 250.00 (never float)
  "currency":    "INR",
  "enc_pin":     "<PIN block encrypted with remitter bank public key>",
  "purpose":     "P2M",                          // P2P | P2M
  "ref":         "order-8812"
}
Response 200:
{
  "txn_id":   "UPI-2026-08-03-9f3a2c...",
  "state":    "SUCCESS",                          // SUCCESS | FAILED | DEEMED_PENDING
  "rrn":      "422100055021",                     // retrieval reference number (bank-facing id)
  "amount":   25000,
  "completed_at": "2026-08-03T10:04:03Z"
}
Errors:
  400  invalid payload
  402  insufficient funds / debit declined
  401  bad signature
  403  wrong UPI PIN / risk block
  409  txn_id already in progress (returns current state, does not re-run)
  504  deemed pending (deadline hit; will resolve via reconciliation)

POST /api/v1/collect       {txn_id, payee_vpa, payer_vpa, amount, ref}  // pull; payer approves w/ PIN
                           -> 202 {txn_id, state: "COLLECT_PENDING"}
POST /api/v1/resolve-vpa   {vpa}   -> {bank_id, masked_name, active}    // never returns account number
POST /api/v1/link-account  {device, bank_id, account_ref}              // links account, sets PIN at bank

Switch-to-bank leg calls (internal, signed, each idempotent on txn_id):

POST /bank/{bank_id}/debit    {txn_id, account_token, amount, enc_pin, deadline}
                              -> {txn_id, status: DEBITED | DECLINED, rrn}
POST /bank/{bank_id}/credit   {txn_id, account_token, amount, deadline}
                              -> {txn_id, status: CREDITED | FAILED, rrn}
POST /bank/{bank_id}/reverse  {txn_id, orig_rrn}   -> {txn_id, status: REVERSED}
GET  /bank/{bank_id}/status/{txn_id}   -> {status}       // SAFE, non-mutating, for deemed resolution

Reads and status (served off async projections, never the hot switch path):

GET  /api/v1/transactions/{txn_id}          -> full state + both legs + rrn
GET  /api/v1/accounts/{vpa}/transactions    -> paginated history (read model)
POST /api/v1/webhooks/{psp}                 -> async status/receipt callbacks

A pay that completes both legs returns 200 SUCCESS; a pay that hits the deadline returns 504 DEEMED_PENDING with a txn_id the client polls, and the deemed machinery resolves it. The switch never returns a bare success it has not confirmed at both banks.

Data stores

Different access patterns, different stores. Be explicit.

1. Transaction store (the switch’s coordinator ledger) - SQL, ACID, sharded, synchronously replicated. This is the coordinator’s source of truth: one row per transaction carrying the state machine, both legs, and the idempotency guarantee. It needs strong consistency, a unique constraint on txn_id, and atomic state transitions - exactly what a relational database (PostgreSQL, or a distributed SQL like CockroachDB/Spanner for horizontal write scale) provides. NoSQL’s eventual consistency would reintroduce the double-debit and lost-state races we designed out.

Table: transactions                   -- one row per logical payment (state machine)
  txn_id          VARCHAR  PRIMARY KEY  -- UNIQUE; the network-wide idempotency key
  payer_vpa       VARCHAR  INDEX
  payee_vpa       VARCHAR  INDEX
  payer_bank      INT
  payee_bank      INT
  amount          BIGINT               -- paise, integer, never float
  state           VARCHAR              -- INITIATED|DEBITED|SUCCESS|FAILED|FAILED_REVERSED|DEEMED
  debit_rrn       VARCHAR  NULL
  credit_rrn      VARCHAR  NULL
  deadline_at     TIMESTAMP
  created_at      TIMESTAMP
  updated_at      TIMESTAMP
  Shard by: txn_id (hash)              -- even write spread across shards
  Index: (state, deadline_at)          -- to sweep DEEMED / in-flight for recovery

Table: txn_events                     -- APPEND ONLY, immutable audit of every transition
  event_id        UUID     PRIMARY KEY
  txn_id          VARCHAR  INDEX
  from_state      VARCHAR
  to_state        VARCHAR
  message_blob    BYTEA                -- signed message kept for non-repudiation
  created_at      TIMESTAMP
  Shard by: txn_id

Table: outbox                         -- event publish in the SAME txn as the state change
  outbox_id       BIGINT   PRIMARY KEY
  txn_id          VARCHAR
  topic           VARCHAR
  payload         JSON
  sent            BOOL     INDEX (partial: WHERE sent=false)

At each bank, a mirror idempotency table keyed on txn_id (unique) stores the applied result so a retry replays instead of re-applying:

Table: bank_upi_txns   (inside each bank)
  txn_id      VARCHAR PRIMARY KEY   -- UNIQUE; makes debit/credit/reverse apply once
  leg         VARCHAR               -- DEBIT | CREDIT | REVERSAL
  account     VARCHAR               -- internal account ref (dereferenced from token)
  amount      BIGINT
  status      VARCHAR               -- DONE | DECLINED
  rrn         VARCHAR
  response    JSON                  -- stored result for replay

2. Mapper (VPA -> account token) - NoSQL key-value, sharded by VPA, cached. Read every transaction, changes rarely, keyed lookups only, no joins - a wide-column/KV store (DynamoDB/Cassandra) sharded by VPA with a Redis cache in front. Returns (bank_id, account_token), never the raw account.

Key: vpa (e.g. "coffeeshop@ybl")
Value: { bank_id, account_token, masked_name, status, linked_at }
  Shard by: vpa hash    Cache: Redis, short TTL, invalidate on re-link

3. Transaction history read model - NoSQL wide-column, by VPA/user. History is high-volume, append-mostly, read by user in reverse-chronological pages, never joined, and can be eventually consistent. Built by a projector off the event log so history reads never touch the hot transaction shards.

Table: user_history
  vpa          VARCHAR PARTITION KEY
  txn_id       VARCHAR CLUSTERING KEY (created_at DESC)
  direction, amount, counterparty, state, created_at

4. Settlement & reconciliation store - SQL. Net positions per bank per cycle, bank statement lines, match results, and the dispute (UDIR) workflow. Relational because it is joined against the transaction ledger and needs consistency for the disputes state.

Why SQL for the coordinator, NoSQL for the projections: the transaction ledger needs a unique constraint on txn_id, atomic multi-field state transitions, the same-transaction outbox, and a range sweep over (state, deadline_at) to recover deemed transactions - that is precisely relational’s job, and worth sharding SQL by txn_id to scale writes. The mapper and history are read-heavy, key-scoped, join-free, and consistency-tolerant, so they go to a KV store and a wide-column store that scale reads cheaply. Match the consistency guarantee to the data: absolute for the money state machine, relaxed for the projections of it.

Bottlenecks & Scaling

Where it breaks first, in order, and the fix for each.

1. Double-debit under client/switch retries and lost responses (breaks first, worst impact). The failure a real person feels - charged twice, or money gone with no credit. Fix: the network-wide txn_id as an idempotency key with a DB unique-constraint backstop at the switch and at both banks, so debit, credit, and reversal each apply exactly once no matter how many retries arrive. Ambiguous timeouts use a safe status query, never a blind retry of a mutating call.

2. Stranded / deemed transactions from a slow or unreachable bank. A bank that does not answer inside the deadline leaves money “debited but not credited.” Fix: the stateful coordinator + deemed rule + auto-reversal SLA - every transaction is driven to SUCCESS or fully-reversed FAILURE, biased to returning money on doubt, with reconciliation finalizing anything the synchronous path could not. A recovery sweeper scans (state=DEBITED/DEEMED, deadline_at < now) and drives each forward or back.

3. A single bank being down taking the network with it. If the switch blocks or fails broadly when one bank is slow, one bank’s outage becomes everyone’s outage. Fix: per-bank isolation - circuit breakers and bulkheads per bank adapter, per-bank timeouts and concurrency limits, and published per-bank health so PSPs can warn or route. A transaction only fails if a bank it actually touches is down; unrelated bank pairs are unaffected.

4. Switch write throughput on the transaction ledger. The strongly-consistent SQL coordinator is the scaling ceiling; ~180K state-mutating message hops/sec cannot ride read replicas. Fix: shard the transaction ledger by txn_id (hash) so writes spread evenly across shards - and because a transaction’s entire lifecycle keys on one txn_id, all its state transitions land on one shard, so there is no cross-shard transaction in the hot path. Keep each state transition a single-row update so lock hold time is minimal. Reads come from projections, never the primaries.

5. Hot payee (a huge merchant) and hot bank. A single popular merchant VPA or a single large bank receives a disproportionate share of credits; naive per-row or per-account contention on that side serializes. Fix: the credit is keyed by txn_id, not by the payee row, so there is no single hot row at the switch - credits to one merchant spread across shards by txn_id. At the bank, hot-account credit contention is the bank’s problem, mitigated by balance sub-sharding / batched posting for mega-merchants. The mapper hot key (a viral merchant QR resolved millions of times) is absorbed by the Redis cache and read replicas.

6. Mapper read load (every transaction resolves a VPA). 30K resolutions/sec against the mapping store. Fix: aggressive caching of VPA -> (bank, token) with short TTL and invalidation on re-link; shard the mapper by VPA; resolve by handle suffix so load spreads across owning PSPs/banks rather than one central table.

7. The dual-write problem (switch DB commit vs event publish). Committing a state change but failing to publish the event leaves notifications/settlement blind. Fix: the outbox pattern - the event is inserted in the same ACID transaction as the state change, and a relay publishes it at-least-once; consumers dedupe on txn_id. The switch DB is the single source of truth for what happened.

8. Settlement scale. 333M transactions/day cannot each move real inter-bank money. Fix: net multilateral settlement in cycles - clear per transaction in the switch ledger, but move only each bank’s net position per cycle through pre-funded central-bank settlement accounts. The settlement file must sum to zero across banks, which is also an integrity check.

9. Ledger-vs-bank drift (lost responses, bank-side failures, reversals). Internal correctness cannot detect divergence from banks we do not control. Fix: daily and intraday reconciliation against each bank’s own transaction logs, an automated dispute (UDIR) workflow with the auto-reversal SLA, and a suspense account that keeps the books balanced while a break is worked (its balance trending to zero is a health metric, a rising balance is a page).

10. Storage growth (~120 TB/year, never deleted). Regulatory retention forbids purging. Fix: time-partition the transaction ledger and event log; keep recent (~90d) partitions hot, roll older ones to cheaper cold-but-queryable storage. Archive per-bank idempotency rows after the retry+dispute window (short) since their active use is brief.

11. Single points of failure. No single stateful node loss may stop or lose money. Fix: the SQL coordinator uses synchronous replication with automatic failover (no committed transaction lost on a node death), Kafka and Redis replicated, stateless switch handlers and PSP backends horizontally scaled behind load balancers, and the outbox + recovery sweeper guarantee nothing in flight vanishes silently.

Shard keys, stated plainly: transactions, txn_events, outbox, and each bank’s idempotency table -> txn_id (the whole lifecycle co-locates on one shard, so no cross-shard transaction in the hot path; even write spread by hash). Mapper -> vpa (keyed resolution, spread by handle owner). user_history -> vpa/user (single-partition reverse-chronological reads). Kafka topics -> partition by txn_id (per-transaction ordering + even spread). Settlement -> aggregated per bank per cycle. Never shard the transaction ledger by time - it concentrates all current writes on the newest partition, a permanent hot spot.

Wrap-Up

The trade-offs that define this design:

  • The switch coordinates the money; the banks hold it. We reject a distributed lock / 2PC across banks (they will not block under an external coordinator) in favor of a stateful switch that drives an ordered Saga - debit, then credit, else reverse - using each bank’s own local ACID transaction as the unit that actually moves money. We trade the illusion of one global transaction for a coordinator that always reaches a correct terminal state.
  • Debit before credit, and bias to reversal on doubt. Ordering makes the only bad intermediate state (“debited, not credited”) always recoverable, and when a transaction cannot be confirmed as fully SUCCESS, the safe resolution is always money back to the payer. A false failure is cheap; a false success is not.
  • The deadline is a user promise; correctness is a back-office guarantee. The 5-second SLA governs what the user sees, but ambiguous timeouts become deterministic “deemed” transactions resolved by safe status queries and an auto-reversal SLA - so we can time out the user without ever timing out the money.
  • Idempotency by txn_id everywhere money moves. One key, a unique constraint at the switch and both banks, makes debit, credit, and reversal apply exactly once across an at-least-once network. Retries and lost responses become safe.
  • Clear per transaction, settle per bank per cycle, prove it daily. Netting makes 10B monthly transactions physically settleable, and reconciliation against each bank’s own books with an automated dispute machine proves the money actually moved and fixes it when it did not.
  • SQL for the coordinator, NoSQL for the projections. ACID and a unique constraint on txn_id for the state machine; cheap read-scaled KV/wide-column stores for the mapper and history; sharded by txn_id/vpa to scale.

One-line summary: a PSP captures pay intent and an encrypted UPI PIN, a central switch resolves the payee VPA to a bank via a tokenized mapper and then drives an ordered, deadline-bounded, idempotent Saga - debit the payer’s bank in one local ACID transaction, credit the payee’s bank in another, and on any failure or timeout reverse the debit so money is never lost or invented - with every transaction keyed by txn_id, deemed-transaction rules resolving ambiguity to the safe state, net multilateral settlement moving only each bank’s cycle position, and daily reconciliation with automated dispute resolution proving the books across every bank agree, so money moves between two banks in under 5 seconds, exactly once, and provably correct.