A user opens the app, taps “Electricity”, types a consumer number, and a second later sees “Bill: 2,340, due Nov 20”. They pay, and a few seconds later a receipt lands with a reference number the electricity board will honour. That is the product. It looks like a thin shopping cart over a payment gateway, and the interviewer will let you believe that for about a minute before pointing at the thing that actually makes this hard: there is no single “bill API”. There are 1000+ billers - every state electricity board, every water utility, every gas company, every telecom - each with its own protocol, its own uptime, its own idea of what “fetch bill” means, and its own habit of going down on the first of the month when everyone pays at once.

A bill-payment aggregator is not a cart. It is a fan-out integration layer that normalizes a thousand flaky downstream systems into one uniform fetch-pay-receipt API, sitting on top of an exactly-once money-movement core that must never pay twice and must never lose a payment the user was charged for. The entire design is about those two problems: taming heterogeneous, unreliable billers, and moving money correctly when the confirmation from the biller might arrive seconds later, minutes later, or never. Let me build it properly.

Functional Requirements (FR)

In scope:

  • Discover and select a biller. The user picks a category (electricity, water, gas, mobile postpaid/prepaid, DTH, broadband, FASTag, loan EMI) and a specific biller, then enters an identifier (consumer number, mobile number, policy number) plus whatever that biller requires.
  • Fetch the bill in real time. For “bill fetch” billers, query the biller live for the current outstanding amount, due date, and bill details, and show it before payment. For “recharge” style (prepaid mobile, DTH), the user picks a plan amount instead.
  • Pay the bill. Collect money from the user (UPI, card, netbanking, wallet) and deliver a payment to the biller against that identifier, exactly once.
  • Confirm and issue a receipt. Return a biller-acknowledged receipt with a reference number, or a clear pending/failed state, and refund the user automatically if the money was collected but the bill payment failed at the biller.
  • Handle the full lifecycle. A payment can be instantly confirmed, deferred-confirmed (biller acks asynchronously), or ambiguous (we paid the biller but never heard back). All three must resolve deterministically.
  • Reconciliation and settlement. End-of-day, match every user charge against every biller payment against every settlement with the biller, detect mismatches, and drive refunds or re-payments.
  • Transaction history, receipts, notifications, retries, and complaint/dispute handling.

Explicitly out of scope (say it out loud so you own the scope):

  • Being the payment gateway / collecting rail. UPI switch, card network, netbanking - the rail that collects money from the user is a separate payments system. I consume “money collected, here is a collection reference” and emit “pay this biller”. I covered the collection side in the UPI and wallet posts; here I am the layer between collection and the biller.
  • Being the biller’s billing system. How the electricity board computes your bill, meter reading, tariff slabs - that is the biller’s problem. I fetch a number and pay against an identifier.
  • KYC, onboarding the user, fraud scoring. Assumed upstream; I consume an authenticated user and a risk verdict.
  • The biller-onboarding back office. Contracts, API credentials, test certification per biller is an operational process. I consume a biller registry (how to talk to biller X).

The one decision that drives everything: the biller is slow, heterogeneous, and unreliable, but the money movement must be exactly-once and fully recoverable. You cannot make a thousand downstream systems reliable; you can only make your side correct in spite of them. So the architecture is: a normalization layer that hides biller differences behind one interface, and an exactly-once orchestration that treats “did the biller actually get paid?” as a first-class, eventually-resolved question rather than a synchronous boolean.

Non-Functional Requirements (NFR)

  • Scale: 50M transactions/day. Bill fetches run higher than payments (people check and abandon), call it ~1.5-2x fetches. 1000+ billers, heavily skewed - the top 50 billers do most of the volume.
  • Latency: bill-fetch should feel instant, p95 under ~2s (bounded by the biller’s own API), with a hard timeout. Payment submission should ack to the user in under ~3s with either confirmed or “processing”; the user must never be left spinning on a biller that is taking 40 seconds.
  • Availability: the aggregator itself targets 99.99%, and - critically - must degrade per-biller, not globally. One state electricity board being down must not take down mobile recharges or the other 999 billers.
  • Consistency: money movement is exactly-once and strongly consistent - never pay a biller twice for one user intent, never charge a user without either paying the biller or refunding. The user-facing transaction state is strongly consistent; the biller-confirmation is allowed to be eventually consistent (deferred), but it must converge.
  • Durability: every transaction, every biller request/response, every settlement line is a financial and legal record - durably persisted, never lost, auditable for disputes.
  • Idempotency: every stage - collection, orchestration, biller call - is idempotent on a stable key so retries and duplicate submissions collapse to one payment.

Back-of-the-Envelope Estimation (BoE)

Real numbers with the arithmetic, because they set the architecture.

Transaction rate:

50M payments/day
= 50,000,000 / 86,400 s  ~= 580 payments/sec average.
Bill fetches ~1.5-2x payments -> ~1,000 fetches/sec average.
Peak factor is brutal and non-uniform: utility bills cluster on the 1st
and around salary day, and within a day on evenings. Peak ~6-8x average.
Peak payments ~= 3,500-4,500/sec. Peak fetches ~= 6,000-8,000/sec.
Read:write (history/status reads vs money writes) ~10:1 on the app side.

So the write rate is modest (~4.5k payments/sec peak) but every write is an exactly-once money movement with an unreliable downstream. The hard number is not the QPS; it is the tail - a biller’s p99 fetch latency and its failure rate.

Biller skew (the load-bearing fact):

Top 50 billers ~= ~80% of volume.
At peak, the single biggest biller (a large telecom or a big state DISCOM)
can see ~500-1,000 payments/sec alone.
That biller's own API might be rated for far less, so per-biller rate
limiting and queueing is not optional - it is the core of the design.

Storage / year:

Transaction record ~= 1 KB (user, biller, identifier, amount, bill
  details, collection ref, biller ref, status timeline, timestamps).
50M/day * 1 KB = 50 GB/day -> ~18 TB/year of transactions.
Biller request/response audit logs (raw payloads, bigger) ~= 3 KB each,
  a few per txn -> ~150 GB/day -> ~55 TB/year, mostly cold/archivable.
Ledger postings (double-entry, 2-3 per txn) -> ~30-40 GB/day.
Whole estate on the order of ~80-120 TB/year before replication.

Append-only, partitionable by time, cold after settlement. Boring but large.

Bandwidth: each biller round trip ~2-4 KB. Peak ~8k fetches/sec * 3 KB ~= ~24 MB/sec outbound to billers. Trivial. This system is tail-latency and correctness bound, not bandwidth bound.

Cache footprint:

Biller registry (how to call each biller: endpoint, protocol, fields,
  limits, timeout) ~= 1000 billers * a few KB = a few MB. Keep hot.
Recent-bill cache (consumer -> last fetched bill, short TTL) can absorb
  repeat fetches and abandon-then-return flows: maybe 10M entries * 300B
  ~= 3 GB in Redis. Optional but cheap.
Circuit-breaker / health state per biller: ~1000 entries, tiny, hot.

The takeaways: modest exactly-once write rate (~4.5k/sec), extreme biller skew demanding per-biller isolation and rate limiting, tail latency dominated by the worst biller, and boring large append-only storage. The design driver is not throughput; it is normalizing a thousand unreliable downstreams while moving money exactly once.

High-Level Design (HLD)

Three planes. The API/orchestration plane owns the user-facing fetch-pay-receipt flow and the transaction state machine. The biller-integration plane is a fleet of per-biller adapters behind one normalized interface, each with its own queue, rate limiter, and circuit breaker. The money/ledger plane does the exactly-once accounting - collection, biller payment, refund - on a strongly-consistent ledger, with a reconciliation service closing the loop against billers at end of day.

      User app
        │  fetch bill / pay / get receipt
   ┌────▼───────────────────────────────┐
   │  API GATEWAY (auth, rate limit)     │
   └────┬───────────────────────────────┘
        │
   ┌────▼───────────────────────────────┐      ┌───────────────────────┐
   │  PAYMENT ORCHESTRATOR               │◀────▶│ Biller Registry (hot)  │
   │  - transaction state machine        │      │  endpoint, protocol,   │
   │  - idempotent on request_id         │      │  fields, limits, timeout│
   │  - drives collect -> pay -> receipt │      └───────────────────────┘
   └──┬────────────┬───────────────┬─────┘
      │            │               │
 collect money   pay biller     write state
      │            │               │
 ┌────▼─────┐  ┌───▼───────────────────────────────┐   ┌──────────────┐
 │ COLLECTION│  │  BILLER INTEGRATION LAYER          │   │  LEDGER (SQL) │
 │  (UPI/    │  │  ┌──────────────────────────────┐  │   │  double-entry│
 │  card via │  │  │ per-biller ADAPTER            │  │   │  exactly-once│
 │  PG)      │  │  │  - protocol translate        │  │   │  user<->biller│
 │  exactly- │  │  │  - per-biller QUEUE           │  │   │  nostro acct │
 │  once     │  │  │  - RATE LIMIT + CIRCUIT BREAK │  │   └──────────────┘
 └───────────┘  │  │  - timeout + retry + idemp    │  │
                │  └───────────────┬──────────────┘  │
                └──────────────────┼─────────────────┘
                                   │  normalized call
                   ┌───────────────▼──────────────┐
                   │   1000+ BILLERS (DISCOM,      │
                   │   telecom, water, gas, DTH)   │
                   │   each: own API, own uptime   │
                   └───────────────┬──────────────┘
                                   │ ack / async callback / silence
                   ┌───────────────▼──────────────┐
                   │  STATUS RESOLVER + RECON (EOD) │
                   │  - poll pending, ingest acks   │
                   │  - match charge vs pay vs       │
                   │    settlement; refund/retry     │
                   └────────────────────────────────┘

Fetch path (read a bill), target p95 < 2s:

  1. The app calls the gateway with (biller_id, identifier, user_inputs). The orchestrator looks up the biller in the registry to learn how to talk to it.
  2. It checks the recent-bill cache; on miss, it calls the biller’s adapter, which translates the request into that biller’s protocol, enforces the biller’s rate limit, and applies a hard timeout.
  3. The adapter normalizes the response into a uniform {amount, due_date, bill_date, details, fetch_id} and returns it. The fetch_id ties the subsequent payment to this exact bill snapshot.

Pay path (move money), ack to user < 3s, confirm possibly later:

  1. The app submits a payment carrying a client-generated request_id (idempotency key) and the fetch_id. The orchestrator creates (or re-reads, if duplicate) a transaction in state INITIATED and triggers collection from the user via the payment gateway, exactly-once on request_id.
  2. On successful collection, the transaction moves to COLLECTED and is handed to the biller adapter to pay, keyed by the same request_id. The adapter enqueues it on the per-biller queue (respecting rate limit and circuit state).
  3. The biller responds: success -> SUCCESS + receipt; hard failure (bad consumer no., bill already paid) -> FAILED -> auto-refund the collected money; timeout/ambiguous -> PENDING, and the status resolver takes over.
  4. The user gets an immediate ack: SUCCESS with receipt, or PENDING (“payment is processing, receipt shortly”). Nobody waits on a slow biller.

Resolution path (the ambiguous case), eventually consistent but convergent:

  1. For PENDING transactions, the status resolver polls the biller’s status API (idempotent query by request_id) on a backoff, and/or ingests the biller’s asynchronous callback, until it learns success or failure, then issues receipt or refund.
  2. Reconciliation at EOD matches every user charge against every biller payment against the biller’s own settlement file, and drives refunds for charged-but-not-paid and raises disputes for paid-but-not-settled.

The key insight: “did the biller get paid?” is not a synchronous boolean - it is a state that resolves over time, so the transaction is a durable state machine, the user is acked optimistically as PENDING, and correctness comes from exactly-once keys plus a resolver plus reconciliation.

Component Deep Dive

The hard parts, naive-first then evolved: (1) normalizing 1000+ heterogeneous, unreliable billers, (2) the exactly-once money movement across collect-pay-refund, (3) the ambiguous “we paid but did not hear back” state, and (4) protecting good billers from a flooding or failing one.

1. Normalizing 1000+ heterogeneous, unreliable billers

Naive approach: a big if/else (or a switch) in the orchestrator - if biller == X call X's SOAP endpoint, else if biller == Y ....

Where it breaks:

  • A thousand branches in the hot path. Each biller has a different protocol (REST, SOAP, a bank-hosted file drop, a legacy fixed-width socket), different field names, different auth, different error codes. One monolith full of per-biller special cases is unshippable and untestable, and every new biller is a core-code change.
  • One slow biller stalls the shared thread pool. If the orchestrator calls billers inline, a biller taking 40s to time out holds a worker. A few slow billers at peak exhaust the pool and the whole aggregator stalls - a classic shared-fate failure.
  • No uniform retry/timeout/error semantics. Biller A returns HTTP 200 with <status>FAIL</status> in the body; biller B returns 500; biller C just hangs. Without normalization, the orchestrator cannot reason about success/failure uniformly, so the money logic cannot be correct.

First evolution: an adapter per biller behind one interface. Define a normalized contract the orchestrator speaks, and implement it once per biller (or per biller family, since many share a bank-hosted BBPS-style gateway):

interface BillerAdapter {
  fetchBill(identifier, inputs) -> {amount, due_date, details, fetch_id}
  payBill(request_id, identifier, amount, fetch_id) -> {status, biller_ref}
  queryStatus(request_id) -> {status, biller_ref}   // idempotent status probe
}

Now the orchestrator never knows a biller’s protocol; it knows the interface. Each adapter owns translation, auth, field mapping, and mapping the biller’s idiosyncratic errors onto a normalized status set: SUCCESS | HARD_FAIL | RETRIABLE | TIMEOUT. New biller = new adapter, zero core change. But adapters called inline still share fate on latency, and a flooding biller still hurts neighbours.

The answer: adapters as isolated workers, each fronted by its own queue, rate limiter, circuit breaker, and bulkhead.

  • Per-biller queue + bulkhead. Each biller (or biller shard) gets its own work queue and its own bounded worker pool. A biller taking 40s only backs up its own queue and consumes its own workers; the other 999 billers are untouched. This bulkheading is what turns “one biller is down” from a global outage into a localized, contained slowdown.
  • Per-biller rate limiting. The registry stores each biller’s contracted rate (say 200 req/sec). The adapter’s limiter never exceeds it, smoothing the 1st-of-month spike by buffering in the queue rather than hammering the biller into collapse. The user sees PENDING; the resolver confirms when the queue drains.
  • Per-biller circuit breaker. When a biller’s error/timeout rate crosses a threshold, trip the breaker: stop calling it, fail fast to PENDING (for payments already collected) or show “this biller is temporarily unavailable” (for fetches), and probe periodically with a trickle of requests to detect recovery. This stops pouring load onto a dying biller and stops burning worker time on calls that will time out anyway.
  • Registry-driven, not code-driven, config. Endpoint, protocol, timeout, rate limit, required fields, retry policy live in the biller registry (hot-cached), so operational tuning (“DISCOM X’s API got slower, raise timeout, lower rate”) is a config change, not a deploy.

The principle: hide biller heterogeneity behind one normalized interface, and isolate each biller’s unreliability behind its own queue, limiter, and breaker so failure is contained, never shared. The orchestrator speaks one language to a thousand very different, very flaky systems.

2. The exactly-once money movement (collect -> pay -> refund)

This is the correctness core. There are two money movements per transaction - collect from the user, pay the biller - plus a possible refund, and all must be exactly-once and must never leave money stranded.

Naive approach: collect money, then call the biller, then mark done. On any error, “handle it.”

collect(user, amount)          # charge the user
pay_biller(biller, amount)     # pay the biller
mark SUCCESS

Where it breaks:

  • Charged but not paid, money stranded. Collection succeeds, the biller call times out, the process crashes. The user is debited; the biller was never paid; there is no record of intent to recover from. The user’s money evaporated.
  • Double payment on retry. The biller call times out; we retry; both actually land. The biller is paid twice; the user is furious; recovering an over-payment from a state electricity board is a month-long dispute.
  • Refund races. We decide to refund a failed payment, but the biller’s delayed success arrives after the refund - now the user got the service and the money back. Or two refund attempts both fire.
  • No single source of truth for “where is this transaction”. “Handle the error” has nowhere to look.

The answer: a durable transaction state machine, an idempotency key end to end, and a double-entry ledger with a nostro account - so every rupee is always accounted for in some state.

  • Durable state machine, persisted before every external call. The transaction is a row whose state transitions are written before the side effect that would advance them:
INITIATED -> COLLECTED -> PAID -> SUCCESS            (happy path)
INITIATED -> COLLECTED -> PENDING -> SUCCESS         (deferred biller ack)
INITIATED -> COLLECTED -> PENDING -> FAILED -> REFUNDED
INITIATED -> FAILED                                   (collection failed, no charge)
COLLECTED -> FAILED -> REFUNDED                       (biller hard-fail after charge)

A crash at any point leaves a durable state a recovery job can resume. There is never a side effect without a preceding durable intent.

  • One idempotency key, request_id, threaded through every hop. Collection is idempotent on it (the PG dedups), the biller payBill carries it (the adapter and ideally the biller dedup on it), the refund is keyed by it. A retry at any layer collapses to the original outcome instead of a second movement. Where the biller itself cannot dedup, the adapter maintains a processed(request_id -> biller_ref) table and, on retry, first calls queryStatus(request_id) to check whether the prior attempt actually landed before re-submitting - so an ambiguous timeout never blindly double-pays.

  • Double-entry ledger with a nostro (settlement) account. Money is never just “subtracted”; it moves between accounts in balanced postings, so it is always somewhere:

On COLLECTED:   debit user_wallet/PG_clearing, credit aggregator_nostro
On PAID:        debit aggregator_nostro,       credit biller_payable
On REFUNDED:    debit aggregator_nostro(reverse), credit user_refund

Every state the money can be in maps to a ledger balance. “Charged but not paid” is not stranded money - it is a visible, non-zero aggregator_nostro position for that transaction, which reconciliation and the resolver act on. Money is stored as integer paise, never a float.

  • Idempotent, race-safe refund. The refund is itself a state transition guarded by the state machine: a transaction can only go FAILED -> REFUNDED once (conditional update WHERE status='FAILED'), and before refunding, the resolver does a final queryStatus to rule out a late biller success. A late success after refund is resolved by reconciliation as a dispute, not silently.

The mental model: the transaction is a durable state machine, money lives in a double-entry ledger so it is always accounted for, and one idempotency key makes collect, pay, and refund each exactly-once. “Where is the money?” always has an answer in a row.

3. The ambiguous “we paid but never heard back” state

Naive approach: treat a biller timeout as a failure and refund.

Where it breaks:

  • The biller often succeeded anyway. Many billers process the payment and then the response times out, or they ack asynchronously minutes later. Treating timeout as failure and refunding means the user gets refunded but the bill got paid - the aggregator eats the cost, or worse, the user is told “failed” while their electricity bill shows paid.
  • Blind retry double-pays (section 2), and blind refund strands the biller-paid money on our side.
  • The user cannot wait. Holding the request open until the biller eventually responds blows the 3s ack budget and ties up resources.

The fix: make PENDING a real, first-class state resolved by an asynchronous resolver, never guessed synchronously.

  • Ack optimistically as PENDING. On timeout/ambiguous, the transaction goes to PENDING and the user is told “processing, receipt shortly” within the budget. The money is already collected and sitting in nostro - safe and accounted for.
  • Resolver reconciles via idempotent status probes. A background resolver drains PENDING transactions and calls queryStatus(request_id) on the biller with exponential backoff (e.g. 10s, 30s, 2m, 10m…). Because the status probe is idempotent and keyed by request_id, it is safe to call repeatedly. The authoritative answer comes from the biller’s own record of that request_id, not from our guess.
  • Ingest asynchronous callbacks. Many billers (BBPS-style) push an async confirmation callback. The resolver consumes these (idempotently, dedup on biller_ref/request_id) to short-circuit polling. Callback and poll converge on the same state machine transition, guarded so only the first wins.
  • Bounded resolution with a terminal decision. After a max window (say a few hours, per biller SLA), an unresolved payment is forced to a terminal state by policy: if the biller still cannot confirm, refund the user and flag for reconciliation to catch a late settlement. The window is bounded so no transaction is PENDING forever.
biller timeout -> state=PENDING (money safe in nostro)
resolver loop:
   r = biller.queryStatus(request_id)   # idempotent
   SUCCESS  -> state=SUCCESS, issue receipt
   HARD_FAIL-> final queryStatus, then state=FAILED -> REFUNDED
   UNKNOWN  -> backoff, retry; on max-window -> refund + flag recon
callback ingest -> same transitions, dedup so first-writer-wins

The principle: ambiguity is resolved over time by an idempotent status probe against the biller’s own record, never by a synchronous guess - and reconciliation is the backstop for whatever the resolver cannot settle.

4. Protecting good billers from a flooding or failing one

Naive approach: one shared worker pool and one shared rate budget across all billers.

Where it breaks:

  • Noisy-neighbour collapse. The 1st-of-month DISCOM spike or one biller timing out at 40s consumes the shared pool; mobile recharges and all other billers starve. One biller’s bad day becomes everyone’s outage.
  • Thundering herd on recovery. A biller comes back after an outage and all queued-up work fires at once, knocking it straight back down.

The fix: bulkheads, per-biller isolation, and graceful shed/backpressure (ties together section 1’s mechanisms applied to load).

  • Bulkhead per biller (or per biller shard). Bounded, isolated worker pool and queue per biller. A biller can only ever consume its own slice of capacity; it physically cannot starve others. The big-50 billers get larger dedicated pools; the long tail shares smaller pooled bulkheads sized to their tiny volume.
  • Backpressure, not unbounded buffering. Each per-biller queue is bounded. When full (biller is down or throttled below inflow), new payments for that biller are either queued-with-PENDING (money collected, will drain) or, for fetches, fast-failed with “biller temporarily unavailable”. The pressure is contained to that biller and surfaced honestly instead of silently piling up memory.
  • Circuit breaker + jittered recovery. On trip, stop calling; on recovery, release queued work with rate limiting and jitter so the biller is eased back up, not slammed. This prevents the herd from re-killing a just-recovered biller.
  • Priority within a biller’s queue. Payments (money already collected, must settle) outrank fetches (speculative, abandonable) in the same biller’s queue, so a fetch flood never delays real payments.

The principle: isolate every biller behind its own bounded resources so failure and load are contained, shed fetch load before payment load, and ease billers back up on recovery instead of stampeding them.

API Design & Data Schema

The app talks to the gateway; the orchestrator talks to collection, adapters, and the ledger; the resolver and recon run behind.

User-facing

GET /api/v1/billers?category=electricity&region=KA
  -> 200 { billers:[ {biller_id:"KA-DISCOM-01", name:"BESCOM",
           type:"FETCH", fields:[{key:"consumer_no", label, regex}] } ] }

POST /api/v1/bills/fetch
  body: { biller_id, inputs:{consumer_no:"123..."} }
  -> 200 { fetch_id:"f_9a..", amount:234000,  # paise
           due_date:"2026-11-20", bill_date:"2026-11-02",
           details:{units:312, period:"Oct"} }
  -> 200 { status:"UNAVAILABLE", reason:"biller_down" }   # per-biller degrade

POST /api/v1/payments
  headers: { Idempotency-Key: request_id }
  body: { biller_id, inputs, fetch_id, amount:234000,
          pay_method:{type:"UPI", vpa:"user@bank"} }
  -> 200 { request_id, status:"SUCCESS", receipt_id, biller_ref:"BESCOM/88.." }
  -> 200 { request_id, status:"PENDING", message:"processing" }
  -> 200 { request_id, status:"FAILED", reason:"bill_already_paid" } # auto-refund

GET /api/v1/payments/{request_id}
  -> 200 { status:"SUCCESS|PENDING|FAILED|REFUNDED", receipt_id?,
           biller_ref?, refund_ref?, timeline:[...] }

Internal (orchestrator -> biller adapter)

POST /adapter/{biller_id}/pay
  headers: { Idempotency-Key: request_id }
  body: { request_id, identifier, amount, fetch_id }
  -> 200 { status:"SUCCESS|HARD_FAIL|RETRIABLE|TIMEOUT", biller_ref? }

GET /adapter/{biller_id}/status?request_id=..   # idempotent probe
  -> 200 { status, biller_ref? }

Data stores - the right tool per plane

1. Transactions + ledger - strongly-consistent relational store (SQL), sharded by user_id. The transaction state machine and the double-entry ledger need ACID: a state transition and its ledger postings must commit atomically, and the idempotency check must be a unique constraint. This is SQL’s home turf; an eventually-consistent KV store would corrupt money.

transactions  (SQL, sharded by user_id)
  request_id      PK / UNIQUE        -- idempotency, exactly-once
  user_id         indexed
  biller_id       indexed
  identifier       (consumer_no etc, encrypted)
  fetch_id
  amount          BIGINT  (paise, integer money - never float)
  status          INITIATED|COLLECTED|PAID|PENDING|SUCCESS|FAILED|REFUNDED
  collection_ref
  biller_ref
  refund_ref
  created_at, updated_at
  INDEX (status, updated_at)          -- resolver scans PENDING efficiently

ledger  (SQL, append-only, partitioned by month, sharded by account)
  posting_id   PK
  txn_ref      indexed      (= request_id)
  account      indexed      (user / nostro / biller_payable / refund)
  amount       BIGINT signed
  posted_at

processed_biller_call  (SQL, in adapter)
  request_id   PK / UNIQUE
  biller_ref
  result
  created_at

2. Biller registry + health/circuit state - in-memory KV (Redis), hot. How to call each biller, its limits, and live breaker/health state. Small (a few MB), read on every request, tolerant of brief staleness. Pushed to adapters.

Redis:
  HGETALL biller:{biller_id}  -> {endpoint, protocol, timeout, rate_limit, fields}
  GET breaker:{biller_id}     -> OPEN|HALF_OPEN|CLOSED
  recent_bill:{biller_id}:{identifier} -> {amount, due_date} (short TTL)

3. Per-biller work queue - durable message queue, partitioned by biller. Buffers payments and fetches per biller, enforces rate limit and bulkhead, survives restarts so a collected-but-not-yet-paid payment is never lost. Kafka/SQS-style, one logical queue (or partition) per biller.

4. Audit log of raw biller request/response - append-only object store / warehouse, partitioned by day/biller. Every raw payload for disputes, debugging, and reconciliation. Write-once, read-by-range, never updated - cheap columnar/object storage, not the hot SQL.

Why SQL for transactions/ledger, KV for registry, queue for biller work, object store for audit: the planes have opposite needs. The transaction/ledger is a small, must-be-exactly-right financial write - it wants ACID and unique-key idempotency. The registry is a tiny read-hot lookup - in-memory KV. Biller work is bursty, must-survive-restart, per-biller-isolated flow - a durable partitioned queue. Audit is unbounded write-once history - cheap append-only storage. There is no single “bill payment database”.

Bottlenecks & Scaling

Where it breaks first, in order, and the fix for each.

1. A slow/down biller stalling shared resources (breaks first). Inline biller calls on a shared pool let one 40s biller exhaust workers and take down the aggregator. Fix: per-biller bulkhead + queue + circuit breaker. Isolate each biller’s resources so failure is contained; trip the breaker to fail fast; drain with jitter on recovery. The single most important isolation in the design.

2. Biller skew / hot biller. The top biller alone can see ~1,000 payments/sec at peak, far above its API’s rated capacity. Fix: per-biller rate limiting with a durable buffering queue - smooth the spike into the biller at its contracted rate, ack users as PENDING, and let the resolver confirm as the queue drains. Shard a mega-biller’s queue across partitions if one partition can’t keep up.

3. Exactly-once money movement under retries and ambiguity. Blind retries double-pay; blind refunds strand money. Fix: end-to-end request_id idempotency + status-probe-before-retry + double-entry ledger with nostro. Correctness is enforced authoritatively at the ledger (unique request_id) and at the adapter (queryStatus before re-submitting). Money is always in an accounted-for ledger state.

4. The PENDING backlog / resolver throughput. Spikes create large PENDING populations that the resolver must drain against slow billers. Fix: index (status, updated_at), run the resolver as a horizontally-scaled, sharded-by-biller worker set with per-biller rate limits (reuse the same bulkheads), and prefer callbacks over polling where billers support them to cut probe load.

5. Transactions DB as a write hotspot. ~4.5k exactly-once writes/sec plus state transitions and ledger postings. Fix: shard by user_id so different users are fully parallel; a single user’s few transactions never contend. Ledger partitioned by month and sharded by account; reads of history served from read replicas, authoritative writes on the primary shard.

6. Thundering herd on biller recovery. A recovered biller gets slammed by the whole queued backlog and falls over again. Fix: half-open circuit + jittered, rate-limited release of queued work, easing the biller back up rather than stampeding it.

7. Collection vs biller-payment split-brain. Collection and biller payment are two systems; a crash between them must not strand money. Fix: the durable state machine with intent-before-side-effect plus the nostro ledger account means a crash leaves a recoverable state and a visible nostro balance; a recovery job resumes COLLECTED transactions to PAID or REFUNDED. Never a side effect without a durable preceding intent.

8. Reconciliation at 50M/day. Matching every charge against every biller payment against each biller’s settlement file is a heavy three-way join, and mismatches (charged-not-paid, paid-not-settled, double-paid) must be caught. Fix: run recon as a partitioned batch/stream job keyed by request_id/biller_ref, comparing our ledger, our biller-call log, and the biller’s settlement file; exceptions to a dispute queue with automated rules (charged-not-paid -> refund or re-drive; double-paid -> reclaim). The append-only ledger and idempotency make recon a deterministic set-difference.

9. Stale bill fetch -> paying the wrong amount. A user fetches a bill, waits, the bill changes, then pays the stale amount. Fix: the fetch_id binds the payment to a bill snapshot with a short validity; on payment the orchestrator can re-validate against the biller for high-value or expired fetches before paying.

10. Single-region / data residency. Financial data residency and biller locality. Fix: run the orchestrator and ledger regionally, keep each user’s shard in-region, and route to geographically-near biller gateways; recon rolls up to a central clearing view. The consistency boundary is a single user’s transaction plus the ledger, so no cross-region distributed transaction is needed.

Wrap-Up

The trade-offs that define this design:

  • Normalization plus isolation for the billers. One uniform fetch/pay/status interface hides a thousand protocols, and a per-biller bulkhead-queue-limiter-breaker contains each biller’s unreliability so one bad biller is a local slowdown, never a global outage. You cannot make the downstreams reliable; you make your side correct and isolated in spite of them.
  • The transaction as a durable state machine, money as double-entry ledger. “Where is this payment and where is the money?” always has an answer in a row. Intent is persisted before every side effect, so a crash is recoverable and money is never stranded - it is always a visible ledger position.
  • Exactly-once via one idempotency key end to end. request_id threaded through collection, biller payment, and refund, enforced at the ledger’s unique constraint and guarded by a status-probe before any retry, makes duplicate submissions, retries, and callback races all collapse to one movement.
  • PENDING as a first-class, resolver-settled state. “We paid but did not hear back” is resolved over time by idempotent status probes and async callbacks against the biller’s own record, never guessed synchronously - and reconciliation is the backstop that forces eventual convergence and drives refunds or disputes.
  • Optimistic user ack, eventual biller confirmation. The user is acked in under 3s as SUCCESS or PENDING; the biller’s truth arrives when it arrives. Strong consistency on the user’s money and transaction state; bounded eventual consistency on the biller confirmation.

One-line summary: a bill-payment aggregator that normalizes 1000+ heterogeneous, unreliable billers behind one fetch-pay-status interface - each isolated by its own queue, rate limiter, and circuit breaker - on top of an exactly-once money core where every transaction is a durable state machine over a double-entry ledger, keyed by one idempotency key, with ambiguous payments resolved asynchronously by a status resolver and guaranteed to converge by end-of-day reconciliation.