A loan disbursal platform looks like a form plus a yes/no, and interviewers love it because it is not. The “form” fans out to a half-dozen external systems you do not control - credit bureaus, KYC/identity providers, bank account verification, a fraud graph, an ML risk model - some of which are slow, some flaky, some rate-limited, all of which cost money per call. The “yes/no” is a regulated underwriting decision that must be explainable, reproducible years later, and never made twice on the same application. And the payoff at the end is disbursal: real money leaving your account to a customer’s bank, over a rail you cannot roll back. A duplicate disbursal is money gone.

That reshapes the whole design. The naive view is “validate the form, score it, pay out.” The real problem is “orchestrate a multi-stage underwriting pipeline across unreliable external dependencies, make an auditable risk decision, catch fraud rings and synthetic identities before money moves, and disburse to a bank exactly once - at 1M applications/day, without ever double-paying or approving someone you should not.” The core invariant: every application reaches exactly one terminal decision, that decision is reproducible and explainable, and an approved loan is disbursed exactly once, only after KYC and fraud clear. Let me build it properly.

Functional Requirements (FR)

In scope:

  • Apply for a loan. A customer submits an application: requested amount, tenure, purpose, their identity details, income, and a target bank account for disbursal.
  • KYC / identity verification. Verify the applicant is who they claim: government ID checks, name/DOB/address match, liveness/selfie where required, and sanction/PEP screening. No money moves for an unverified identity.
  • Pull credit history. Fetch the applicant’s credit report and score from one or more bureaus (CIBIL, Experian, Equifax in India; equivalents elsewhere), plus any internal repayment history.
  • Risk decisioning via ML. Run a risk/underwriting model that outputs a probability of default, a recommended decision (approve/reject/refer), an approved amount, and an interest rate. The decision must be explainable (reason codes) and policy rules sit on top of the model.
  • Fraud detection. Screen every application against fraud signals: device/identity graphs, velocity (same device/IP/account applying repeatedly), synthetic-identity patterns, and known-bad lists - before approval and before disbursal.
  • Approve / reject / refer. Produce one terminal outcome per application. Approved applications proceed to loan creation and disbursal; referred ones go to a manual underwriting queue; rejected ones get reason codes.
  • Disburse to a bank account. On approval and after the customer accepts terms, transfer the principal to the verified bank account over a payment rail, exactly once, and create the loan + repayment schedule.
  • Status and audit. The applicant tracks their application state; every decision input, model version, rule version, and money movement is recorded immutably for regulators.

Explicitly out of scope (state it so you own the scope):

  • Repayment / collections / servicing. I create the loan and the schedule on disbursal; EMI collection, NACH mandates, delinquency, and collections are a separate servicing system.
  • Training the ML model. I consume a deployed, versioned model as a scoring service. The feature store and training pipeline are named but not designed here.
  • The bureau, KYC, and payment-rail internals. I integrate credit bureaus, an identity/KYC provider, and a disbursal rail (IMPS/NEFT/RTGS/UPI, or ACH) as black-box APIs. I do not build the central-bank rail.
  • The lending product/pricing engine beyond a rate output. I apply a pricing policy; I do not design a full treasury/pricing desk.

The decision that drives everything: underwriting is a long-running, multi-stage pipeline over unreliable external dependencies ending in an irreversible payout, so it is a persisted state machine driven by a saga, not a single request-response. Correctness comes from ordering (verify, then decide, then clear fraud, then pay), idempotency at every external boundary, explainable+versioned decisions, and an immutable audit log - not from one synchronous transaction.

Non-Functional Requirements (NFR)

  • Scale: 1M applications/day. That is ~12/sec average, ~40-120/sec at peak (evenings, salary-day, festival/sale-financing bursts). Reads (status checks, dashboards) outnumber writes maybe 20:1. This is moderate throughput, high stakes: the hard part is dependency orchestration, decision correctness, and exactly-once disbursal - not raw QPS.
  • Latency: the synchronous “apply” call returns in under ~1s with RECEIVED. The underwriting decision is async; the product target is a decision in seconds to a few minutes for the clean automated path (bounded by the slowest bureau/KYC call), with referred cases taking longer in a human queue. Disbursal after acceptance completes in minutes (rail-bound).
  • Availability: 99.95%+ for apply and status. Underwriting degrades safely: if a bureau or KYC provider is down, applications queue and retry, they are never auto-rejected or auto-approved on a missing check.
  • Consistency: strong for the decision record, the loan ledger, and disbursal. One application yields one decision; an approved loan is disbursed exactly once; a disbursal and the loan-creation ledger entries commit atomically. We never trade consistency for availability on the money path.
  • Durability: absolute, zero data loss. Once an application is accepted, it survives any node loss. Every decision input snapshot, model/rule version, fraud verdict, and disbursal record is durably replicated and never hard-deleted.
  • Auditability and explainability: every decision must be reproducible years later - the exact credit data, features, model version, rule set, and fraud signals that produced it - and every rejection must carry regulator-acceptable reason codes (adverse action). This is a legal requirement, not a nice-to-have.

The tension: the decision wants many slow, costly, flaky external calls, while the money path wants perfect ordering, exactly-once, and strong consistency. We resolve it by making underwriting an async saga over a durable state machine, caching and deduping external calls, gating disbursal behind an explicit terms-acceptance + fraud-clear step, and recording everything immutably.

Back-of-the-Envelope Estimation (BoE)

Real numbers, because they tell us where the cost and the risk actually are.

Write volume (applications):

1,000,000 applications/day
= 1,000,000 / 86,400 sec
≈ 11.6 apply/sec average
Peak (daily volume concentrates in ~10 active hours, plus salary-day/festival bursts):
  1M / 36,000 ≈ 28/sec active-hours avg
  burst ~3-4x => ~100/sec peak

Each application drives many external calls and state transitions:

per application, external calls:
  KYC/identity         1-2
  credit bureau        1-2  (sometimes multi-bureau)
  bank acct verify     1
  fraud scoring        1-2
  ML risk scoring      1
  disbursal (approved) 1   (~30-40% of apps approved => 0.35 avg)
  ≈ 6-8 external calls per application

1M apps * ~7 = ~7M external calls/day ≈ 80/sec avg, ~300/sec peak

This is the real load: not our QPS, but 7M billed, latency-variable calls to third parties. Bureau and KYC calls cost real money (often tens of rupees / tens of cents each) and run 200ms-3s. That drives two decisions: cache/dedupe aggressively and never call a dependency you do not need yet.

State transitions / events:

~10-14 events per application (received, kyc_started, kyc_passed, bureau_pulled,
  scored, fraud_cleared, decided, accepted, disbursing, disbursed ...)
1M * ~12 = ~12M events/day ≈ 140/sec avg, ~500/sec peak

Trivial for a well-provisioned SQL primary. The number that matters is that each event is durable, ordered, and immutable.

Storage per year:

application row              ~1 KB  (applicant, amount, target acct, state)
decision snapshot            ~4 KB  (credit data pulled, features, model out, reasons)
fraud + kyc records          ~2 KB
event log (~12 * 300B)       ~3.6 KB
loan + schedule (approved)   ~1 KB * 0.35
------------------------------------------------
≈ 11 KB per application, round to ~12 KB with indexes/overhead

1M/day * 12 KB ≈ 12 GB/day
* 365 ≈ 4.4 TB/year

Loan and decision records are retained 7-10 years by regulation and never deleted, so plan ~30-40 TB over the window. Modest; the story is retention and immutability (WORM), not sharding for space.

Cache sizing (dedupe / idempotency):

Idempotency keys for apply + disbursal, kept ~7-30 days:
  ~1M/day * 14 days ≈ 14M keys * ~300B ≈ 4.2 GB
Credit-report cache (a report is reusable for a window, e.g. 24-72h):
  cache by applicant PAN/SSN; hit rate reduces bureau spend directly.
  active distinct applicants/day ~0.8M * ~5 KB report ≈ 4 GB/day of cache

Read load (status + dashboards):

Applicants poll status, ops dashboards query queues:
  ~20x write reads => ~230/sec avg, ~2000/sec peak
Served from a read replica / read model, never the transactional primary.

The headline: raw write QPS is small (~12/sec); the cost and the risk are the millions of slow, paid external calls, the explainable/reproducible decision, and the exactly-once irreversible disbursal. The design spends its effort there.

High-Level Design (HLD)

The spine: an Application API accepts and dedupes an application, an Underwriting Orchestrator (saga) drives a persisted state machine through KYC, Credit Bureau, Fraud, and ML Risk Scoring services (each with its own cache and idempotency), a Decision Engine combines model output with policy rules into one explainable verdict, a Disbursal Service moves money exactly once over a Payment Rail and creates the loan atomically, and an append-only Audit Log records every input and transition.

                 ┌───────────────────────────────────────┐
                 │  Applicant (app / partner API)          │
                 └──────────────────┬────────────────────┘
                                    │ POST /applications (Idempotency-Key)
                 ┌──────────────────▼────────────────────┐
                 │           Application API               │
                 │  authN/Z + validate + idempotency       │
                 │  Stateless.                              │
                 └──────┬───────────────────────┬─────────┘
            idempotency │                       │ status reads
               check    │                       └────────────┐
            ┌───────────▼─────┐             ┌────────────────▼───────┐
            │ Idempotency /    │             │  Status Read Model      │
            │ App Store (SQL)  │             │  (per-applicant, NoSQL) │
            └───────────┬─────┘             └─────────────────────────┘
                        │ app.received (outbox -> bus)
            ┌───────────▼────────────────────────────────┐
            │       Underwriting Orchestrator (Saga)       │
            │  drives the state machine, step by step      │
            └─┬────────┬─────────┬─────────┬───────┬───────┘
      kyc     │  credit│   fraud │   score │  decide│
     ┌────────▼┐ ┌─────▼────┐ ┌──▼──────┐ ┌─▼─────┐ ┌▼────────────────┐
     │  KYC /   │ │ Credit   │ │ Fraud   │ │ ML    │ │ Decision Engine  │
     │ Identity │ │ Bureau   │ │ Service │ │ Risk  │ │ model + policy   │
     │ Service  │ │ Service  │ │ (graph) │ │ Score │ │ rules -> verdict │
     └────┬─────┘ └────┬─────┘ └────┬────┘ └───┬───┘ └───────┬─────────┘
          │ provider   │ bureau     │ features │ features     │ APPROVE/REJECT/REFER
     ┌────▼─────┐ ┌────▼─────┐ ┌────▼────┐ ┌───▼────────┐     │
     │ KYC/AML  │ │ CIBIL /  │ │ Device/ │ │ Feature    │  ┌──▼───────────────┐
     │ provider │ │ Experian │ │ id graph│ │ Store      │  │  Manual Underwrite│
     └──────────┘ └──────────┘ └─────────┘ └────────────┘  │  Queue (REFER)    │
                                                            └──────────────────┘
                 ┌───────────────────────────────────────┐
   APPROVED +    │        Disbursal Service               │
   accepted ────►│  exactly-once payout + create loan      │
                 │  atomically; deterministic rail key     │
                 └──────┬───────────────────────┬─────────┘
                        │ ISO/IMPS/NEFT msg      │ ledger + loan + schedule (one txn)
                 ┌──────▼───────────┐     ┌──────▼──────────┐
                 │  Payment Rail     │     │  Loan Ledger     │
                 │  (IMPS/NEFT/UPI)  │     │  (SQL, ACID)     │
                 │  irreversible     │     └──────────────────┘
                 └──────┬───────────┘
                        │ ack + settlement + returns
                 ┌──────▼──────────────────────────────────┐
                 │  Confirmation Consumer + Reconciliation    │
                 │  DISBURSED / RETURNED / FAILED; daily recon │
                 └────────────────────────────────────────────┘

        ── every transition ──►  Append-only Audit Log (immutable, WORM)

Synchronous path (what the applicant waits on):

  1. Client calls POST /applications with {amount, tenure, purpose, applicant, target_account} and a client-generated Idempotency-Key.
  2. Application API authenticates, validates the payload (amount in range, required fields, bank account format), and hits the idempotency layer: a known key returns the stored original response and does nothing else.
  3. On a new key, it writes the applications row in state RECEIVED and an outbox event in one ACID transaction, and returns 202 Accepted with application_id and state UNDER_REVIEW. The slow work is now the Orchestrator’s problem.

Asynchronous path (the saga drives underwriting):

  1. The Orchestrator consumes app.received and walks the state machine. It runs KYC first (cheap-to-fail: no point pulling a bureau report for a fake identity), then fraud pre-screen, then credit bureau pull, then assembles features and calls ML Risk Scoring, then the Decision Engine combines model output + policy rules.
  2. The outcome is one of APPROVED, REJECTED (with reason codes), or REFER (to a human underwriting queue). Everything - inputs, versions, verdict - is snapshotted to the immutable decision record.
  3. On APPROVED, the applicant is shown an offer. When they accept terms, a final fraud re-check runs (fraud signals age), then the Disbursal Service creates the loan + schedule and pays out exactly once over the rail in a saga. State -> DISBURSING -> DISBURSED.
  4. The rail acknowledges and later confirms settlement; the Confirmation Consumer advances to DISBURSED or handles RETURNED/FAILED. Reconciliation matches our ledger against the rail’s statement daily.

The structural insight: there is no single transaction spanning five external systems and a payout, so underwriting is a durable, monotonic state machine plus a saga. Each stage is an idempotent, cached, recoverable step; the only strongly-consistent moment is the atomic loan-creation + disbursal commit; everything is recorded immutably.

Component Deep Dive

The hard parts, naive-first then evolved: (1) orchestrating slow, flaky, costly external dependencies, (2) the ML risk decision made explainable and reproducible, (3) fraud detection before money moves, (4) exactly-once irreversible disbursal with atomic loan creation.

1. Orchestrating slow, flaky, costly external dependencies

The job: take an application through KYC, fraud, bureau, and scoring - where each call is 200ms-3s, can fail or time out, is rate-limited, and costs money - and still reach a decision reliably without paying for calls you did not need or getting stuck when one provider is down.

Naive approach: one synchronous request that calls everything in line

def apply(req):
    kyc   = kyc_provider.verify(req.identity)        # 1-3s
    report = bureau.pull(req.pan)                     # 1-3s
    fraud = fraud.score(req)                          # 100ms-1s
    score = model.score(features(kyc, report, fraud)) # 50-200ms
    decision = decide(score)
    return decision                                   # client waited 3-8s

Where it breaks:

  • The client holds a connection for 3-8 seconds, and p99 is worse because it is the sum of every dependency’s tail. One slow bureau call makes the whole request time out, and the client retries - re-running KYC and re-pulling the bureau, paying twice.
  • Any dependency outage fails the whole application. Bureau down at 9pm means every application at 9pm fails. There is no “retry just that step.”
  • No recovery. A crash mid-flight loses all the work done so far; the retry starts from zero and re-pays for KYC and bureau.
  • Wasted spend. It pulls a bureau report (costly) even when KYC already failed (fake identity) - money spent on applications that can never be approved.

A synchronous fan-out over paid, flaky dependencies is slow at the tail, non-recoverable, and burns money on doomed applications.

Evolved approach: async saga over a persisted state machine, with per-step idempotency, caching, and circuit breakers

Make underwriting a durable state machine the Orchestrator drives one step at a time, each step persisted before and after.

1. Order steps cheapest-and-most-eliminating first. KYC and a lightweight fraud pre-screen run before the expensive bureau pull. If identity fails or the application is an obvious fraud (known-bad device, impossible velocity), we reject without ever paying the bureau. Only applications that survive the cheap gates incur the costly calls.

RECEIVED -> KYC_CHECK -> FRAUD_PRESCREEN -> BUREAU_PULL -> SCORING -> DECISION
                 |              |                                        |
                 v              v                                        v
              KYC_FAIL      FRAUD_BLOCK                    APPROVED / REJECTED / REFER
              (reject)       (reject)

2. Each external call is idempotent and cached. The Orchestrator derives a deterministic key per (application_id, step) and the service dedupes on it. Results are persisted so a re-drive after a crash reuses the KYC result or bureau report rather than re-calling. Bureau reports are additionally cached by applicant identity for a freshness window (say 24-72h), so a reapply or a multi-product application reuses one paid pull.

def run_step(app, step):
    if app.has_result(step):            # persisted from a prior attempt
        return app.result(step)         # no external call, no spend
    key = f"{app.id}:{step}"
    res = call_with_breaker(step, app, idempotency_key=key)
    db.persist_step_result(app.id, step, res)   # committed before advancing state
    return res

3. Circuit breakers + typed retries + bulkheads per dependency. Each provider sits behind its own circuit breaker and bounded retry (exponential backoff + jitter, only on transient errors). If the bureau is down, its breaker opens and those applications park in BUREAU_PULL and retry later - they do not fail, and they do not block KYC for other applications (separate thread pools / bulkheads so one slow dependency cannot exhaust all workers).

4. The outbox wakes the saga durably. app.received is written in the same transaction as the application row; a relay publishes at-least-once; the Orchestrator dedupes on application_id. No window where an application exists but nothing drives it.

5. A recovery sweeper scans for applications stuck in any non-terminal state past an SLA and re-drives them. Because every step is idempotent and its result persisted, re-driving is safe and free for already-completed steps.

6. Parallelize independent steps where safe. Once KYC passes, the bureau pull and the fraud deep-scan have no ordering dependency on each other - run them concurrently and join before scoring, cutting wall-clock without extra risk. (KYC still gates both, because neither should run for an unverified identity.)

Why this is right: the saga turns a fragile 8-second synchronous call into a resumable pipeline where each paid step runs at most once, failures isolate to the step and provider that failed, outages degrade to “retry later” instead of “reject,” and cheap eliminating checks run before expensive ones so you never pay a bureau for an application KYC already killed.

2. The ML risk decision: explainable, reproducible, and policy-gated

The job: turn credit data + features into an approve/reject/refer decision, an approved amount, and a rate - and make that decision explainable (reason codes for adverse action), reproducible years later, and governed by policy, not just a raw model probability.

Naive approach: call the model, threshold the probability, store the yes/no

def decide(app):
    p_default = model.predict(features(app))   # 0.0 - 1.0
    return "APPROVE" if p_default < 0.2 else "REJECT"

Where it breaks:

  • Not explainable. A regulator (and the law - adverse action notices, fair-lending rules) requires why someone was rejected. “The model said 0.34” is not a reason code and is not defensible.
  • Not reproducible. If you only store the yes/no, you cannot answer “what did the model see?” six months later when the applicant disputes it or an auditor samples it. The model version, the feature values, and the bureau snapshot are gone.
  • No policy layer. A pure model will happily approve a loan that violates a hard rule (applicant underage, loan amount over the product cap, exposure limit exceeded, a required document missing). Business and regulatory policy are not negotiable by a probability.
  • Model drift and silent failure. A model silently degrading or a feature pipeline returning nulls skews every decision, and nothing catches it.

A bare threshold on a probability is unexplainable, irreproducible, and unsafe.

Evolved approach: a decisioning pipeline - feature snapshot, versioned model, reason codes, hard-rule policy layer, immutable record

Separate scoring (the model) from deciding (policy over the score), and snapshot everything.

1. Snapshot the inputs. At decision time, assemble a frozen feature vector from the bureau report, KYC result, fraud signals, and internal history, and store it with the decision. The bureau report itself (raw) is stored too. This is what makes the decision reproducible: given the snapshot and the model version, you can replay the exact score.

2. Versioned model as a scoring service. The model is deployed behind a scoring service with an explicit model_version. The decision record stores the version. Rolling a new model never rewrites old decisions, and you can always load the version that made a past decision. Models are shadow-tested and canaried before taking traffic.

3. Reason codes / explainability. The scoring service returns not just p_default but the top contributing factors (reason codes via SHAP/feature attribution mapped to a regulator-approved code list: “high credit utilization,” “short credit history,” “recent delinquencies”). Rejections carry these codes into the adverse-action notice.

4. Policy rules on top of the model (the decision engine). The model output is an input to a deterministic rules layer that enforces hard constraints the model may not override:

LayerExample ruleEffect
Hard eligibilityage >= 18, valid KYC, bureau score >= floor, not on exclusion listfail => REJECT regardless of model
Exposure / limitsrequested amount <= product cap; total exposure <= per-customer capcap the approved amount or REFER
Risk bandsmodel p_default band -> approve / refer / reject + rate tiersets rate and amount
AffordabilityEMI <= X% of declared/derived income (FOIR)reduce amount or REFER
Manual review triggersborderline score, high amount, thin file, conflicting signalsREFER to human underwriter

The engine produces the verdict: decision, approved amount, rate, reason codes, and whether it routes to the manual queue. Policy is versioned too (policy_version), so “which rules were in effect” is reproducible.

5. Guardrails against drift and bad features. Feature values are validated (ranges, null checks) before scoring; a null-heavy or out-of-range vector routes to REFER rather than scoring on garbage. Score distributions are monitored; a sudden shift pages the ML team and can trip a policy to be more conservative.

6. REFER is a first-class outcome. Not every application should be auto-decided. Borderline, high-value, or thin-file cases go to a manual underwriting queue where a human sees the same snapshot and reason codes and records their decision - itself audited. This keeps the automated path safe and gives a defensible answer for the hard cases.

Why this is right: splitting scoring from deciding lets the model inform while policy governs; snapshotting inputs + versioning the model and rules makes every decision reproducible and explainable; reason codes satisfy adverse-action law; and REFER plus drift guardrails mean the system fails to a human or a conservative default instead of silently approving on a broken model.

3. Fraud detection before money moves

The job: catch fraud - synthetic identities, stolen identities, fraud rings reusing devices/accounts, velocity abuse - before approval and again before disbursal, because an approved-and-disbursed fraudulent loan is money gone that you will likely never recover.

Naive approach: a static rules blocklist checked once at apply time

def fraud_check(app):
    if app.device_id in blocklist: return BLOCK
    if app.ip in blocklist:        return BLOCK
    return PASS

Where it breaks:

  • Static lists miss rings. A fraud ring using fresh devices and SIMs is not on any blocklist. The tell is the pattern - 50 applications in an hour from the same device fingerprint, bank account, or address across different “identities” - which a per-application list check cannot see.
  • Synthetic identities pass KYC. An identity assembled from a real SSN/PAN + fake name can clear document checks. You catch it by linkage (this “new” person shares a phone/device/account with three charged-off loans), not by a single-record lookup.
  • Checked once, at the wrong time. Fraud signals change between application and disbursal (which may be hours/days later after terms acceptance). A one-time check at apply lets a profile that turned bad in the interim still get paid.
  • No velocity / graph awareness. Reusing one bank account across many approved loans is invisible to a stateless per-request check.

A static, single-point blocklist catches yesterday’s fraud and misses the structure of today’s.

Evolved approach: layered fraud - fast pre-screen, graph/velocity deep-scan, ML fraud score, and a mandatory re-check at disbursal

Fraud is two gates (pre-approval and pre-disbursal), each layered fast-to-slow.

1. Fast pre-screen (before the bureau pull). Cheap deterministic checks: known-bad device/IP/account lists, impossible velocity (same device/account/phone over a threshold in a window), geovelocity, disposable-email/VOIP-number heuristics. Catches the obvious and cheap-to-catch, and does it before we pay the bureau.

2. Identity/device graph for linkage (the ring catcher). Maintain a graph linking applications to shared entities - device fingerprint, phone, email, bank account, address, IP. At decision time, query the applicant’s neighborhood: how many other identities share this device/account? Any linked to charged-off or confirmed-fraud loans? A dense cluster of “distinct” applicants around one device/account is the signature of a ring or synthetic-identity farm.

query_graph(application):
  neighbors = graph.ego(application, hops=2,
                        via=[device, phone, email, bank_account, address])
  features = {
    distinct_identities_on_device,
    accounts_sharing_this_bank_acct,
    links_to_known_fraud,
    cluster_density,
    velocity_1h / velocity_24h,
  }
  return features   # fed into the fraud ML score

3. ML fraud score. Graph features + behavioral signals (how the form was filled, time-on-page, copy-paste patterns) + historical labels feed a fraud model producing a fraud probability and reason codes. This complements the credit-risk model: credit risk asks “will they repay?”; fraud asks “are they real and are they who they say?” High fraud score => FRAUD_BLOCK or REFER to a fraud-ops queue.

4. Mandatory re-check at disbursal (fails closed). Because disbursal can happen hours/days after approval (terms acceptance), a fresh fraud re-evaluation runs immediately before money moves. New velocity (the account now appears on 20 approved loans), a newly confirmed-fraud link, or a changed device blocks disbursal even on an approved application. If the fraud service is unavailable, disbursal holds, it does not proceed unscreened.

5. Feedback loop. Confirmed fraud (charge-offs, manual confirmations) is written back as labels/edges into the graph and the next model training set, so the system learns the newest patterns instead of relying on a static list.

Why this is right: layering fast deterministic checks before expensive ones keeps the clean majority cheap; the identity/device graph turns per-application blindness into ring-level detection and catches synthetic identities by linkage; an ML score generalizes beyond hand-written rules; and re-checking at disbursal with fail-closed semantics closes the window where a profile goes bad between approval and payout - the exact window a fraudster exploits.

4. Exactly-once irreversible disbursal with atomic loan creation

The job: when an approved loan’s terms are accepted and fraud re-clears, move the principal to the verified bank account exactly once over an irreversible rail, and create the loan + repayment schedule atomically with the payout - even though the service can crash mid-flight and the rail call can fail ambiguously.

Naive approach: create the loan, then call the rail, then mark disbursed

def disburse(app):
    loan = create_loan(app)          # write loan + schedule
    rail.transfer(app.target_account, app.amount)   # money leaves
    loan.status = 'DISBURSED'
    save(loan)

Where it breaks:

  • Crash between transfer and record. The rail pays out, then the service dies before marking DISBURSED. On retry the application looks un-disbursed and pays again. Two irreversible payouts, real money gone, a manual clawback you may lose.
  • Ambiguous rail failure. You call transfer, the connection drops before the response. Paid or not? Retrying may double-pay; not retrying may leave an approved loan unpaid forever.
  • Loan created but payout failed (or vice versa). Create-loan and pay-out are two separate writes with no atomicity. A crash leaves a loan on the books with no money sent, or money sent with no loan/schedule - both are reconciliation nightmares and customer-facing wrong.
  • Double disbursal from a double-click / duplicate accept. Two “accept terms” events race and both trigger disbursal.

At-least-once plumbing plus a non-idempotent, irreversible payout is the worst case in the system: duplicated real money you cannot undo.

Evolved approach: deterministic rail key + pre-written DISBURSING state + atomic local loan creation + saga with reconciliation

Compose exactly-once from ordering, a provider idempotency key, a durable intent-to-pay record, and an atomic local commit.

1. Disbursal is reachable only from APPROVED + ACCEPTED + FRAUD_RECLEARED. The state machine forbids paying out an un-accepted or un-recleared loan. The Disbursal Service reads persisted state, not an in-memory flag, so a restart resumes correctly. Terms acceptance is itself idempotent (dedup on application_id) so a double-click produces one disbursal trigger.

2. Atomic local write first: create the loan, schedule, ledger hold, and a DISBURSING intent in one ACID transaction. Before any external call, in a single committed transaction we create the loan, the repayment schedule, the double-entry ledger entries (debit a funding/disbursal clearing account, credit a payable), set state DISBURSING, store the exact rail_idem_key we will use, and insert the outbox trigger.

BEGIN;
  INSERT INTO loans (loan_id, application_id, principal, rate, tenure, status)
    VALUES (:loan, :app, :amt, :rate, :tenure, 'DISBURSING');
  INSERT INTO repayment_schedule (...);            -- all EMIs
  INSERT INTO ledger_entries (...) VALUES          -- double-entry, sums to zero
    (:tx, :funding_clearing, 'DEBIT',  :amt, 'DISBURSAL'),
    (:tx, :payable_rail,     'CREDIT', :amt, 'DISBURSAL');
  UPDATE applications SET state='DISBURSING', rail_idem_key=:key
    WHERE application_id=:app AND state='ACCEPTED';   -- 0 rows => illegal, ROLLBACK
  INSERT INTO outbox (topic, payload) VALUES ('disbursal.send', ...);
COMMIT;

The loan and the intent to pay commit together. After any crash, recovery sees DISBURSING with a known rail key and knows exactly what to send - never “did I pay?”

3. Deterministic idempotency key toward the rail. We derive rail_idem_key = loan_id + ":disburse" and pass the same key on every attempt. IMPS/NEFT/UPI gateways (and ACH processors) accept a client reference / idempotency key and guarantee one payout per key; a retry after an ambiguous failure either completes the original or returns its known result.

def send_disbursal(loan):
    if loan.status in ('DISBURSED',):          # already paid
        return
    if loan.status != 'DISBURSING':
        raise IllegalTransition
    key = f"{loan.id}:disburse"
    resp = rail.transfer(account=loan.target_account,
                         amount=loan.principal,
                         idempotency_key=key)    # SAME key on every retry
    if resp.status == 'ACCEPTED':
        db.set_state(loan.id, 'SENT', rail_ref=resp.rail_ref)   # await settlement
    elif resp.permanent_reject:
        with db.transaction():
            ledger.reverse_hold(loan)            # unwind the clearing entries
            db.set_state(loan.id, 'DISBURSAL_FAILED')
            app.set_state('DISBURSAL_FAILED')    # offer a retry with corrected details
    else:
        raise RetryableError                     # transient: re-drive with backoff

4. Settlement confirmation flips to DISBURSED. The rail acks, then confirms settlement; the Confirmation Consumer advances SENT -> DISBURSED (money truly delivered, loan active) or handles RETURNED (bad account: reverse the ledger, mark failed, prompt the customer to fix the account) idempotently - a duplicate callback finds the loan already terminal and no-ops.

5. Recovery sweeper + reconciliation. A sweeper re-drives loans stuck in DISBURSING/SENT past the settlement window, safe because the rail key makes a re-send a no-op if the first landed. Daily reconciliation matches our disbursal ledger against the rail’s settlement file on the shared reference; any break (we think disbursed, rail did not; rail disbursed, we have no record) posts to a suspense account and is worked by ops. This is the only thing that proves the money actually moved as recorded.

Why this is right: you cannot get exactly-once from an irreversible rail on trust, so you build it - the atomic local commit ties loan creation to the intent to pay, the pre-written DISBURSING state makes recovery deterministic, and the deterministic rail key makes the external payout happen once no matter how many times you call. Reconciliation catches the residual divergence. At-least-once plumbing over an irreversible rail becomes exactly-once money movement with a loan that always matches the payout.

API Design & Data Schema

API

All mutating calls require an Idempotency-Key.

POST /api/v1/applications
Headers: Authorization: Bearer <token>
         Idempotency-Key: <client-generated UUID, reused on retries>
Body:
{
  "amount":   50000000,          // integer minor units (paise) = 500000.00
  "currency": "INR",
  "tenure_months": 24,
  "purpose": "personal",
  "applicant": {
    "name": "Asha Rao",
    "dob": "1994-02-11",
    "pan": "ABCDE1234F",
    "phone": "+91...",
    "address": { ... }
  },
  "target_account": { "ifsc": "HDFC0000123", "account_no": "...", "name": "Asha Rao" }
}
Response 202:
{
  "application_id": "app_9f3a...",
  "state": "UNDER_REVIEW",
  "amount": 50000000,
  "created_at": "2026-11-12T09:04:00Z"
}
Errors:
  400  invalid payload / bad IFSC or PAN format
  401  bad token
  409  idempotency key still in progress (retry shortly)
  422  immediately ineligible (e.g. amount outside product range)

Apply returns 202, not 201: accepted and under review, final decision is async. Reads and lifecycle:

GET  /api/v1/applications/{id}               -> state, decision summary, offer (if any)
GET  /api/v1/applications/{id}/events        -> immutable event log (audit view)
POST /api/v1/applications/{id}/accept        -> accept the offer; triggers disbursal saga
POST /api/v1/applications/{id}/cancel        -> only valid before DISBURSING
GET  /api/v1/applications/{id}/decision      -> reason codes, model/policy versions (audit)
POST /api/v1/webhooks/{rail}                 -> rail ack / settlement / return callbacks
POST /api/v1/webhooks/kyc                    -> async KYC provider callbacks
GET  /api/v1/ops/queue/refer                 -> manual underwriting queue (internal)
GET  /api/v1/ops/queue/fraud                 -> fraud review queue (internal)

Data stores

Different access patterns, different stores. Be explicit.

1. Applications, decisions, loans, ledger, state machine - SQL, ACID, synchronously replicated. This is where the one-decision-per-application invariant, the atomic loan-creation + disbursal commit, the two-bucket/clearing ledger, and the monotonic state transitions live. A relational database (PostgreSQL, or distributed SQL like CockroachDB/Spanner for HA) is the correct, non-negotiable choice. At ~12 writes/sec we do not need sharding for throughput; we run a primary with synchronous standby replicas and automatic failover for durability. NoSQL eventual consistency would reintroduce double-decision and double-disbursal races.

Table: applications                   -- one row per application, state projection
  application_id    UUID    PRIMARY KEY
  applicant_id      BIGINT  INDEX
  amount            BIGINT  -- minor units
  currency          CHAR(3)
  tenure_months     INT
  purpose           STRING
  target_account    JSON    -- ifsc, account_no, name (verified flag)
  state             STRING  -- RECEIVED|KYC_CHECK|FRAUD_PRESCREEN|BUREAU_PULL|
                            --  SCORING|DECISION|APPROVED|REJECTED|REFER|ACCEPTED|
                            --  DISBURSING|SENT|DISBURSED|DISBURSAL_FAILED|CANCELLED
  version           BIGINT  -- optimistic guard for transitions
  rail_idem_key     STRING  NULL
  decision_id       UUID    NULL
  created_at        TIMESTAMP
  updated_at        TIMESTAMP
  INDEX (state)                        -- for ops queues / sweeper scans

Table: decisions                      -- APPEND ONLY, immutable, reproducible
  decision_id       UUID    PRIMARY KEY
  application_id    UUID    INDEX
  outcome           STRING  -- APPROVED | REJECTED | REFER
  approved_amount   BIGINT  NULL
  rate_bps          INT     NULL
  p_default         FLOAT
  fraud_score       FLOAT
  reason_codes      JSON    -- adverse-action codes
  feature_snapshot  JSON    -- frozen feature vector
  bureau_snapshot   JSON    -- raw report (or pointer to blob)
  model_version     STRING
  policy_version    STRING
  decided_by        STRING  -- system | underwriter:<id>
  created_at        TIMESTAMP

Table: loans
  loan_id           UUID    PRIMARY KEY
  application_id    UUID    UNIQUE      -- one loan per application
  principal         BIGINT
  rate_bps          INT
  tenure_months     INT
  status            STRING  -- DISBURSING|SENT|DISBURSED|DISBURSAL_FAILED
  target_account    JSON
  rail_ref          STRING  NULL
  created_at        TIMESTAMP

Table: repayment_schedule
  schedule_id       UUID    PRIMARY KEY
  loan_id           UUID    INDEX
  installment_no    INT
  due_date          DATE
  emi_amount        BIGINT
  UNIQUE(loan_id, installment_no)

Table: ledger_entries                 -- APPEND ONLY, immutable
  entry_id     UUID     PRIMARY KEY
  ref_id       UUID     INDEX     -- loan_id / disbursal id
  account_id   BIGINT   INDEX     -- internal GL account
  direction    CHAR(1)  -- D | C
  kind         STRING   -- DISBURSAL | REVERSAL | ADJUSTMENT
  amount       BIGINT
  currency     CHAR(3)
  created_at   TIMESTAMP
  Invariant: per ref_id, SUM(credits) - SUM(debits) = 0

Table: application_events             -- APPEND ONLY, hash-chained audit log
  event_id     UUID     PRIMARY KEY
  application_id UUID   INDEX
  seq          INT      -- per-application monotonic sequence
  type         STRING   -- state change, step result, decision, operator action
  actor        STRING   -- system | underwriter:<id> | fraud-ops | rail
  payload      JSON     -- old->new state, model/policy version, scores, rationale
  prev_hash    BYTES
  hash         BYTES    -- H(prev_hash || payload) : tamper evidence
  created_at   TIMESTAMP
  UNIQUE(application_id, seq)

Table: idempotency_keys
  idem_key     STRING   PRIMARY KEY   -- UNIQUE; the real dedup guarantee
  application_id UUID   NULL
  status       STRING   -- IN_PROGRESS | DONE
  response     JSON     -- stored original response for replay
  created_at   TIMESTAMP
  TTL/archive: 30 days hot

Table: outbox
  outbox_id    BIGINT   PRIMARY KEY
  payload      JSON
  topic        STRING
  sent         BOOL     INDEX (partial: WHERE sent=false)

2. Status / history read model - NoSQL wide-column (Cassandra/DynamoDB), by applicant. Applicant status polls and lists are read by applicant_id, never joined, tolerate eventual consistency. Built by a projector off the event bus so status reads never touch the transactional primary.

3. Fraud identity/device graph - graph store (Neo4j/JanusGraph) or a graph built on wide-column. Nodes: applications, devices, phones, emails, bank accounts, addresses. Edges: “used.” Queried for ego-network linkage and cluster density at decision and disbursal time. Updated continuously from applications and fraud labels.

4. Feature store - low-latency KV (Redis/DynamoDB) + offline store. Online store serves features to the scoring service at decision time; offline store (warehouse) is for training and for reproducing historical feature values.

5. Credit-report / document cache - KV with TTL + blob storage. Bureau reports cached by applicant identity for the freshness window to avoid re-paying; raw reports and KYC documents in encrypted blob storage, referenced from the decision snapshot.

6. Audit cold storage - WORM object storage. application_events, decisions, and raw bureau/KYC snapshots exported to write-once-read-many storage with a compliance lock for the 7-10 year retention window.

Why SQL for the core, specialized stores for the rest: the decision/loan/disbursal path needs multi-row ACID, the unique constraint that enforces one loan per application, monotonic state transitions, and same-transaction idempotency+outbox - exactly what relational databases exist for, and at this volume one primary handles it. Fraud linkage is inherently a graph query (NoSQL/SQL joins cannot cheaply do 2-hop ego networks), features need single-digit-ms KV reads, and status is key-scoped and eventual-consistency-tolerant. Match the store to the guarantee: absolute for money and decisions, specialized for graph and features, relaxed for read projections.

Bottlenecks & Scaling

Where it breaks first, in order, and the fix. At ~12 writes/sec, most “bottlenecks” here are correctness, dependency, and cost failures, not throughput ceilings.

1. Duplicate irreversible disbursal (breaks first, worst impact). A retry or crash pays out a second time, or a loan is created without a payout (or vice versa). Fix: atomic local commit of loan + schedule + ledger + DISBURSING intent in one ACID transaction; a client Idempotency-Key with a DB unique-constraint backstop; a deterministic rail idempotency key passed on every send; the pre-written DISBURSING state for deterministic recovery; the UNIQUE(application_id) on loans; consumers dedupe rail callbacks on loan_id. This is the single most important mechanism in the system.

2. Slow/flaky/costly external dependencies. Bureau, KYC, or fraud providers are slow, down, rate-limited, or billed per call; a synchronous fan-out would fail and re-pay. Fix: the async saga so the client never waits on the tail; per-step idempotency + persisted results so each paid call runs at most once and re-drives are free; credit-report cache by identity to avoid re-paying within a freshness window; circuit breakers + bulkheads + typed retries so one provider’s outage parks those applications and isolates from others; cheapest-eliminating-first ordering (KYC/fraud pre-screen before the paid bureau pull) so we never pay a bureau for a doomed application.

3. Dual-write problem (DB commit vs bus publish). Application accepted but the driving event lost leaves it stuck with nothing progressing it. Fix: the outbox pattern - trigger events written in the same ACID transaction as the state change; a relay publishes at-least-once; a recovery sweeper re-drives any application stuck in a non-terminal state past SLA (safe because steps are idempotent).

4. Decision not reproducible / not explainable. An auditor or a disputing applicant asks why, months later. Fix: immutable decisions record with the frozen feature + bureau snapshot, model_version, policy_version, p_default, fraud_score, and reason codes; the hash-chained event log; raw snapshots in WORM storage. Reproducing a past decision is loading the snapshot and the versioned model.

5. Fraud rings and synthetic identities / stale fraud at disbursal. Pattern-based fraud invisible to per-record checks; a profile that turns bad between approval and payout. Fix: the identity/device graph for linkage and velocity, an ML fraud score over graph+behavioral features, a two-gate design (pre-approval and a mandatory fail-closed re-check at disbursal), and a feedback loop writing confirmed fraud back as labels/edges.

6. Scoring/feature pipeline as a chokepoint or drift source. A slow, unavailable, or drifting model/feature pipeline stalls or skews every decision. Fix: the scoring service is stateless and horizontally scaled behind the feature store (KV, single-digit-ms); feature validation routes garbage vectors to REFER instead of scoring on nulls; drift monitoring pages and can trip a conservative policy; models are canaried/shadowed before taking traffic. If scoring is down, applications park and retry, never auto-approve.

7. Rail outage or ambiguous responses at disbursal. The rail is down or answers ambiguously; the loan is in limbo. Fix: queue and hold (DISBURSING/SENT persists) rather than dropping; typed retries with backoff+jitter; a circuit breaker and, where available, a secondary rail/PSP; a DLQ for payouts that exhaust retries, worked by ops. Availability degrades to “paying shortly,” never “lost or double-paid.”

8. Ledger drift from the rail (returns, failed credits, fees). Internal correctness cannot detect external divergence (a RETURNED credit to a closed account). Fix: daily/intraday reconciliation against the rail’s settlement file, a suspense account keeping the ledger balanced while breaks are worked, an owned break-management workflow, and RETURNED handling that reverses the ledger and prompts the customer to correct the account. Internal checks too: global SUM(ledger entries) == 0 and per-ref debits == credits.

9. Audit integrity and retention. Decisions and money movement must be immutable, tamper-evident, and retained for years with zero loss. Fix: append-only, hash-chained event and decision logs; WORM cold storage with compliance lock/legal hold; separation of duties (no update/delete on the logs); continuous export so nothing lives only on the hot primary.

10. Single points of failure / durability. Any node loss must lose no accepted application or decision. Fix: SQL with synchronous replication and automatic failover (no committed transaction lost); bus, graph, feature, and read stores replicated; stateless API, Orchestrator, and scoring horizontally scaled; the outbox, DLQ, and sweeper guarantee nothing in flight vanishes silently.

Shard keys / partitioning, stated plainly: at this volume the SQL core does not need sharding for throughput - it needs synchronous replication for durability. If volume grew 100x, shard applications/application_events by application_id, loans/ledger_entries by loan_id, and the read model by applicant_id for single-partition reverse-chronological reads. The fraud graph partitions by entity but must support cross-partition ego queries (that is why it is a graph store, not a sharded KV). Never shard by time - it concentrates all current writes on the newest partition. Kafka topics partition by application_id for per-application ordering.

Wrap-Up

The trade-offs that define this design:

  • No single transaction across five external systems and a payout, so a saga over a monotonic state machine. Underwriting is a resumable, persisted pipeline; correctness comes from ordering, per-step idempotency, and an append-only state machine - not a synchronous request we cannot make reliable.
  • Cheapest-eliminating checks first, everything cached and deduped. KYC and fraud pre-screen gate the paid bureau pull; results are persisted and reports cached by identity. The real scaling constraint is millions of slow, billed external calls, so we minimize and dedupe them.
  • Scoring informs, policy decides, and every decision is reproducible. The ML model outputs a probability and reason codes; a versioned policy layer enforces hard rules and routes borderline cases to humans. Snapshotting inputs + versioning model and rules makes every decision explainable and replayable years later.
  • Fraud is a graph problem checked twice, fail-closed. Linkage and velocity over an identity/device graph catch rings and synthetic identities a blocklist cannot; a mandatory re-check at disbursal closes the window between approval and payout.
  • Exactly-once against an irreversible rail, with the loan tied to the payout. An atomic local commit of loan + ledger + DISBURSING intent, a deterministic rail key on every send, pre-written recovery state, and daily reconciliation turn at-least-once plumbing into exactly-once disbursal where the books always match the money.

One-line summary: a moderate-throughput, high-stakes loan platform accepts an application and drives it through an async saga - KYC, fraud pre-screen, a cached credit-bureau pull, ML risk scoring, and a versioned policy engine - to one explainable, reproducible decision, re-checks fraud over an identity graph just before paying, then disburses to the verified bank account exactly once via a deterministic rail key with the loan created atomically against the payout, recording every input and transition to a tamper-evident WORM audit log and reconciling daily - so each application gets one defensible decision and an approved loan is paid exactly once, only after identity and fraud clear.