Venmo looks like a wallet with a Twitter feed bolted on, and that framing is exactly what makes it interesting. The money side demands strong consistency and an immutable ledger - a balance that goes negative from a race is theft, a double-send is a refund and an angry user. The social side demands fan-out, ranking, and privacy - a feed of “Alice paid Bob for dinner” that must respect who is allowed to see what, at read rates 100x the write rate. And sitting across both is fraud: an account takeover draining a balance, a scammer collecting payments for goods that never ship, a stolen card funding a top-up that gets clawed back. Get the ledger right but the fraud wrong and you lose real money; get fraud right but the feed wrong and you leak who paid whom.

The core tension: money wants strong consistency and small blast radius, the feed wants cheap fan-out and eventual consistency, and fraud wants to inspect every transaction in the write path without slowing it down. This design keeps a tiny strongly-consistent ledger core, pushes the feed to async projections with privacy enforced at write and read time, and runs fraud as a synchronous-but-fast risk gate plus an async model. Let me build it end to end.

Functional Requirements (FR)

In scope:

  • Send money by identity. A user sends money to a friend by @username, phone number, or email - not by an internal account id they will never know. Identity resolution is a first-class feature, not an afterthought.
  • Request money. A user requests money from someone; the counterparty approves or declines. A request is not a transfer until approved.
  • Hold a balance and move money atomically. Each user has a Venmo balance. A transfer debits the sender exactly what it credits the receiver, or neither happens. Balance never goes negative.
  • Fund and cash out. Top up the balance from a linked bank/card, or cash out the balance to a linked bank. These touch external rails and settle asynchronously.
  • Social feed of transactions. Payments produce a feed entry (“Alice paid Bob - pizza”) visible per a privacy setting: public, friends-only, or private. A user sees a personalized feed of their friends’ payments plus their own.
  • Privacy controls. Per-transaction and default privacy. Amounts are never shown in the feed regardless - only participants, note, and time. The note and emoji are the social payload.
  • Fraud detection. Every money movement passes a risk decision. Suspicious transfers are challenged, held, or blocked. Account takeover and stolen-instrument funding are caught.

Explicitly out of scope (own the scope by naming it):

  • KYC / onboarding identity verification. I assume users are onboarded and identity-verified; I resolve handles to accounts, I do not build the KYC pipeline.
  • The card networks and bank rails. I integrate a payment processor and ACH as black boxes; I do not design Visa or the ACH network.
  • Chat, comments, likes beyond the feed entry. The feed carries payments and reactions on them; I do not build a full messaging product.
  • Crypto, Venmo debit card, merchant checkout. Consumer P2P is the surface here.

The decision that drives everything: money is an append-only double-entry ledger, the balance is a cached projection of it, and the feed is a second projection of the same payment events - with privacy applied when the projection is built and re-checked when it is read. Every choice below follows from separating the strongly-consistent money core from the two eventually-consistent read models (balance cache, feed) built off it.

Non-Functional Requirements (NFR)

  • Scale: 80M registered users, ~25M DAU. Assume ~2 money movements per active user per day at average, with strong daily and weekly peaks (Friday night, rent day, splitting a group trip). Feed reads dominate massively - every app open loads a feed.
  • Latency: the send-money path the user waits on returns p99 under ~500ms including the fraud gate (external rail settlement is async). Identity resolution (typeahead as they type a handle) p99 under ~50ms. Feed load p99 under ~200ms.
  • Availability: 99.99% on read and send. A user must almost always be able to check balance, resolve a friend, and initiate a send. Money movement can degrade to “accepted, settling” but never silently fail.
  • Consistency: strong for balances - a balance never goes negative from a race, a transfer is atomic, the ledger always balances (debits = credits). This is the one place we do not trade consistency for availability. The feed and friend graph are eventually consistent; a payment appearing in a friend’s feed a second late is fine.
  • Durability: absolute for money. Once committed, a transaction survives any single node loss. Ledger data is synchronously replicated and never hard-deleted; corrections are new entries.
  • Privacy: a payment must never be visible to someone the privacy setting excludes. This is a correctness requirement, not a preference - a leak here is a headline.

The tension to state up front: the money core wants to be small, strongly consistent, and carefully sharded; the feed wants to be huge, cheap, and eventually consistent; fraud wants to sit in the write path but must not become the latency bottleneck. We resolve it by keeping the ACID ledger write tiny, making the fraud gate a fast synchronous lookup backed by an async model, and building the feed as a fan-out projection with privacy baked in.

Back-of-the-Envelope Estimation (BoE)

Real numbers with the arithmetic, because they set the architecture.

Transaction write volume:

25M DAU * ~2 money movements/day = 50M transactions/day
50,000,000 / 86,400 sec ≈ 580 transactions/sec average
Peak (Friday night, rent day ~8x avg) ≈ 4,600 transactions/sec write

Each transfer is at least two ledger entries (debit sender, credit receiver), and produces one payment event that fans out to the feed.

Peak ledger-entry writes ≈ 4,600 tx/sec * 2 = ~9,200 entries/sec

Modest. Money throughput is never the hard part; correctness under concurrency is.

Feed read volume:

25M DAU * ~6 feed opens/day = 150M feed reads/day
≈ 1,700 reads/sec average, peak ~15,000 reads/sec
Add identity-resolution typeahead: every send starts with typing a handle,
25M DAU * ~3 lookups/day * (several keystrokes cached) ≈ a few thousand/sec

The read:write ratio on the social side is ~25:1 and on balance reads even higher. That gap is why the feed is a precomputed projection, not a live join over the graph.

Feed fan-out volume:

50M payments/day, each with a privacy setting.
Public/friends payments fan out to the sender's + receiver's friends.
Average friend count ~50; assume ~half of payments are friends-or-public.
25M fannable payments/day * ~50 friends = 1.25B feed-entry writes/day
≈ 14,500 feed writes/sec average, peak ~100K+/sec

This is the real volume driver - fan-out on write is 250x the payment rate. It forces a hybrid fan-out (push for normal users, pull for high-follower accounts), covered in the deep dive.

Ledger storage per year:

Per ledger entry ≈ 200 bytes with indexes (entry_id, tx_id, account_id,
  direction, amount minor units, currency, created_at, metadata_ref).
50M tx/day * 2 entries = 100M entries/day
100M * 200 bytes ≈ 20 GB/day
* 365 ≈ 7.3 TB/year

Ledger data is never deleted (7-10 year regulatory retention), so plan ~50-70 TB over the window. Hot recent data on fast storage, older partitions on cheaper cold storage but still queryable.

Feed storage:

1.25B feed entries/day. If we keep ~500 entries per user hot (a few weeks),
80M users * 500 * ~120 bytes ≈ 4.8 TB hot feed store, TTL'd and capped.

Feed entries are cheap and disposable - we can always rebuild them from the payment event log. They live in a wide-column store, capped per user, not kept forever.

Balance cache:

80M accounts * (account_id 8 + balance 8 + version 8 + currency 3) ≈ 27 bytes
Round to ~60 bytes: 80M * 60 ≈ 4.8 GB

Tiny - every balance fits in a small Redis cluster.

Identity index:

80M users, each with 1 username + ~1 phone + ~1 email = ~3 handles.
240M handle -> account_id mappings * ~60 bytes ≈ 14 GB.

Small enough to cache aggressively, which is what makes sub-50ms typeahead resolution possible.

The headline: raw money throughput (~5K tx/sec) is easy; the hard, expensive parts are the 100K+/sec privacy-aware feed fan-out, the strong-consistency guarantee on balances under concurrent sends, correct identity resolution, and a fraud gate that inspects every transaction without blowing the latency budget. That is where the design earns its keep.

High-Level Design (HLD)

The spine: an API gateway authenticates and routes, an Identity Service resolves handles to account ids, a Fraud/Risk gate scores the transfer synchronously, a strongly-consistent Ledger Service moves money in one ACID transaction under an overdraft guard and an idempotency key, and an event bus carries each payment.posted event to async consumers: the Feed Service (privacy-aware fan-out), the Notification Service, the Fraud model trainer, and the Settlement Worker for external funding/cash-out.

                 ┌─────────────────────────────┐
                 │   Client (mobile app)        │
                 └──────────────┬──────────────┘
                                │ POST /payments (Idempotency-Key)
                 ┌──────────────▼──────────────┐
                 │        API Gateway           │
                 │  authN/Z, rate limit, route  │
                 └───┬───────┬──────────┬───────┘
      resolve handle │       │ risk     │ reads
          ┌──────────▼──┐ ┌──▼────────┐ │
          │ Identity Svc │ │ Fraud Gate│ │
          │ handle->acct │ │ score/hold│ │
          │ (idx + cache)│ └──┬────────┘ │
          └──────────────┘    │ approve  │
                              ┌▼──────────▼──────────────┐
                              │      Ledger Service        │
                              │ ONE ACID tx:               │
                              │  guard balance>=amount     │
                              │  insert debit+credit       │
                              │  update balances(versioned)│
                              │  write tx row + outbox      │
                              └──┬──────────────────┬──────┘
                       writes    │                  │ outbox relay
                    ┌────────────▼──┐        ┌───────▼────────────┐
                    │  Ledger DB     │        │  Event Bus (Kafka) │
                    │ (SQL, sharded, │        │  payment.posted    │
                    │  sync replica) │        └─┬────┬─────┬───┬───┘
                    └────────────────┘          │    │     │   │
       ┌────────────────────────────────────────┘    │     │   │
       │                    ┌─────────────────────────┘     │   │
 ┌─────▼──────────┐  ┌──────▼────────────┐  ┌───────────────▼┐ ┌▼──────────────┐
 │ Feed Service    │  │ Notification Svc  │  │ Fraud Trainer  │ │ Settlement    │
 │ privacy-aware   │  │ push/email        │  │ features+model │ │ Worker        │
 │ fan-out (hybrid)│  └───────────────────┘  └────────────────┘ │ (bank/card    │
 └─────┬───────────┘                                            │  rails, ACH)  │
       │ writes                                                 └───┬───────────┘
 ┌─────▼───────────┐        ┌─────────────────┐                     │ webhooks
 │ Feed Store      │        │ Balance Cache    │              ┌──────▼──────────┐
 │ (wide-column,   │        │ (Redis)          │              │ Reconciliation   │
 │  capped/user)   │        └─────────────────┘              │ (ledger vs rails)│
 └─────────────────┘                                          └─────────────────┘

Send-money write flow (synchronous, the user is waiting):

  1. Client calls POST /payments with {to_handle, amount, note, privacy} and a client-generated Idempotency-Key. The handle is what the user typed (@bob, a phone, an email).
  2. Gateway authenticates and rate-limits. The Identity Service resolves to_handle -> to_account_id (and validates the sender’s account). A bad handle fails fast here.
  3. The Fraud Gate scores the transfer with low-latency features (velocity, device, relationship graph, amount vs history). It returns approve / challenge (step-up auth) / hold / block. Only approved transfers proceed to money movement.
  4. The Ledger Service checks the idempotency key (retry-safe), then in one ACID transaction: guards sender.balance >= amount with a conditional update (locks the row, prevents negative and races), inserts a debit and matching credit entry (double-entry, must sum to zero), bumps both balances with a version, writes the transaction row POSTED, and inserts a payment.posted row into the outbox - all committed atomically.
  5. Ledger returns success to the client. The outbox relay publishes payment.posted to Kafka.

Async flow (feed, notifications, fraud model, settlement):

  1. The Feed Service consumes payment.posted, applies the privacy setting, and fans the entry out to the appropriate friends’ feeds (push) - or records it for pull if a participant is a high-fan-out account.
  2. The Notification Service pushes “You received $20 from Alice” to the receiver.
  3. The Fraud Trainer consumes every event to update features and (offline) retrain the model; it also runs async deeper checks that can retroactively freeze funds.
  4. The Settlement Worker handles funding (top-up) and cash-out legs that touch external rails, tracked by a per-transaction state machine and proven correct by Reconciliation against the processor’s statements.

The structural insight: the strongly-consistent core is exactly one ACID transaction (the ledger write under a guard). Identity resolution and the fraud gate sit in front of it as fast lookups; the feed, notifications, model training, and external settlement all hang off the async event bus. The small core is what we scale carefully; the rest scales the easy way.

Component Deep Dive

The hard parts, naive-first then evolved: (1) identity resolution by username/phone, (2) strongly consistent balances under concurrent transfers, (3) the privacy-aware social feed and its fan-out, (4) real-time fraud detection.

1. Identity resolution - sending money by username or phone

The job: a user types @bob, +1-555-..., or [email protected], and we resolve it to exactly one account, fast, as they type, without ever sending money to the wrong person.

Naive approach: query the users table by handle on each send

Store username/phone/email as columns on users and SELECT ... WHERE username = ? when sending.

def resolve(handle):
    return db.query("SELECT account_id FROM users WHERE username = ?", handle)

Where it breaks:

  • Typeahead hammers the primary. Resolution runs on every keystroke of the recipient field, thousands/sec, against the transactional users table - the same database the ledger depends on. A social feature is now stealing capacity from money movement.
  • Multiple handle types, one lookup. A phone can be formatted five ways (+15551234567, 555-123-4567, (555) 123-4567). A raw column match misses them. Emails have case and dots. Usernames get changed. One column per type with exact match is brittle.
  • Ambiguity and reassignment. A username freed by one user and taken by another; a phone number recycled by the carrier. If we resolve by the current owner without care, money goes to whoever holds the handle now - potentially a stranger. This is a real Venmo-class incident.
  • No confirmation surface. Resolving silently to an id means the sender never confirms they picked the right “Bob” among many. Sending money is irreversible-ish; picking the wrong recipient is the classic support nightmare.

Exact-column lookup against the transactional DB is slow, brittle across formats, and dangerous around handle reassignment.

Evolved approach: a normalized identity index, separate store, with confirmation and reassignment safety

Build identity resolution as its own service backed by a dedicated identity index, not the transactional users table.

  • Normalize every handle to a canonical form on write. Phone numbers to E.164 (+15551234567), emails lowercased with provider-specific rules, usernames case-folded. Store (handle_type, canonical_handle) -> account_id. The client normalizes the same way for typeahead so cache keys match.
  • Dedicated index store, heavily cached. The mapping is ~14 GB (small), read-mostly, and latency-critical. Put it in a key-value store (or a search index for prefix typeahead) with a Redis cache in front. Typeahead prefix queries (@bo -> @bob, @bob2) hit the index/cache, never the ledger DB. Sub-50ms.
  • One current owner, full history retained. Each handle maps to exactly one active account, but we keep the history of who held it and when. When a username or phone is released and reclaimed, the old mapping is retired (marked inactive with an end timestamp), and the new owner gets a fresh mapping. Resolution only ever returns the active mapping, and only after it has been re-verified for phone/email (a reclaimed phone must re-verify before it can receive money, closing the recycled-number hole).
  • Resolution returns a person, not just an id - and the client confirms. The API returns {account_id, display_name, username, avatar, mutual_friends} so the app shows a confirmation card (“Send $20 to Bob Smith @bobsmith?”) before committing. For phone/email that matches multiple or no Venmo account, the flow degrades gracefully (invite, or disambiguation list). The human confirmation is a deliberate correctness layer on top of the technical resolution.
  • Handle changes are eventually consistent and safe. Changing your username updates the index; in-flight sends already resolved to your account id are unaffected because money moves on account_id, never on the handle string. The handle is only ever an input to resolution; the ledger only ever sees ids.
GET /identity/resolve?type=username&q=bob
-> [{account_id, display_name:"Bob Smith", username:"bobsmith",
     avatar, mutual_friends: 12, verified: true}, ...]  # ranked, for a picker

Why this is right: resolution is decoupled from the money DB (protects the ledger, enables sub-50ms typeahead), normalization handles the messy real-world formats, active-only mappings with re-verification close the reassignment/recycled-number holes, and a mandatory client confirmation makes the irreversible “wrong Bob” mistake a deliberate two-step rather than a silent one-tap.

2. Strongly consistent balances under concurrent transfers

The job: move money so the ledger is always internally consistent (every debit has an equal credit, balances never go negative from a race, any balance is provable from history), while serving high balance-read volume.

Naive approach: a balance column you read, modify, and write

Store one balance per account, mutate it directly on each send.

def send(sender, receiver, amount):
    s = get_balance(sender)          # read
    if s < amount: raise Insufficient
    set_balance(sender, s - amount)  # modify-write
    set_balance(receiver, get_balance(receiver) + amount)

Where it breaks:

  • Lost updates / negative balance under concurrency. Two sends from the same account both read balance = 100, both check >= 80, both write 20 - the account paid out 160 it did not have. On Venmo this is a Friday-night group-split reality: a user fires several sends at once. A check-then-write with no atomic guard prints money.
  • No audit trail. A bare number cannot answer “where did this $50 go” for a dispute or a chargeback. There is nothing to reconstruct.
  • Partial failure. Debit saves, process dies before credit - money vanished. Two independent writes are not atomic.
  • No conservation check. A bug crediting without debiting silently invents money; nobody notices until the books do not add up.

A mutable balance with no ledger is race-prone, unauditable, and non-atomic.

Evolved approach: append-only double-entry ledger + balance as a guarded projection

Use double-entry accounting: every transaction is a set of entries whose debits and credits sum to zero. Money is only moved, never created or destroyed.

  • Entries are immutable and append-only. A send of $20 from Alice to Bob is debit Alice 20, credit Bob 20 (their sum is zero). Mistakes are fixed by a reversing transaction, never by editing history.
  • Balance is a projection. The authoritative balance is SUM(credits) - SUM(debits) for the account. Because summing on every read is absurd at scale, keep a materialized balance column updated in the same ACID transaction as the entries, plus periodic snapshot rows so rebuilds do not scan from the beginning of time.

The whole move is one database transaction, and the overdraft guard and the race guard are the same conditional update:

BEGIN;
  -- guard overdraft AND serialize concurrent sends in one shot:
  UPDATE accounts
     SET balance = balance - :amount, version = version + 1
   WHERE account_id = :sender
     AND balance >= :amount;          -- 0 rows affected => insufficient funds
  -- if 0 rows -> ROLLBACK, return "insufficient funds"

  UPDATE accounts
     SET balance = balance + :amount, version = version + 1
   WHERE account_id = :receiver;

  INSERT INTO ledger_entries (tx_id, account_id, direction, amount) VALUES
    (:tx, :sender,   'DEBIT',  :amount),
    (:tx, :receiver, 'CREDIT', :amount);   -- must sum to zero

  INSERT INTO transactions (tx_id, type, state) VALUES (:tx, 'P2P', 'POSTED');
COMMIT;

Why this is right:

  • The conditional UPDATE ... WHERE balance >= amount is the concurrency guard. The row lock serializes concurrent sends from the same account; the second one re-checks the (now-lower) balance and either passes or rolls back. Balance can never go negative and there is no lost update - no separate lock service needed.
  • Atomicity is free - debit, credit, entries, and tx row commit together. Partial states are impossible.
  • Auditability is inherent. Every dollar’s path is a chain of immutable entries; a dispute is a query.
  • Conservation is checkable. A background job asserts global SUM(entries) == 0 and per-account materialized_balance == SUM(entries). A violation means a bug and we know immediately.
  • Reads scale separately. Balance reads hit the Redis-cached materialized balance (invalidated by version on write); history reads hit the async feed/history projection. The hot ledger shards see only writes and cache misses.

When one side is external (funding from a bank, cash-out to a bank), the counter-account is a system account (bank_funding_clearing, cashout_clearing). Double-entry still holds, and those system accounts are exactly what reconciliation compares against the processor. Money is stored as an integer count of minor units (cents), never a float - floats lose pennies and pennies are the whole game.

Idempotency wraps this. Each send carries a client Idempotency-Key; the ledger reserves it with a unique constraint, stores the result in the same transaction, and replays the stored response on any retry - so a timeout-and-retry or a double-tap moves money exactly once. The unique constraint, not a cache, is the real guarantee.

3. The privacy-aware social feed and its fan-out

The job: turn each payment into a feed entry that appears in the right people’s feeds, respecting a per-payment privacy setting, at ~100K feed writes/sec peak and ~15K feed reads/sec, without ever leaking a payment to someone who should not see it.

Naive approach: on feed load, query recent payments of everyone you follow

Build the feed at read time: find the viewer’s friends, query their recent payments, filter by privacy, merge, sort, return.

def feed(viewer):
    friends = get_friends(viewer)                      # ~50
    pays = db.query("SELECT * FROM payments WHERE actor IN (?) "
                    "ORDER BY created_at DESC LIMIT 30", friends)
    return [p for p in pays if visible_to(p, viewer)]  # privacy filter at read

Where it breaks:

  • Read-time fan-out is expensive and hits the money DB. Every feed open (15K/sec peak) runs a big IN-list query over the payments of dozens of friends and sorts - against the transactional store the ledger needs. Feed load is now competing with money movement for the primary.
  • Privacy filtering after the query still touched the data. You have already read private payments into the app tier to filter them out; a filter bug leaks them. Privacy applied only at read time is fragile.
  • No handling of high-fan-out accounts. A popular or business-ish account whose every payment should reach thousands of viewers makes the read-time query explode for each of those viewers.
  • Repeated work. The same friend’s payment is filtered and sorted again for every viewer, every open. Nothing is precomputed.

Read-time fan-out over the transactional store does not scale, and privacy-at-read-only is a leak waiting to happen.

Evolved approach: hybrid fan-out with privacy resolved at write time, re-checked at read

Precompute feeds by fanning out on write, store them in a dedicated feed store, and apply privacy when the entry is created - with a cheap re-check on read for changes.

  • Fan-out on write (push). When payment.posted arrives, the Feed Service computes the audience from the privacy setting and writes a lightweight feed entry into each recipient’s feed list in a wide-column store keyed by viewer_id. A feed entry is {payment_id, actor, target, note, emoji, created_at, privacy_snapshot} - never the amount (Venmo never shows amounts socially). Feed reads become a single partition scan by viewer_id, capped at a few hundred entries, nowhere near the ledger.
  • Privacy resolved when the entry is written. The audience is derived from the setting at fan-out time:
Privacy setting Audience the entry is written to
Public actor’s and target’s followers/friends feeds, plus a global/public timeline shard
Friends-only intersection-safe: friends of actor and friends of target only
Private only the actor’s and target’s own feeds

Because a private payment is never fanned out to anyone else, there is no private data sitting in other users’ feeds to leak. Privacy is enforced by not writing, which is far safer than writing then filtering.

  • Hybrid: pull for high-fan-out accounts. Pushing to millions on every payment is wasteful and slow (the celebrity/business problem). For accounts above a fan-out threshold, do not push; instead their followers pull those payments at read time and merge them into the pushed feed. This bounds write amplification: normal users get cheap reads (pure push), high-fan-out accounts get cheap writes (pull), and a viewer’s feed is merge(pushed_entries, pulled_entries_from_followed_celebs).
  • Re-check privacy and mutations at read. Settings and relationships change. On read, the feed entry carries a privacy_snapshot; the read path does a cheap final check (is the viewer still friends with the actor? has the payment been made private retroactively? has it been deleted/disputed?) against a fast relationship/override cache before rendering. A payment flipped to private after the fact is suppressed at read even though the stale entry exists; a nightly job also purges now-invisible entries. Write-time enforcement is the primary defense; read-time re-check is the backstop for changes.
  • Cap and rebuild. Feed lists are capped per user (e.g. latest ~500) and TTL’d. The feed is a disposable projection - it can always be rebuilt from the durable payment.posted event log, so we never treat it as a source of truth.
# fan-out on write
def on_payment(evt):
    audience = audience_for(evt.privacy, evt.actor, evt.target)   # set of viewer_ids
    if is_high_fanout(evt.actor):                                 # pull path
        record_for_pull(evt.actor, evt)                           # followers pull later
    else:
        for viewer in audience:                                   # push path
            feed_store.prepend(viewer, feed_entry(evt))           # capped list, no amount

Why this is right: fan-out on write turns feed load into a single cheap partition read that never touches the ledger; resolving privacy by audience selection at write time means private payments are simply never present in others’ feeds (a structural guarantee, not a filter); the hybrid push/pull bounds fan-out cost for high-follower accounts; and the read-time re-check plus rebuild-from-log handles the fact that privacy and relationships change over time.

4. Real-time fraud detection

The job: inspect every money movement for fraud - account takeover draining a balance, stolen-card funding that will be charged back, scam collection - and block or challenge the bad ones, all inside the ~500ms send budget, without blocking the good 99.9%.

Naive approach: a nightly batch job over yesterday’s transactions

Run fraud analysis offline: each night, score yesterday’s payments and flag suspicious ones for review.

# cron, 2am
for tx in yesterdays_transactions():
    if risky(tx): flag_for_review(tx)

Where it breaks:

  • The money is already gone. By the time the batch runs, an attacker who took over an account has already drained the balance and cashed out. Detection after settlement is a report, not a defense.
  • No inline decision. The naive send path never asks “should this happen?” - it just moves money and hopes the batch catches it later. There is no challenge, no hold, no block at the moment that matters.
  • Static rules rot. A hand-written if amount > X ruleset is trivially probed and evaded, and it floods review with false positives on legitimate large sends.
  • No feedback loop. Nothing feeds confirmed fraud back into the model, so the same pattern works repeatedly.

Batch-only fraud detection is a post-mortem, not a control; the money left before you looked.

Evolved approach: a synchronous risk gate backed by a real-time feature store, plus async deep analysis

Split fraud into a fast inline gate that decides in the send path and an async model + human review that learns and catches slower patterns.

  • Inline risk gate in the write path. Before the ledger moves money, the Fraud Gate returns approve / step-up / hold / block within a tight latency budget (single-digit to low-tens of ms). It scores from precomputed features it can read fast, not heavy computation:

    • Velocity features from a real-time counter store (Redis): sends in the last minute/hour/day, unique recipients, amount summed over windows. Sudden bursts are the signature of a drained account.
    • Device and session signals: new device, new geo, changed password recently, session age. Account takeover almost always shows a device/geo anomaly.
    • Relationship graph: have these two transacted before? Are they friends? A first-ever large send to a brand-new recipient is riskier than the weekly rent to a roommate.
    • Instrument risk for funding: a top-up from a freshly linked card, or a card with prior chargebacks, is high-risk because a stolen-card funding will be clawed back after the money is cashed out.
    • A model score: a gradient-boosted / neural model served behind a low-latency inference endpoint, taking the above features and returning a probability. The precomputed feature store is what makes sub-tens-of-ms scoring possible.

  • Graduated response, not just block.

Decision When Action
Approve low score proceed to ledger immediately
Step-up medium score / new device challenge (biometric, OTP, PIN) then proceed if passed
Hold high score, ambiguous post the transfer but freeze the funds (receiver cannot cash out) pending review
Block very high / known-bad reject the send, alert, lock the account for takeover signals

A hold is important: for stolen-instrument funding, we let the P2P entry post but keep the money unwithdrawable until the funding settles/clears, so a scammer cannot cash out before a chargeback lands. This is where fraud and the settlement state machine intersect - cash-out is gated on funding being cleared, not merely posted.

  • Async deep analysis and retroactive action. The Fraud Trainer consumes every payment.posted event to:

    • Update the real-time feature store (velocity counters, graph edges) so the next inline decision is smarter.
    • Run heavier graph/anomaly models offline (money-mule rings, circular flows, coordinated new accounts) that cannot fit the inline budget.
    • Retroactively freeze or reverse funds when async analysis or a chargeback confirms fraud - possible because the ledger is append-only and reversals are just new entries.
    • Feed confirmed fraud labels (from chargebacks, user reports, manual review) back into training, closing the loop the batch approach lacked.

  • Idempotent and fail-safe. The gate is called before the ledger tx and is idempotent on the same idempotency key (a retry re-uses the prior decision, so a challenge is not re-issued). If the gate is down, policy decides fail-open (approve, log) for small amounts to protect availability, fail-closed (challenge/hold) above a threshold - a deliberate availability-vs-loss trade documented up front.

def risk_gate(send, key):
    if cached := decision_cache.get(key): return cached   # idempotent
    f = feature_store.features(send.sender, send.receiver, send.amount, send.device)
    score = model.infer(f)
    d = policy(score, f)          # approve | step_up | hold | block
    decision_cache.put(key, d)
    return d

Why this is right: the inline gate stops fraud before the money moves, which the batch job structurally cannot; a precomputed feature store makes real-time scoring fit the latency budget; graduated responses (challenge/hold/block) avoid blocking legitimate users while stopping attackers; holding funds until funding clears defeats the stolen-card cash-out; and the async loop keeps the model learning from confirmed fraud and can reverse what slipped through - all riding on the same append-only ledger that makes reversals safe.

API Design & Data Schema

API

Mutating calls require an Idempotency-Key.

POST /api/v1/payments
Headers: Authorization: Bearer <token>
         Idempotency-Key: <client UUID, reused on retries>
Body:
{
  "to_handle": "@bobsmith",        // or "+15551234567" / "[email protected]"
  "to_account": "acc_bob",         // optional: pre-resolved id from the picker
  "amount": 2000,                  // integer minor units (cents) = $20.00
  "currency": "USD",
  "note": "pizza 🍕",
  "privacy": "friends"             // public | friends | private
}
Response 201:
{
  "transaction_id": "tx_9f3...",
  "state": "POSTED",
  "from_balance_after": 4550,
  "created_at": "2026-07-27T20:04:00Z"
}
Errors:
  400 invalid payload        401 bad token
  402 insufficient funds     404 handle not resolvable
  409 idempotency in progress (retry shortly)
  422 risk declined          423 step-up required (challenge token returned)

POST /api/v1/requests        {from_handle, amount, note} -> money request; approve/decline
POST /api/v1/requests/{id}/approve   (creates a payment)
POST /api/v1/topups          {source: bank/card ref, amount, Idempotency-Key} -> 202 PENDING
POST /api/v1/cashouts        {to_bank_account, amount, Idempotency-Key}       -> 202 PENDING

GET  /api/v1/identity/resolve?type=username&q=bo   -> ranked picker candidates
GET  /api/v1/accounts/{id}/balance                 -> {balance, currency, as_of}
GET  /api/v1/feed?cursor=...                        -> personalized feed page (no amounts)
GET  /api/v1/users/{id}/transactions               -> own history (with amounts)
PATCH /api/v1/payments/{id}/privacy  {privacy}      -> retroactive privacy change
POST /api/v1/webhooks/{processor}                   -> funding/cash-out status + chargebacks

Top-ups and cash-outs return 202 (external settlement is async, final state via webhook + reconciliation); P2P sends between two Venmo balances return 201 (fully committed in one ledger transaction). 423 carries a challenge token for the step-up flow.

Data stores

Different access patterns, different stores - stated explicitly.

1. Ledger + accounts + transactions - SQL, ACID, sharded by account_id, synchronously replicated. The one place the design depends on strong consistency, row locks, and multi-row atomic transactions. A relational store (PostgreSQL, or distributed SQL like CockroachDB/Spanner) is the correct, non-negotiable choice - the double-entry write, the balance >= amount guard, and same-transaction idempotency + outbox all need ACID. NoSQL’s eventual consistency would reintroduce the exact races we designed out.

Table: accounts
  account_id   BIGINT   PRIMARY KEY
  user_id      BIGINT   INDEX
  currency     CHAR(3)
  balance      BIGINT   -- minor units; materialized projection
  version      BIGINT   -- optimistic concurrency
  status       STRING   -- active|frozen|closed
  Shard by: account_id

Table: ledger_entries                 -- APPEND ONLY, immutable
  entry_id     UUID     PRIMARY KEY
  tx_id        UUID     INDEX
  account_id   BIGINT   INDEX
  direction    CHAR(1)  -- D | C
  amount       BIGINT   -- minor units, always positive; direction gives sign
  created_at   TIMESTAMP
  Shard by: account_id
  Invariant: per tx_id, SUM(credits) - SUM(debits) = 0

Table: transactions                   -- one row per logical operation (state machine)
  tx_id        UUID     PRIMARY KEY
  type         STRING   -- p2p|topup|cashout|request|reversal|adjustment
  state        STRING   -- POSTED|PENDING_RAIL|SETTLED|FAILED|HELD|REVERSED
  privacy      STRING   -- public|friends|private
  note         STRING
  risk_decision STRING  -- approve|stepup|hold|block
  provider_ref STRING   NULL
  amount       BIGINT
  created_at   TIMESTAMP
  Shard by: tx_id

Table: idempotency_keys
  idem_key     STRING   PRIMARY KEY   -- UNIQUE; the real dedup guarantee
  tx_id        UUID     NULL
  status       STRING   -- IN_PROGRESS | DONE
  response     JSON
  TTL: 48h hot

Table: outbox
  outbox_id    BIGINT   PRIMARY KEY
  tx_id        UUID
  topic        STRING
  payload      JSON
  sent         BOOL     INDEX (partial WHERE sent=false)

2. Identity index - key-value / search, cached. (handle_type, canonical_handle) -> account_id, active-only, plus a retired-mappings history table. Prefix-searchable for typeahead. Read-mostly, ~14 GB, sits behind Redis. Never queried against the ledger DB.

Table: identity_map
  handle_type       STRING  -- username|phone|email
  canonical_handle  STRING  -- E.164 / lowercased / case-folded
  account_id        BIGINT
  active            BOOL
  verified_at       TIMESTAMP
  PRIMARY KEY (handle_type, canonical_handle, active)

3. Balance cache - Redis. account_id -> {balance, version}, serves the balance read flood, invalidated by version on each ledger commit. A cache, never truth.

4. Feed store - NoSQL wide-column (Cassandra / DynamoDB), by viewer. Precomputed feed entries, capped and TTL’d per user, rebuildable from the event log. Never carries amounts.

Table: user_feed
  viewer_id    BIGINT   PARTITION KEY
  created_at   TIMESTAMP CLUSTERING KEY (DESC)
  payment_id   UUID
  actor, target, note, emoji, privacy_snapshot
  Shard by: viewer_id     -- single-partition reverse-chronological read

5. Fraud feature store - Redis + a columnar store. Real-time velocity counters and graph edges in Redis for inline scoring; historical features in a columnar warehouse for offline training. Plus a fraud_cases table (SQL) for the review/breaks workflow.

Why SQL for the core, NoSQL for the projections: the ledger write needs multi-row ACID, row-level locking for the overdraft guard, and same-transaction idempotency+outbox - exactly what relational databases exist for, worth giving up some horizontal-scale convenience to get (we shard SQL by account_id). The feed, identity index, and balance cache are read-heavy, key-scoped, and tolerant of eventual consistency, so they go to wide-column/KV/cache stores that scale reads cheaply. Match the guarantee to the data: absolute for money, relaxed for the projections of it.

Bottlenecks & Scaling

Where it breaks first, in order, and the fix for each.

1. Feed fan-out write amplification (breaks first at social scale). ~100K+ feed writes/sec at peak, 250x the payment rate, dominated by high-fan-out accounts. Fix: hybrid push/pull fan-out - push for normal users (cheap reads), pull for high-follower accounts (cheap writes), merged at read. Cap feed lists per user and TTL them; the feed is a rebuildable projection, not truth. Fan-out workers scale horizontally off Kafka, partitioned by viewer_id.

2. Double-spend / negative balance under concurrent sends and retries. A user firing several sends at once, or a timeout-and-retry, moving money twice or overdrawing. Fix: the conditional UPDATE ... WHERE balance >= amount (row lock serializes concurrent sends, prevents negative) inside one ACID transaction, plus a client Idempotency-Key with a DB unique-constraint backstop and same-transaction result storage. The constraint, not a cache, is the guarantee.

3. Identity typeahead hammering the money DB. Resolution on every keystroke, thousands/sec, would steal capacity from the ledger. Fix: a separate identity index behind Redis, normalized handles, prefix-searchable. The ledger DB never sees a resolution query.

4. Balance read flood. Every app open checks balance; summing entries or hitting SQL each time crushes the primary. Fix: materialized balance + Redis cache, invalidated by version. Hot SQL shards see only writes and cache misses.

5. Hot account / write contention. A popular account (or a system clearing account) taking thousands of credits/sec serializes on one row. Fix: balance sharding for hot accounts - split the balance into N sub-rows, route each credit to a random shard, sum on read; double-entry still holds per shard. Batch/aggregate for system accounts.

6. Ledger write throughput and the ACID core. The strongly-consistent SQL core is the scaling ceiling; you cannot add read replicas for writes. Fix: shard the ledger by account_id so writes spread. A cross-shard transfer needs a distributed transaction or a Saga (reserve-then-credit with compensation); keep the ACID transaction tiny to minimize lock hold time. Reads never hit the primary (cache + feed projection).

7. The dual-write problem (DB commit vs Kafka publish). Committing money then failing to publish payment.posted leaves the feed, fraud model, and settlement blind. Fix: the outbox pattern - the event is inserted in the same ACID transaction as the ledger write; a relay publishes it at-least-once; consumers dedupe on tx_id.

8. Fraud gate latency and availability. A slow or down fraud gate either blows the 500ms budget or blocks all sends. Fix: score from a precomputed feature store (Redis) so inference is single-digit ms; cache the decision per idempotency key; a documented fail-open-small / fail-closed-large policy if the gate is unavailable, trading a little loss for availability on small amounts and safety on large ones.

9. Privacy leaks in the feed. A payment reaching someone the setting excludes. Fix: enforce privacy at fan-out (audience selection) so private payments are never written to others’ feeds - a structural guarantee, not a filter - plus a cheap read-time re-check against a relationship/override cache and a purge job for retroactive changes.

10. Storage growth (~7TB/year ledger, never deleted). Regulatory retention forbids purging. Fix: time-partition the ledger; recent partitions on fast storage, older on cheaper cold storage still queryable; balance snapshot rows cap rebuild scans; archive idempotency keys after 48h.

11. Single points of failure. No single stateful node loss may stop or lose money. Fix: SQL ledger with synchronous replication and automatic failover (no committed tx lost); Kafka, Redis, and the feed store replicated; stateless API/workers horizontally scaled behind load balancers; DLQ + outbox mean nothing in flight vanishes.

Shard keys, stated plainly: accounts, ledger_entries -> account_id (entries co-located for rebuild, writes spread across accounts). transactions, idempotency_keys -> tx_id / key. user_feed -> viewer_id (single-partition reverse-chronological reads). identity_map -> canonical_handle. Kafka payment.posted -> partition by account_id (per-account ordering + even spread); feed fan-out topic -> by viewer_id. Balance buckets for hot accounts -> (account_id, shard_n). Never shard the ledger by time - it concentrates all current writes on the newest partition.

Wrap-Up

The trade-offs that define this design:

  • One small strongly-consistent core, everything else async. The only strongly-consistent operation is one ACID ledger transaction with a balance >= amount guard. Identity resolution, the fraud gate, the feed, notifications, and external settlement are all fast lookups or async projections. We keep the expensive consistency small so it stays fast and shardable.
  • The ledger is truth; balance and feed are projections. Money is an append-only double-entry ledger; the balance is a cached projection and the feed is a second projection of the same payment events. We trade the simplicity of a mutable balance for auditability, provable conservation, and a feed we can always rebuild.
  • Identity resolves to ids, not handles; humans confirm. Sending by username/phone is a normalized, cached index decoupled from the money DB, with active-only mappings and re-verification to close reassignment holes - and a mandatory confirmation card so the irreversible “wrong Bob” is a deliberate two-step.
  • Privacy is enforced by not writing, not by filtering. The feed applies privacy at fan-out via audience selection, so private payments are structurally absent from others’ feeds, with a read-time re-check as the backstop for changes. Amounts never appear socially.
  • Fraud sits in the write path but stays fast. A synchronous risk gate scoring precomputed features decides approve/challenge/hold/block before money moves, cashing out is gated on funding clearing, and an async model plus append-only reversals catch and undo what slips through.

One-line summary: an idempotent Payments API resolves a friend by username/phone through a cached identity index, passes a fast feature-store-backed fraud gate, and moves money in one strongly-consistent ACID double-entry transaction under an overdraft guard - then emits a payment.posted event via an outbox to async workers that fan a privacy-scoped, amount-free entry into friends’ feeds with a hybrid push/pull, notify the receiver, and keep the fraud model learning - so money moves once and is always conserved, friends see only what they are allowed to, and fraud is stopped before the money leaves.