A car drives through a toll lane at 30 km/h, an overhead antenna reads its RFID tag, and by the time the nose of the car reaches the barrier the boom has lifted and a rupee amount has left a prepaid wallet. That is the whole product in one sentence, and the interviewer will let you feel good about it for about a minute before pointing at the parts that are actually hard. The barrier has to decide in under half a second while the wallet that funds the toll lives in a different bank’s ledger a network hop away. The same tag gets read three times because the car crawled under the antenna. The plaza in a valley loses connectivity for twenty minutes and cars keep coming. And the one thing that absolutely cannot go wrong - the number in the wallet - has to stay exactly right across 100 million debits a day even though two lanes, a retry, and a settlement job might all touch the same balance at once.

Toll collection is not a barcode scanner wired to a subtraction. It is a low-latency authorization at the edge sitting on top of a strongly-consistent financial ledger in the core, with a settlement network between them, and the entire design is about keeping those two worlds - “open the barrier now” and “debit the wallet exactly once, correctly” - from poisoning each other. Let me build it properly.

Functional Requirements (FR)

In scope:

  • Read the tag and identify the vehicle. An RFID reader at each lane reads the tag ID as the vehicle passes, resolves it to a linked prepaid wallet/account, and determines the vehicle class (car, LCV, truck axle count) which sets the toll amount.
  • Decide the barrier in real time. Within a few hundred milliseconds, validate the tag (exists, active, not blacklisted, sufficient balance) and signal the boom to lift or stay down.
  • Auto-debit the toll from the prepaid wallet. Deduct the plaza-and-class-specific fare from the tag’s linked wallet, exactly once per passage, and credit it to the toll operator’s account.
  • Route across issuer and acquirer. The tag’s wallet is held by an issuer bank; the plaza is onboarded by an acquirer bank. The debit is routed through a central switch (the NETC-style network) between them.
  • Handle duplicates and retries. The same physical passage must debit once even if the tag is read multiple times or a request is retried after a timeout.
  • Operate degraded. When a plaza loses connectivity to the core, it must still process vehicles safely and reconcile later.
  • Reconciliation and settlement. End-of-day, match every barrier-open event against every debit and every inter-bank settlement, detect mismatches, and move money between banks.
  • Notifications and statements. Push a debit notification to the user and expose transaction history.

Explicitly out of scope (say this out loud so you own the scope):

  • The wallet top-up / recharge flow. Loading money into the prepaid wallet (UPI, netbanking, auto-recharge) is a payments problem handled by the issuer bank. I consume “current balance” and emit debits; I do not design the funding rail.
  • Tag issuance and KYC. Manufacturing tags, linking them to vehicles and bank accounts, and the KYC behind that are a provisioning system. I consume a tag -> account mapping.
  • The physical lane hardware. RFID antenna tuning, ANPR camera calibration, boom-barrier motors, and the RS-based lane controller protocol are hardware engineering. I treat the lane controller as a client that emits “tag X read at lane Y” and consumes an “open/deny” decision.
  • Toll-price policy. Who sets the fare per class per plaza, exemptions (ambulances, army), and dynamic congestion pricing are a pricing/policy layer. I read a fare table.

The one decision that drives everything: the barrier decision and the money movement are two different operations with two different consistency and latency requirements, and they must be decoupled. The barrier is an availability-first, low-latency authorization; the debit is a consistency-first, exactly-once financial write. Conflating them - blocking the boom on a synchronous cross-bank ledger commit - is the mistake the whole design exists to avoid.

Non-Functional Requirements (NFR)

  • Scale: 50,000 toll plazas, ~100M transactions/day. That is ~1,160 tx/sec average and, with commute peaks concentrated on highway plazas, ~5,000-6,000 tx/sec at peak. ~100M active tags. Each plaza has 4-20 lanes.
  • Latency: the lane decision (tag read -> open/deny signal) must be p99 under ~300ms, hard budget ~500ms. A car at 30 km/h covers ~4 meters in 500ms; miss the budget and either the barrier hits the car or throughput collapses and traffic backs up onto the highway.
  • Availability: the lane must stay operational at ~99.99%+, including when the central switch or the issuer bank is slow or unreachable. A plaza that stops passing cars because a bank in another city is down is unacceptable. Degrade, do not stop.
  • Consistency: the wallet balance is strongly consistent and the debit is exactly-once. No double-debit for one passage, no debit lost, no balance going wrong under concurrency. This is the non-negotiable core; everything else bends around it.
  • Durability: every barrier-open event and every debit is a financial and legal record and must be durably persisted and never lost. Reconciliation and dispute resolution depend on it. Zero tolerance for dropped transaction logs.
  • Idempotency: every stage - lane, switch, issuer - must be idempotent on a stable passage key so retries and duplicate reads collapse to one debit.
  • Auditability: the full path of every rupee (who read the tag, who authorized, who debited, who got settled) must be reconstructable for disputes and regulatory audit.

Back-of-the-Envelope Estimation (BoE)

Real numbers with the arithmetic, because they set the architecture.

Transaction rate:

100M transactions/day
= 100,000,000 / 86,400 s  ~= 1,160 tx/sec average.
Peak factor ~4-5x (morning + evening commute on highway plazas)
Peak ~= 5,000-6,000 tx/sec.
Reads (validation lookups) >> writes: every passage does at least one
tag+balance validation read; some do speculative reads. Call it ~2x the
debit rate on the read path -> ~10,000-12,000 validation reads/sec peak.

So this is a modest write rate (~6k debits/sec peak) but a strict-latency, strict-consistency write rate. 6k/sec of ordinary writes is nothing; 6k/sec of exactly-once cross-bank ledger debits under a 300ms budget is the whole problem.

Tag / mapper data:

100M active tags.
Mapper entry (tag_id -> issuer_id, account_ref, status, class) ~= 60 bytes.
Full mapper ~= 100M * 60B ~= 6 GB.  Fits in a Redis cluster in memory.
Blacklist (lost/stolen/low-balance-flag tags), say 2M entries * 40B ~= 80 MB.

The tag -> issuer + status mapper is small enough to keep hot in memory and replicate to the edge. That is the load-bearing fact for the fast path.

Storage / year:

Transaction record ~= 500 bytes (tag, plaza, lane, class, amount,
  timestamps, txn_id, status, issuer/acquirer refs).
100M/day * 500B = 50 GB/day  -> ~18 TB/year of raw transactions.
Double-entry ledger: 2 postings per debit, ~200B each = ~80 GB/day
  -> ~29 TB/year on the issuer ledger side.
Settlement/recon records: comparable order. Call the whole estate
  ~80-100 TB/year across transactions + ledger + recon, before replication.

Storage is large but boring - it is append-only, partitionable by time, and cold after settlement. No cleverness needed beyond partitioning and archival.

Bandwidth:

Each debit round trip ~= 2 KB request + 1 KB response across the switch.
Peak ~6,000 tx/sec * ~3 KB = ~18 MB/sec across the central switch.
Trivial. This system is latency- and consistency-bound, not bandwidth-bound.

Cache / edge footprint per plaza:

A plaza needs, locally: the fare table (class -> amount, a few KB),
a rolling recent-passages set for de-dup (last ~10 min, a few thousand
entries, ~1 MB), and optionally a replicated slice of the mapper/blacklist
for the tags it commonly sees. Edge footprint is megabytes, not gigabytes.

The takeaways: a small hot mapper (6GB) that can live in memory and at the edge, a modest but latency-and-consistency-critical write rate (~6k debits/sec), boring but large append-only storage, and negligible bandwidth. The design driver is not throughput or storage - it is doing an exactly-once, strongly-consistent cross-bank debit under a 300ms barrier budget while surviving connectivity loss.

High-Level Design (HLD)

Three planes. The lane/edge plane at each plaza reads tags and makes a fast barrier decision. The switch plane (the NETC-style central network) routes each transaction from the acquirer side to the correct issuer. The ledger/settlement plane at the issuer bank holds the strongly-consistent wallet and does the exactly-once debit, and a settlement service moves money between banks end-of-day. The barrier decision is decoupled from the ledger commit: the lane authorizes fast and the debit settles through the switch, with reconciliation catching anything that diverges.

   Vehicle + RFID Tag
          │ tag read (tag_id, lane, ts)
   ┌──────▼─────────────────────────────┐
   │  LANE CONTROLLER (per lane)         │
   │  - reads tag, ANPR plate backup     │
   │  - de-dup vs recent passages        │
   │  - builds passage_id (idempotency)  │
   └──────┬──────────────────────────────┘
          │
   ┌──────▼─────────────────────────────┐        ┌──────────────────────┐
   │  PLAZA EDGE SERVER (per plaza)      │        │ Local cache:          │
   │  - fare table lookup (class->amt)   │◀──────▶│  fare table           │
   │  - fast validate (mapper/blacklist) │        │  mapper slice+blacklist│
   │  - decide OPEN / DENY  (<300ms)     │        │  recent-passage dedup │
   │  - enqueue debit (durable outbox)   │        └──────────────────────┘
   └──────┬───────────────────┬──────────┘
   OPEN/  │ barrier            │ debit request (async, at-least-once)
   DENY   ▼                    │
      [ BOOM BARRIER ]         │
                               ▼
             ┌─────────────────────────────────────────┐
             │  ACQUIRER BANK GATEWAY                    │
             │  (bank that onboarded this plaza)         │
             │  - authenticates plaza, forwards to switch│
             └───────────────────┬─────────────────────┘
                                 │
             ┌───────────────────▼─────────────────────┐
             │  CENTRAL SWITCH (NETC-style)              │
             │  - MAPPER: tag_id -> issuer bank          │
             │  - routes debit to correct issuer         │
             │  - idempotent on passage_id (txn_id)      │
             │  - logs every txn (durable)               │
             └───────────────────┬─────────────────────┘
                                 │
             ┌───────────────────▼─────────────────────┐
             │  ISSUER BANK - WALLET LEDGER              │
             │  - strongly consistent balance           │
             │  - atomic conditional debit (exactly-once)│
             │  - double-entry posting                   │
             │  - emits debit notification               │
             └───────────────────┬─────────────────────┘
                                 │ response (debited / insufficient / dup)
             ┌───────────────────▼─────────────────────┐
             │  SETTLEMENT & RECONCILIATION (batch/EOD)  │
             │  - match passages vs debits vs postings   │
             │  - net inter-bank positions, move money   │
             │  - flag exceptions -> dispute queue       │
             └───────────────────────────────────────────┘

Fast path (barrier decision), target < 300ms:

  1. The lane controller reads the tag, captures a backup ANPR plate image, and forms a stable passage_id from (tag_id, plaza_id, lane_id, coarse_timestamp) for idempotency and de-dup.
  2. The plaza edge server checks its recent-passages set to drop a duplicate read of the same car, looks up the vehicle class -> fare, and runs a fast validation: tag exists, active, not blacklisted, and (if a balance hint is available locally or via a quick issuer check) has sufficient balance.
  3. The edge server returns OPEN or DENY to the barrier immediately, and writes the debit intent to a durable local outbox. The barrier is not blocked on the cross-bank debit.

Consistency path (the debit), settles asynchronously, exactly-once:

  1. The edge outbox forwards the debit request (carrying passage_id as the idempotency key) to the acquirer bank gateway, which authenticates the plaza and forwards to the central switch.
  2. The switch’s mapper resolves tag_id -> issuer bank, routes the debit there, and logs it. The switch is idempotent on passage_id - a retried request returns the original result, not a second debit.
  3. The issuer bank performs the atomic conditional debit on the strongly-consistent wallet ledger (double-entry), returns debited / insufficient / duplicate, and emits a user notification. This is the only place the balance is authoritatively mutated.
  4. Settlement/reconciliation runs continuously and at end-of-day, matching every OPEN event to a debit and every debit to inter-bank money movement, netting positions between acquirer and issuer banks, and pushing mismatches to a dispute queue.

The key insight: the boom lifts on a fast, availability-first authorization; the wallet moves on a slow, consistency-first, exactly-once debit; and a reconciliation loop guarantees the two eventually agree. The rest of the design is making each of those three correct.

Component Deep Dive

The hard parts, naive-first then evolved: (1) the exactly-once, strongly-consistent wallet debit, (2) idempotency and duplicate tag reads, (3) the barrier decision under a bank round trip, and (4) degraded/offline plaza operation.

1. The exactly-once, strongly-consistent wallet debit

This is the “hard part” the problem foregrounds. Everything else is plumbing around getting this one write right.

Naive approach: read the balance, subtract, write it back.

bal = SELECT balance FROM wallets WHERE tag = ?
if bal >= fare:
    UPDATE wallets SET balance = bal - fare WHERE tag = ?
    return DEBITED
else:
    return INSUFFICIENT

Where it breaks:

  • Lost updates under concurrency. A tag can be read in two lanes seconds apart (multi-lane plaza, or two plazas on a short stretch), or the same debit can be retried after a timeout while the first is still in flight. Two read-modify-write cycles interleave: both read balance 300, both subtract 100, both write 200. One debit vanished. The user paid once for two passages, or the balance is simply wrong. At 6k debits/sec this is not rare; it is constant.
  • No exactly-once. A network timeout on the response leaves the caller not knowing if the debit happened. Retrying blindly double-debits. Not retrying risks a lost debit. Read-modify-write has no way to make the retry safe.
  • Torn state on crash. If the process dies between the read and the write, the outcome is undefined and unrecoverable from the balance alone - there is no record of intent.

First evolution: make the write atomic and conditional. Replace read-modify-write with a single conditional update that both checks and decrements in one statement, so concurrent debits serialize on the row:

UPDATE wallets
SET balance = balance - :fare, version = version + 1
WHERE tag = :tag AND balance >= :fare
-- affected rows = 1 -> debited; 0 -> insufficient (or row locked, retried)

This kills the lost-update race: the database takes a row lock, the decrement is atomic, and balance >= fare is evaluated against the committed value, so two concurrent debits can never both succeed on a balance that only covers one. Strong consistency on the balance is now enforced by the single-row atomic update. But it still does not give exactly-once - a retry of a debit that already succeeded would decrement again, because the conditional update has no memory of “I already applied this passage.”

The answer: an append-only double-entry ledger keyed by an idempotent transaction id, with the balance as a derived, atomically-maintained projection.

  • Idempotency key = passage_id. Every debit carries the stable passage_id built at the lane. Before applying, the issuer checks a processed-transactions table (unique index on passage_id). If the id is already present, it returns the stored original result without touching the balance. First writer wins; every retry is a no-op that replays the same answer.
  • Double-entry postings, append-only. The debit is not an in-place edit of a number; it is two immutable postings - debit the user wallet, credit the toll operator’s receivable - written in one transaction. The ledger is the source of truth; the running balance is a maintained projection of it.
  • One atomic transaction does all three: insert the passage_id row (fails on duplicate -> idempotent short-circuit), insert the two ledger postings, and apply the conditional balance update. Either all commit or none do. There is no window where the balance moved but the ledger did not, or the id was recorded but the money was not.
BEGIN
  INSERT INTO processed_txn(passage_id, ...) VALUES (...);   -- UNIQUE(passage_id)
      -- duplicate key -> ROLLBACK, return stored result (exactly-once)
  UPDATE wallets SET balance = balance - :fare, version = version + 1
      WHERE tag = :tag AND balance >= :fare;                 -- 0 rows -> INSUFFICIENT, ROLLBACK
  INSERT INTO ledger(txn_id, tag_wallet, -:fare), (txn_id, operator_acct, +:fare);
COMMIT
  • Strong consistency, scoped to one account. All of a tag’s debits touch one wallet row, so they serialize on that row - no distributed transaction needed for the balance itself. The wallet is sharded by account (see Bottlenecks), so different users’ debits are fully parallel; only same-account contention serializes, which is exactly what correctness requires and is rare in practice (a car is in one lane at a time).
  • Insufficient balance and the barrier. The debit can come back INSUFFICIENT after the barrier already opened (because the fast path opened optimistically or on a stale balance hint). That is handled by policy: mark the passage as a negative-balance / to-recover debit, allow it through (barriers do not trap cars over a few rupees), block the tag on further passages until topped up, and recover on next recharge. The ledger records the shortfall as a receivable; it is a business decision, not a lost debit.

The mental model: the balance is never mutated directly - it is the projection of an append-only, idempotent ledger, and one atomic transaction guarantees debit-once-or-not-at-all. Strong consistency comes from serializing on a single account row; exactly-once comes from the unique passage_id; durability comes from the ledger being the immutable source of truth.

2. Idempotency and duplicate tag reads

Naive approach: debit on every read the antenna produces. The reader fires a debit each time it sees a tag.

Where it breaks:

  • Physical multi-read. RFID antennas read a tag many times per second while it is in range. A car crawling in traffic sits under the antenna for seconds, producing dozens of reads. Debit-per-read charges the user dozens of times for one passage.
  • Cross-layer retries. The edge retries the debit to the switch on timeout; the switch retries to the issuer. Without a shared key, each retry is a fresh debit. One passage, three debits.
  • Tag read at adjacent lanes. Wide antennas can pick up a tag in the neighbouring lane; two lanes each fire a debit for the same car.

The fix: one stable passage_id per physical passage, enforced idempotent at every layer.

  • Coalesce raw reads into one passage at the lane. The lane controller debounces: the first read of a tag starts a passage; subsequent reads of the same tag within a short window (say 30-60s) and same lane are folded into it. Only one debit intent leaves the lane per passage.
  • Deterministic passage_id. passage_id = hash(tag_id, plaza_id, lane_id, floor(timestamp / window)). It is derived, not random, so a retry regenerates the same id rather than a new one - the retry is automatically de-duplicated without the edge having to remember it. The coarse time bucket makes reads within the same passage collide to one id while a genuine second passage an hour later gets a new id.
  • Idempotency enforced at every hop. The edge outbox, the switch, and the issuer all key on passage_id. The switch keeps a processed-set (with TTL for the online window) and returns the original response on a repeat; the issuer enforces it durably via the unique index in section 1. Idempotency at the issuer is the authoritative one; the earlier layers are optimizations that stop needless traffic.
  • Adjacent-lane de-dup. Because the plaza edge server sees all its lanes, it can drop a second lane’s read of a tag already claimed by an active passage in a neighbouring lane within the same second, and the ANPR plate acts as a tiebreaker when tag reads are ambiguous.

The principle: a physical passage is the unit of billing, and it gets exactly one identity that every layer agrees on. De-dup at the lane for efficiency, but make correctness depend only on the issuer’s durable idempotency on passage_id, so no lost or duplicated read anywhere in the chain can produce a double-debit.

3. The barrier decision under a bank round trip

Naive approach: on tag read, call the issuer synchronously, wait for the debit result, then open the barrier.

Where it breaks:

  • Latency budget blown. The debit crosses acquirer -> switch -> issuer -> back. Even at 50ms per hop plus a ledger commit, that is easily 200-400ms of network on a good day and seconds on a bad one. Put that on the barrier’s critical path and either the boom is too slow (traffic backs up) or, worse, opens after the car has stopped and started reversing.
  • Availability coupling. If the issuer bank or the switch is slow or down, every lane at every plaza stalls. A payments outage in one bank should not close 50,000 toll plazas. Blocking the barrier on the debit makes the least-available component gate the most-availability-critical action.
  • Tail latency dominates throughput. At 6k/sec, the p99.9 debit latency, not the median, sets how fast lanes clear. Synchronous debit ties lane throughput to the worst-case ledger response.

The fix: authorize fast at the edge, debit asynchronously, and reconcile.

  • Decouple decision from debit. The barrier decision uses only fast, local-ish signals: is the tag well-formed, active, and not blacklisted, and (best-effort) is the balance plausibly sufficient. That check hits the edge’s replicated mapper/blacklist slice and, optionally, a fast balance-hint lookup - all sub-10ms. The boom lifts on that. The actual debit is enqueued to a durable outbox and settles through the switch asynchronously, exactly-once.
  • Balance hint, not authoritative balance, on the fast path. The edge can cache a recent balance hint per frequently-seen tag (refreshed by the issuer), used only to catch the obvious “clearly empty wallet” case. The authoritative sufficiency check is the issuer’s conditional update. The fast path optimizes for “almost never wrongly deny a good tag”; the rare optimistic open on an actually-empty wallet becomes a recoverable receivable (section 1).
  • Durable outbox at the edge. The debit intent is written to local durable storage before the barrier opens, so a plaza crash after opening the boom never loses the debit. The outbox drains to the switch with at-least-once delivery and retries; idempotency on passage_id makes the retries safe.
  • Blacklist is the real gate. The one thing the fast path must get right is denying blacklisted tags (stolen, cloned, chronic non-payers). The blacklist is small (~80MB), replicated to every edge, and refreshed continuously, so the deny decision is local and instant and does not depend on any remote call.
tag read ->
  edge: valid(tag) && active(tag) && !blacklisted(tag)  [local, <10ms]
        && balance_hint(tag) >= fare (best-effort)
     -> OPEN barrier
     -> write debit intent to durable outbox (passage_id)
  outbox -> switch -> issuer: atomic exactly-once debit  [async, off critical path]
  reconciliation: every OPEN must eventually have a matching debit

The principle: the barrier is gated by a local, fail-operational authorization; the money moves off the critical path with a durable outbox and exactly-once settlement. Availability and latency live at the edge; consistency lives in the core; reconciliation guarantees they converge.

4. Degraded and offline plaza operation

Naive approach: if the plaza cannot reach the switch, stop processing vehicles.

Where it breaks:

  • Rural and mountain plazas lose connectivity routinely. A plaza in a valley with a flaky link cannot be allowed to jam the highway every time the uplink drops. “No network -> closed toll” turns a connectivity blip into a traffic disaster.
  • Central outage cascades. If the switch or a major issuer has an incident, thousands of plazas going hard-down simultaneously is a national-scale failure.

The fix: store-and-forward at the edge with a durable local queue and bounded risk.

  • Local decision authority. Each plaza edge server holds a replicated slice of the mapper, the blacklist, and the fare table, all refreshed when online. When the uplink is down it makes the barrier decision entirely locally: well-formed tag, active, not on the (last-known) blacklist -> OPEN. The blacklist being local is what makes offline mode safe - stolen tags are still denied.
  • Durable offline outbox. Every offline passage is written to the edge’s durable outbox with its passage_id, fare, and captured plate. When connectivity returns, the outbox drains to the switch, and each debit is applied exactly-once by the issuer. Idempotency means a passage that was partially forwarded before the link died is not double-charged on retry.
  • Bounded exposure. Offline mode accepts a bounded financial risk: a tag with insufficient balance that passes offline becomes a receivable recovered on next top-up, and a tag blacklisted after the edge’s last sync might slip through once. That risk is capped (small per-passage amount, short sync interval, per-tag offline pass limits) and is vastly cheaper than closing the plaza. This is an explicit availability-over-strict-consistency trade at the authorization layer, while the ledger stays strictly consistent - the debit still lands exactly-once when the queue drains.
  • Graceful central degradation. If the switch is up but a single issuer is down, that issuer’s debits queue and retry (with idempotency) while every other issuer settles normally; the outage is contained to one bank’s tags, not the whole network.

The principle: the ledger never relaxes consistency, but the edge is allowed to authorize offline against last-known state and settle later, exactly-once. Connectivity loss degrades to store-and-forward, not to a closed barrier.

API Design & Data Schema

The lane talks to the edge over a local protocol; the edge, switch, and issuer talk over authenticated internal APIs; the user gets a read API.

Lane / edge

POST /edge/v1/passage
  body: { tag_id, plaza_id, lane_id, vehicle_class, plate_img_ref, read_ts }
  -> 200 { decision:"OPEN", passage_id:"p_8f2..", fare:115,
           reason:"authorized" }
  -> 200 { decision:"DENY", passage_id:"p_8f2..", reason:"blacklisted" }
     # decision returns in <300ms; debit settles asynchronously

Edge -> switch -> issuer (the debit)

POST /switch/v1/debit
  headers: { Idempotency-Key: passage_id, acquirer_id, signature }
  body: { passage_id, tag_id, plaza_id, lane_id, amount, class, event_ts }
  -> 200 { status:"DEBITED", txn_id, balance_after:1885 }
  -> 200 { status:"INSUFFICIENT", txn_id, shortfall:40 }   # -> receivable
  -> 200 { status:"DUPLICATE", txn_id, original_status:"DEBITED" }  # idempotent replay
  -> 5xx -> caller retries with same Idempotency-Key (safe)

User-facing (read)

GET /api/v1/tags/{tag_id}/transactions?from=..&to=..
  -> 200 { balance:1885, txns:[ {txn_id, plaza:"NH48-Kherki", amount:115,
           ts:.., status:"DEBITED"} , ... ] }

Data stores - the right tool per plane

1. Wallet + ledger - a strongly-consistent relational store (SQL), sharded by account. This is the core. The balance must serialize on a single row and commit atomically with the ledger postings and the idempotency record. This is precisely what a relational engine with row-level locking and ACID transactions is built for; a key-value store with eventual consistency would be wrong here. Sharded by account_id so debits for different users are fully parallel.

wallets      (SQL, sharded by account_id)
  account_id      PK
  tag_id          UNIQUE, indexed
  balance         BIGINT (paise, integer money - never float)
  status          active | blocked | low_balance
  version         BIGINT      -- optimistic concurrency
  updated_at

ledger       (SQL, append-only, partitioned by month, sharded by account_id)
  posting_id      PK
  txn_id          indexed
  account_id      indexed
  amount          BIGINT (signed: - for wallet, + for operator)
  posted_at

processed_txn (SQL, same shard as wallet)
  passage_id      PK / UNIQUE   -- the exactly-once guarantee
  txn_id
  result          DEBITED | INSUFFICIENT
  created_at

Money is stored as an integer count of paise, never a float - floating point in a ledger is a correctness bug waiting to happen.

2. Mapper + blacklist - in-memory KV (Redis/edge cache), replicated to the edge. tag_id -> issuer, account_ref, status, class and the blacklist set. ~6GB, read-hot, latency-critical, tolerant of slight staleness (refreshed continuously). Wrong store choice would be a disk-backed DB on the fast path.

Redis / edge cache:
  HGETALL tag:{tag_id}  -> {issuer, account_ref, status, class}
  SISMEMBER blacklist {tag_id}
  fare:{plaza_id}:{class} -> amount

3. Transaction log - append-only store into a warehouse, partitioned by day/plaza. Every passage and every decision, durable and immutable, for reconciliation, disputes, and analytics. NoSQL/columnar is fine here - it is write-once, read-by-range, never updated in place.

transactions (append-only -> object store / warehouse)
  txn_id, passage_id, tag_id, plaza_id, lane_id, class, amount,
  decision, debit_status, acquirer_id, issuer_id, read_ts, settle_ts
  partitioned by (date, plaza_id)

4. Edge outbox - durable local queue at each plaza. Persists debit intents before the barrier opens and drains to the switch with at-least-once delivery. Local disk / embedded durable queue, so a plaza-local crash or uplink loss never loses a debit.

Why SQL for the wallet/ledger but in-memory KV for the mapper and append-only storage for logs: the planes have opposite needs. The wallet is a small, high-contention, must-be-exactly-right financial write - it wants ACID and single-row serialization, which is SQL’s home turf. The mapper is a huge-fan-out, read-hot, staleness-tolerant lookup - it wants in-memory KV at the edge. The transaction log is unbounded, write-once, read-by-range - it wants cheap append-only columnar storage. There is no single “toll database”; each plane gets the store that matches its consistency, latency, and lifetime.

Bottlenecks & Scaling

Where it breaks first, in order, and the fix for each.

1. Synchronous debit on the barrier path (breaks first). Blocking the boom on a cross-bank ledger commit blows the 300ms budget and couples lane availability to the least-available bank. Fix: decouple decision from debit - authorize locally at the edge and settle the debit asynchronously through a durable outbox with exactly-once semantics. The single most important separation in the design.

2. Wallet row as a serialization point / hot account. All of a tag’s debits serialize on one row. A fleet account (a logistics company with thousands of trucks) or a single tag hit rapidly can contend. Fix: shard wallets by account_id so different accounts are fully parallel - 6k debits/sec across 100M accounts is almost no per-row contention. For genuine hot accounts (large fleets), split the balance into sub-wallet buckets and debit a bucket chosen by hash(passage_id), summing buckets for the displayed balance - this spreads contention across N rows while keeping the total exactly right. Same-tag contention is naturally rare because a vehicle is in one lane at a time.

3. Exactly-once under retries and duplicate reads. Without a shared key, multi-reads and cross-layer retries double-debit. Fix: deterministic passage_id coalesced at the lane and enforced idempotent at the issuer via a unique index; every hop replays the original result on a repeat. Correctness depends only on the issuer’s durable idempotency; earlier layers just cut traffic.

4. Central switch as a single point of failure and throughput ceiling. Every debit routes through it; if it is down, no settlement happens anywhere. Fix: the switch is horizontally scaled and partitioned by issuer (route by tag -> issuer), multi-region active-active, and stateless except for an idempotency cache with TTL. Because debits are asynchronous and idempotent, a switch blip queues at the edge outbox and drains on recovery rather than failing passages. Its mapper is a replicated in-memory KV, not a bottleneck.

5. Issuer bank outage. One issuer down would stall all its tags’ debits. Fix: per-issuer queueing with retry contains the outage to that issuer’s tags; all other issuers settle normally. Idempotency makes the drain-on-recovery safe. The barrier still opens (fast path is local), so the outage is invisible at the lane.

6. Connectivity loss at the plaza. A flaky uplink cannot be allowed to close the lane. Fix: store-and-forward at the edge - local mapper/blacklist/fare replicas make the decision offline, a durable outbox holds debits, and everything settles exactly-once when the link returns. Bounded, capped financial risk in exchange for never closing the barrier.

7. Storage growth (18-30 TB/year per plane). Transaction and ledger volume is large. Fix: partition by time (and plaza for transactions), keep the hot window on fast storage, and archive settled partitions to cold object storage. Ledger stays append-only; nothing is updated in place, so partitioning is clean.

8. Reconciliation load and mismatches. Matching 100M passages against 100M debits against inter-bank settlements daily is a heavy join, and mismatches (open-but-not-debited, debited-but-no-open, double-settled) must be caught. Fix: run reconciliation as a partitioned batch/stream job keyed by passage_id, comparing the three sources; exceptions go to a dispute queue with automated resolution rules (e.g. open-without-debit -> re-drive the debit; duplicate -> refund the extra). The idempotency and append-only ledger make recon a deterministic set-difference rather than guesswork.

9. Blacklist freshness vs edge staleness. A tag blacklisted centrally must reach 50,000 edges fast, or a stolen tag rides free. Fix: push blacklist deltas continuously to edges (small, ~80MB full, deltas tiny); accept a bounded propagation window; and enforce the authoritative check again at the issuer, which denies the debit even if the edge opened. The edge is the fast filter; the issuer is the backstop.

10. Read-path fan-out for balance hints and mapper lookups. ~12k validation reads/sec. Fix: serve them from the replicated in-memory mapper/blacklist at the edge with a short-TTL local cache; these never touch the SQL wallet on the fast path. The wallet is read authoritatively only inside the debit transaction.

11. Multi-region and data residency. Banks and plazas span regions; financial data has residency rules. Fix: run the switch active-active per region, keep each issuer’s ledger in its home region, and route by issuer so the strongly-consistent write stays regional. The edge always talks to a nearby switch node; settlement replicates asynchronously to a central clearing view. Because the consistency boundary is a single account in a single issuer, no cross-region distributed transaction is ever needed.

Wrap-Up

The trade-offs that define this design:

  • Decoupled authorization and debit. The barrier lifts on a fast, local, availability-first authorization; the wallet moves on a slow, consistency-first, exactly-once debit off the critical path. This separation is what lets a 300ms boom decision sit on top of a cross-bank ledger without either poisoning the other.
  • Strong consistency scoped to a single account. The balance is a projection of an append-only double-entry ledger, mutated only inside one atomic transaction that serializes on a single sharded account row and is guarded by a unique passage_id. Exactly-once and correct-under-concurrency come for free from that, with no distributed transaction, because a car is only ever in one lane.
  • Idempotency as the spine. One deterministic passage_id per physical passage, enforced idempotent at every hop and authoritatively at the issuer, is what makes duplicate reads, cross-layer retries, and offline drains all collapse to exactly one debit.
  • Availability over strict authorization at the edge, never at the ledger. The plaza is allowed to open the barrier offline against last-known blacklist/balance and settle later; the ledger never relaxes consistency. Bounded, recoverable risk at the edge buys a lane that never closes because a bank is slow.
  • Reconciliation as the safety net. A continuous set-difference over passages, debits, and settlements catches anything that diverges and routes it to automated dispute resolution, so the fast path and the ledger provably converge.

One-line summary: a toll system where each lane reads an RFID tag and makes a sub-300ms, fail-operational barrier decision against a replicated edge mapper/blacklist, then settles the toll asynchronously through a central issuer-routed switch as an exactly-once, strongly-consistent debit on an append-only double-entry ledger sharded by account - with a deterministic passage_id guaranteeing debit-once across duplicate reads, retries, and offline drains, and a reconciliation loop guaranteeing the barrier and the wallet always agree.