“Book a train ticket” sounds like inserting a row. Then you look at Tatkal. At 10:00:00 sharp, a few days’ worth of demand for a scarce quota lands in a single second. Two or three million people press “Book” against a train that has maybe eighty Tatkal seats. The seats sell out in ten to twenty seconds. Every one of those millions must get a truthful answer - a confirmed seat with a berth number, a waitlist position, or a clean rejection - and no seat may ever be sold twice, because a double-booked berth is two paying passengers standing in the same coach at midnight. This is not a CRUD service with a form in front. It is a scarce-inventory contention problem where the read load is enormous, the write load is fierce and concentrated, and correctness on the write path is absolute.

The core of this problem is that a tiny, fixed pool of seats must be allocated exactly once each, under a burst of millions of concurrent writers, while staying fair and fast. Oversell is illegal and physical. Underselling (leaving seats empty because you were too cautious with locks) wastes revenue and angers users. The 10am spike is 100x to 1000x the baseline load and it targets the hottest rows in the database - the exact opposite of a load you can spread out. Every decision in this design exists to hold those together: never oversell, don’t undersell, answer everyone quickly, and stay fair. Let me build it properly.

Functional Requirements (FR)

In scope:

  • Search trains. A user searches by source, destination, and date and gets trains with per-class availability (SL, 3A, 2A, 1A, CC) and quota (General, Tatkal, Ladies, Senior Citizen).
  • Book a ticket against a quota. A user selects a train, class, quota, and passenger list, and the system allocates seats/berths from that quota’s inventory or places the booking on the waitlist. This is the heart of the system.
  • Tatkal window. Tatkal inventory opens at a fixed instant (10:00 for AC classes, 11:00 for non-AC, one day before travel). Before that instant the quota is closed; at that instant it opens to everyone at once.
  • Generate a PNR. Every confirmed or waitlisted booking gets a unique 10-digit PNR (Passenger Name Record) that ties together the passengers, train, journey, and current status.
  • Waitlist and RAC. When confirmed berths run out, bookings go to RAC (Reservation Against Cancellation, a shared side-berth) and then to a plain waitlist (WL) with a position number.
  • Auto-upgrade and confirmation on cancellation. When a confirmed passenger cancels, the highest RAC moves to confirmed and the highest WL moves to RAC - automatically. Optionally, an auto-upgrade scheme promotes passengers to a higher empty class for free.
  • Payment and hold. A seat is held while the user pays; if payment fails or times out, the hold is released back to inventory.
  • Cancel and refund. A user can cancel; the seat returns to the pool and triggers the waitlist promotion cascade.
  • Chart preparation. A few hours before departure the chart is finalized: remaining vacancies are allocated, RAC/WL are settled, and the coach-berth layout is frozen.

Explicitly out of scope (name it so you own the scope):

  • Payment gateway internals. I integrate with UPI/card/netbanking gateways and treat them as an external async service with callbacks; I do not build the gateway.
  • Dynamic pricing and fare computation. Fare rules, Tatkal surcharge, and refund slabs are a pricing service I call; I store the computed fare.
  • Physical train operations. Actual coach composition changes, train cancellations by the railway, and running-status/GPS are separate systems; I consume coach layout as reference data.
  • Auth, KYC, and account management. Login, Aadhaar linking, and the virtual waiting queue’s identity are handled upstream; a request that reaches booking is already an authenticated user.
  • Food/tourism/other IRCTC verticals. I design train reservation only.

The decision that drives everything: seat inventory is not a set of individually locked rows; it is a per-(train, date, class, quota) counter plus a berth-assignment step, guarded by a single serialization point per that key. Oversell prevention, the Tatkal burst, and the waitlist cascade all reduce to “serialize the decrements on one hot key correctly and cheaply,” and the rest of the system exists to keep that hot key from melting.

Non-Functional Requirements (NFR)

  • Scale: ~8 crore (80M) registered users, ~13,000 trains/day, ~1.5M tickets booked/day overall. But the load is savagely skewed: in the Tatkal minute, 2-3 million users may hit the system concurrently, generating tens of millions of read/search requests and hundreds of thousands to millions of booking attempts in the first 60 seconds, against inventory that is a few tens of seats per train-class.
  • Latency: search should be under ~300ms; a booking decision (confirmed/RAC/WL/reject) should return in under ~1-2s even at peak. The user must not be left staring at a spinner not knowing if their money left their account.
  • Availability: 99.9%+ overall, but the Tatkal window is where availability is judged. The system may shed and queue read load, but the booking commit path must stay correct even while degraded.
  • Consistency: the seat-allocation path needs strong consistency - a seat count and its berth assignment must be linearizable per (train, date, class, quota). No two confirmations for the same berth, ever. Search availability can be eventually consistent (a few seconds stale is fine and expected).
  • Durability: a confirmed booking and its payment must never be lost. Once money is captured and a PNR is confirmed, that fact survives any crash.
  • Fairness: under the burst, allocation should be roughly first-come-first-served by arrival at the booking commit, not “whoever has the fastest connection or a bot.” Fairness is a real requirement here, not a nicety.

The tension to state up front: strong consistency on a single hot key fights throughput under a million concurrent writers. A naive “lock the train row and decrement” serializes correctly but caps you at the speed of one lock and one disk, which the Tatkal burst obliterates. We resolve it by admission control (a virtual queue) to flatten the burst, narrowing the serialized critical section to an in-memory atomic decrement per (train, class, quota) key, and making the slow work (payment, berth assignment persistence, notifications) asynchronous around that tiny synchronous core. Strong consistency stays only where physics demands it - the count decrement - and everything else is pushed off the hot path.

Back-of-the-Envelope Estimation (BoE)

Real numbers with the arithmetic, because they dictate the architecture.

Baseline vs peak load:

Registered users:            ~80,000,000
Tickets booked per day:      ~1,500,000
Baseline booking rate:       1.5M / 86,400s = ~17 bookings/sec average.

Tatkal is the spike. Say 3,000,000 users converge at 10:00:00:
  Each opens the app, searches, and mashes "Book" a few times.
  Reads (search + availability + refresh): assume ~10 reads/user in the
    first minute = 3M * 10 = 30,000,000 reads / 60s = ~500,000 read QPS peak
    (bursting far higher in the first 5 seconds).
  Booking attempts: assume ~1.5 attempts/user in the first minute =
    3M * 1.5 = 4,500,000 attempts / 60s = ~75,000 booking attempts/sec,
    with the first 10 seconds carrying most of it -> 200,000+ attempts/sec burst.

So the read path is a ~500K QPS problem and the write path is a ~100-200K attempts/sec problem - but almost all of those attempts must be rejected quickly, because the inventory is tiny.

The inventory is minuscule:

A popular train, sleeper class, Tatkal quota: ~80-120 seats.
AC 3-tier Tatkal: ~30-60 seats.
So a single hot (train, date, class, quota) key has ~O(100) units of stock.
Across ~13,000 trains and ~6 classes, total Tatkal seats/day are maybe
  13,000 * ~200 avg = ~2.6M seat-units, but demand concentrates on a few
  hundred popular trains on popular routes. A dozen trains absorb the storm.

The headline: hundreds of thousands of writers per second are fighting over a few dozen seats on each hot key. ~99.9% of booking attempts on a hot train will fail or go to waitlist. The engineering is almost entirely about saying “no” correctly and fast, and “yes” exactly the right number of times.

Storage:

A booking record (PNR): passengers (avg 2) + train/date/class/quota +
  berth assignments + status + fare + payment ref ~ 2-3 KB.
1.5M bookings/day * 3 KB = ~4.5 GB/day of booking data.
Per year: ~1.6 TB/year of booking records, retained for years (audit, refunds,
  legal). Plus payment/txn logs of similar order.

Inventory state is tiny: (train, date, class, quota) -> counts + a berth bitmap.
  ~13,000 trains * ~60-day booking window * ~6 classes * ~5 quotas
  = ~23M inventory keys, each a few hundred bytes with the berth bitmap.
  ~23M * ~500 B = ~12 GB of live inventory - fits comfortably in memory/cache.

The live, hot inventory is only ~10-15 GB. That is the crucial fact: the contended state fits in RAM, so the serialization can happen in an in-memory store (Redis/an in-memory sharded service) rather than round-tripping to disk per attempt.

Bandwidth / cache:

Search results are cacheable: train lists and per-route static data are
  read-heavy and change slowly -> CDN + cache, ~95%+ hit rate.
Availability numbers change fast during Tatkal but are read millions of times;
  cache them with a short TTL (1-2s) so 500K read QPS collapses to a
  few thousand origin refreshes/sec. Users see "AVAILABLE / WL 42" that may be
  1-2s stale - acceptable, because the booking commit re-checks the truth.

The takeaway: the read storm is solved by caching and admission control; the genuinely hard part is the write path on a handful of hot keys - decrementing a tiny counter correctly under a 200K/sec burst, assigning specific berths, generating PNRs, running the waitlist cascade, and holding seats through payment without oversell. That is where the design earns its keep.

High-Level Design (HLD)

The spine: a CDN + search tier absorbs read load; a virtual queue / admission control service flattens the Tatkal burst into a rate the core can handle; stateless booking API servers validate requests and call the inventory service, which is the single serialization point per (train, date, class, quota) and holds the authoritative counts and berth map in memory; a PNR / booking service persists the durable booking in a sharded SQL store; payment runs asynchronously with a hold-and-confirm; and a waitlist/chart worker runs the promotion cascade and chart preparation. A CDC/event stream keeps the read-side availability cache eventually consistent.

              Millions of users at 10:00:00
                        │  search / book
                        ▼
              ┌──────────────────────┐
              │   CDN + Edge          │  static assets, cached search,
              │                       │  cached availability (1-2s TTL)
              └──────────┬───────────┘
                         │ misses + all booking requests
                         ▼
              ┌──────────────────────┐
              │  Virtual Queue /      │  admission control: token per user,
              │  Admission Control    │  releases users into booking at a
              │  (waiting room)       │  controlled rate; FIFO fairness
              └──────────┬───────────┘
                         │ admitted requests (throttled)
                         ▼
              ┌──────────────────────┐
              │  Booking API (stateless, N)   │  authN, validate passengers,
              │                                │  fare calc, idempotency key
              └───┬───────────────┬───────────┘
                  │ search reads   │ booking commit
                  ▼                ▼
        ┌──────────────┐   ┌──────────────────────────────┐
        │ Availability │   │  INVENTORY SERVICE            │
        │ read cache   │   │  sharded by (train,date,class,│
        │ (Redis)      │◄──│  quota) key. Atomic decrement  │
        └──────────────┘   │  + berth assignment. The ONE  │
             ▲ CDC         │  serialization point. In-mem. │
             │             └───────┬───────────────┬───────┘
             │                     │ HOLD granted   │ sold-out -> WL/RAC
             │                     ▼                ▼
   ┌─────────┴────────┐   ┌──────────────────┐  ┌──────────────────┐
   │  Booking / PNR    │   │  Payment Service  │  │  Waitlist Manager │
   │  Service (SQL,    │◄──│  (async, gateway  │  │  (RAC/WL queues,  │
   │  sharded by PNR)  │   │  callback, hold   │  │  promotion cascade│
   │  durable record   │   │  timeout release) │  │  on cancel)       │
   └─────────┬────────┘   └──────────────────┘  └────────┬─────────┘
             │ events (booked/cancelled/confirmed)         │
             ▼                                             ▼
   ┌──────────────────────────────────────────────────────────┐
   │  Event Stream (Kafka)  -> notifications (SMS/email),        │
   │  availability cache refresh, chart prep, analytics, audit  │
   └──────────────────────────────────────────────────────────┘
                         │
                         ▼
              ┌──────────────────────┐
              │  Chart Preparation    │  hours before departure: finalize
              │  Worker               │  vacancies, settle RAC/WL, freeze berths
              └──────────────────────┘

Booking flow (the happy path, a confirmed Tatkal seat):

  1. The user has been sitting in the virtual queue since before 10:00. At 10:00 the queue starts admitting users into the booking tier at a controlled rate (say a few thousand/sec per hot train, matched to how fast the core can decide). Admission is roughly FIFO by queue-join time, which is the fairness mechanism.
  2. An admitted request hits a stateless booking API server. It validates the passenger list, computes the fare, and attaches an idempotency key (so a retry does not double-book). It then calls the inventory service for the exact (train, date, class, quota) key.
  3. The inventory service owns that key on exactly one shard/thread. It performs an atomic decrement: if available > 0, it decrements, assigns a specific berth from the berth bitmap, and returns a short-lived HOLD (say 5-10 minutes) tagged with the berth and a hold id. If available == 0, it instead takes a waitlist ticket (RAC if RAC slots remain, else WL position N) atomically. Either way the answer is definitive and returned in milliseconds.
  4. The booking API creates a durable booking/PNR record in the sharded SQL store in PENDING_PAYMENT state with the held berth and generated PNR, then sends the user to payment.
  5. Payment runs against the external gateway. On success callback, the booking flips to CONFIRMED, the hold becomes a permanent allocation, and a booked event is emitted. On failure or hold timeout, the booking is abandoned and the inventory service releases the seat back (increment count, free the berth, and immediately try to promote a waitlisted passenger).
  6. Events fan out: the availability cache is refreshed via CDC, an SMS/email confirmation is sent, and analytics/audit consume the stream. The user sees “Confirmed, S4-32.”

Sold-out / waitlist flow: steps 1-2 identical; at step 3 the count is already 0, so the user gets a RAC or WL PNR immediately. Their booking still goes through payment (railways charge for WL tickets). Later, when confirmed passengers cancel or the chart is prepared, the waitlist manager promotes them and notifies.

The structural insight: the only strongly-consistent, serialized operation is the atomic decrement-and-assign on one small hot key, and it happens in memory. Everything expensive - payment, durable persistence, notifications, cache refresh - is pushed outside that critical section and made asynchronous, so the serialized core stays microseconds-small and the burst is absorbed by admission control in front of it.

Component Deep Dive

The hard parts, naive-first then evolved: (1) seat inventory under the burst without oversell, (2) the Tatkal thundering herd and fairness, (3) PNR generation, (4) waitlist, RAC, and auto-upgrade cascades.

1. Seat inventory: decrement without oversell, at 200K attempts/sec

The job: for a (train, date, class, quota) key with ~O(100) seats, grant each seat to exactly one booking under hundreds of thousands of concurrent attempts, and never grant seat number 101.

Naive approach: a seats table, one row per berth, locked on booking

Model every physical berth as a row: (train, date, class, berth_no, status, pnr). To book, SELECT ... WHERE status='FREE' LIMIT 1 FOR UPDATE, set it to BOOKED, commit.

BEGIN;
SELECT berth_no FROM seats
 WHERE train=12951 AND date='2026-07-23' AND class='3A' AND status='FREE'
 ORDER BY berth_no LIMIT 1 FOR UPDATE;      -- row lock on the first free berth
UPDATE seats SET status='BOOKED', pnr=:pnr WHERE berth_no=:b AND ...;
COMMIT;

Where it breaks:

  • Everyone locks the same rows. At 10:00, 200K transactions/sec all run the same SELECT ... FOR UPDATE targeting the first free berths. They serialize on those few hot rows, forming a giant lock queue. Throughput collapses to the speed of one lock holder at a time - a few hundred to low thousands of commits/sec, not 200K. Most transactions sit blocked, then time out.
  • Lock contention becomes deadlock and timeout storms. With ORDER BY berth_no, everyone grabs berth 1 first; contention is maximal. Transactions hold locks across the app round-trip and pile up; the DB spends its time managing lock queues, not booking.
  • The database is doing OLTP row-locking for a problem that is really “decrement a small integer.” You are paying disk, WAL, and MVCC costs per attempt when 99.9% of attempts should be a fast rejection.
  • Connection exhaustion. 200K concurrent transactions need 200K connections/threads; the DB has hundreds. It falls over long before the seats sell out.

Row-level locking is correct but its throughput ceiling is orders of magnitude below the burst, and it puts the hottest, most contended work on the slowest tier (disk-backed SQL).

Evolved approach: in-memory atomic counter per key + async berth assignment

Split the problem into the part that must be serialized (the count) and the part that can be lazy (which specific berth). Keep the authoritative count and a berth bitmap in an in-memory, single-owner store (Redis, or a sharded in-memory inventory service), sharded so that each hot key lives on exactly one owner.

  • The count is one integer per key. Booking a seat is a single atomic decrement with a floor at zero: in Redis, a small Lua script that does if avail > 0 then avail = avail - 1; assign berth; return HOLD else return WAITLIST end, executed atomically on the one shard owning the key. Redis is single-threaded per shard, so this is the serialization point - no lock queue, no MVCC, just an in-memory compare-and-decrement that runs in microseconds. One shard can do ~100K+ such ops/sec, and a hot key sells out its ~100 seats in well under a second anyway.
  • Berth assignment from a bitmap, in the same atomic step. Alongside the count, keep a bitmap of free berths for the key (100 bits = tiny). The same Lua script clears the lowest free bit and returns that berth number, so the count and the specific berth are decided together, atomically, with no second round-trip and no chance of two bookings getting the same berth.
  • Holds, not immediate sale. The decrement grants a HOLD with a TTL (the payment window). The seat is out of inventory during the hold. If payment confirms, the hold is finalized; if it times out, a reaper (or Redis key expiry + a release script) increments the count, sets the berth bit back to free, and triggers waitlist promotion. This is what lets us hold a seat through a slow payment without either overselling or losing the seat if the user vanishes.
  • Durability behind the memory. The in-memory count is the fast serialization layer, but the durable truth of who holds what lives in the SQL booking record and an append-only inventory event log. On a shard restart, rebuild the in-memory count/bitmap from the last snapshot plus the event log (confirmed + held - released). The in-memory store is a fast, authoritative cache of a small integer, reconstructable from the durable log - not the only copy.
Inventory key: inv:{train}:{date}:{class}:{quota}
  fields:  avail (int), rac_avail (int), wl_next (int), berth_bitmap (bytes)

BOOK (atomic Lua on the owning shard):
  if avail > 0:
      avail -= 1
      berth = lowest_free_bit(berth_bitmap); set that bit
      return HOLD(berth, hold_id, ttl=8m)
  elif rac_avail > 0:
      rac_avail -= 1
      return RAC(rac_position)
  else:
      pos = wl_next; wl_next += 1
      return WAITLIST(pos)

RELEASE (on payment fail / hold timeout / cancel):
  avail += 1; clear berth bit; enqueue promote-waitlist(key)

Why this is right:

  • The contended operation is a microsecond in-memory decrement, not a disk transaction. The burst hits a single-threaded shard doing trivial work; it saturates gracefully at ~100K ops/sec and drains the tiny inventory in under a second, then cheaply returns WAITLIST to everyone else.
  • Oversell is structurally impossible. The decrement-with-floor and the bitmap are updated atomically on one owner; there is no interleaving that lets avail go negative or two holders share a berth.
  • The slow, expensive work is outside the critical section. Payment, durable persistence, and notifications happen after the hold is granted; the serialized core does none of it.
  • It degrades to rejection cheaply. Once sold out, every further attempt is a single integer read/compare returning WAITLIST - the exact behavior you want when a million people are still trying after the seats are gone.

This “in-memory atomic counter + bitmap, holds with TTL, durable event log behind it” is the standard pattern for scarce-inventory flash systems (ticketing, flash sales), because it puts the one unavoidable serialization on the cheapest possible operation and pushes everything else off the hot path.

2. The Tatkal thundering herd and fairness

The job: 2-3 million users hit “Book” within the same second against a system whose core can decide maybe tens of thousands of bookings/sec per hot train. Absorb that without collapsing, and be fair (first-come by arrival, not by bot or bandwidth).

Naive approach: let every request hit the booking servers directly

Scale the stateless booking tier horizontally and let all 3M requests flow straight through to the inventory service; auto-scaling will handle it.

Where it breaks:

  • You cannot auto-scale into a 1000x spike in one second. Spinning up capacity takes minutes; the spike is over in seconds. The system is overwhelmed before scaling reacts, and the overload itself (connection floods, retries) makes it worse.
  • Retries amplify the storm. A user who gets a spinner refreshes and re-submits. A failed request is retried by the client. 3M users become 15M requests in the first ten seconds - a self-inflicted DDoS.
  • Fairness is by luck. Whoever has the fastest network, the closest edge, or a scripted bot that fires 50 requests wins the seats. Real users on slow connections lose to bots. That is exactly the outcome the railway wants to prevent.
  • The inventory shard melts anyway. Even the fast in-memory decrement can be swamped if 15M requests/sec slam one shard’s front door with connection setup and teardown.

Letting the herd through unshaped means the infrastructure, not the inventory, becomes the bottleneck, and the outcome is unfair and unstable.

Evolved approach: a virtual queue (waiting room) with admission control

Put a virtual waiting room in front of the booking tier that admits users at a controlled rate, in roughly the order they arrived.

  • Everyone gets a queue token before they can book. When a user opens the Tatkal page (even before 10:00), they receive a signed queue token with a join timestamp/position. The booking tier accepts a request only if it carries a valid, now-admitted token. This single gate turns “3M simultaneous writers” into “a stream the core can actually serve.”
  • Admit at the rate the core can drain. A controller releases tokens for a given hot train at a rate matched to how fast inventory + payment can decide - e.g. release in waves of a few thousand/sec. The first waves take the real seats; later waves get waitlist. Because admission is paced, the inventory shard sees a steady manageable flow, never a 15M/sec wall.
  • FIFO-ish fairness by join time. Admission order follows queue-join order (bucketed into time slices to avoid needing a perfectly global order). A user who joined at 09:58 is admitted before one who joined at 10:01. This defeats the “fastest connection wins” problem and, combined with per-account rate limits and CAPTCHA/bot checks at queue join, blunts scripted bots.
  • Honest feedback instead of spinners. While queued, the user sees “You are number 48,201; estimated wait 2 minutes” - a truthful waiting-room page served entirely from the edge/queue service, not the booking core. This kills the refresh-retry amplification because refreshing does not jump the queue and the user knows their status.
  • Idempotency for the retries that remain. Each admitted booking attempt carries an idempotency key; the inventory/booking path dedupes on it, so a client retry after a timeout never double-decrements or creates two PNRs.
Pre-10:00   user opens page -> issued queue_token(join_ts, user_id, sig)
10:00:00    controller opens hot-train quota; begins admitting tokens
            wave 1: admit tokens with join_ts in [.., t1]  (few thousand)
            wave 2: admit next slice ... paced to inventory drain rate
Booking API: reject request unless token is 'admitted' AND within TTL
            attach idempotency_key = hash(user, train, date, attempt)
Inventory:   serve admitted requests at a rate it can actually handle
Seats gone:  remaining admitted users get WAITLIST; further waves told
             'sold out / WL only' at the edge, no need to hit inventory

Why this is right:

  • The burst is flattened before it reaches anything stateful. The queue converts an impossible instantaneous spike into a paced stream, so the inventory shard and payment tier run at their sustainable rate instead of drowning.
  • Fairness becomes a property of the design, not luck. Admission by join time, per-account limits, and bot filtering at the gate make allocation roughly first-come-first-served for real users.
  • Retry amplification is defused. A truthful waiting-room page plus idempotency keys means refreshing and retrying neither helps the user nor hurts the system.
  • The core stays simple. Inventory and payment never have to be “web-scale”; they only have to be as fast as the admission rate, which the controller sets to what they can safely sustain.

The waiting room is the load-shaping layer that makes strong consistency on a tiny hot key survivable: you cannot make the decrement infinitely parallel, so you meter the arrivals down to what the single-owner decrement can serve.

3. PNR generation: unique 10-digit IDs at booking rate

The job: every booking gets a globally unique 10-digit PNR, generated fast, with no collisions, even when many booking servers create PNRs concurrently during the burst.

Naive approach: a database auto-increment or SELECT MAX+1

Use a single SQL sequence / auto-increment column, or read the current max PNR and add one.

Where it breaks:

  • A single sequence is a global bottleneck and SPOF. Every one of hundreds of thousands of bookings/sec calling one sequence generator serializes on it, and if that node is down, no one can book.
  • SELECT MAX+1 races. Two servers read the same max and generate the same PNR - a duplicate, which is a correctness disaster (two passengers, one PNR).
  • Sequential PNRs leak information and invite scraping. Perfectly incrementing IDs let anyone enumerate other people’s bookings by guessing adjacent numbers.

A single centralized counter cannot keep up with the burst and becomes the very SPOF the rest of the design avoids.

Evolved approach: sharded ID allocation with pre-reserved blocks, formatted to 10 digits

Generate PNRs locally on each booking server from a pre-allocated block, so the hot path needs no coordination.

  • Block allocation. A lightweight ID service hands each booking server a block of, say, 10,000 IDs at a time ([base, base+9999]) from a partitioned space. The server issues PNRs from its block with a local atomic counter and asks for a new block when it runs low. Coordination happens once per 10,000 PNRs, not per PNR - the burst hits a local increment.
  • Structure the 10 digits. The 10-digit PNR encodes a partition/zone prefix + a sequence within the partition (real IRCTC PNRs start with a digit tied to the PRS zone). Different partitions never collide because their prefixes differ, so blocks across servers/zones are disjoint by construction. A check-friendly layout keeps it exactly 10 digits.
  • No guessable adjacency across users. Because PNRs are drawn from many partitions and blocks in parallel, consecutive real bookings do not get consecutive PNRs, which reduces trivial enumeration. (If stronger unguessability is needed, the display PNR can be a formatted mapping over the raw id, but 10 digits limits entropy, so the real defense is authorization checks on lookup, not obscurity.)
  • Uniqueness is guaranteed by disjoint blocks, not by a global lock. Each block is owned by exactly one server until exhausted; within a block the counter is local and monotonic. Two servers can never mint the same PNR because they hold different blocks.
ID service partitions:  P0..Pn  (each maps to a PNR prefix digit range)
Booking server startup / low-water: request block from a partition
  -> gets [base, base+9999], marks it consumed durably in the ID store
Per booking:  pnr_raw = base + local_counter++   (local, lock-free)
              PNR    = format10(partition_prefix, pnr_raw)  // exactly 10 digits
Collision-free: blocks are disjoint; a block is never handed out twice.
Durability: block hand-outs are persisted, so a crashed server's unused
            block is simply retired (small gap), never reissued.

Why this is right:

  • No coordination on the hot path. PNR minting is a local atomic increment; the shared ID service is touched once per 10,000 IDs, so it is nowhere near a bottleneck even at burst.
  • Zero collisions by construction. Disjoint blocks make duplicates impossible without any distributed lock.
  • No SPOF. Any booking server with a non-empty block keeps minting PNRs even if the ID service is briefly unavailable; it only needs the ID service to refill.
  • Cheap and simple. A block counter and a formatter - no Snowflake clock-sync worries needed for a low-rate-per-partition 10-digit space, though a Snowflake-style (timestamp + shard + seq) scheme is a fine alternative if you prefer time-ordered IDs.

The principle: turn a global uniqueness requirement into local, coordination-free generation by partitioning the ID space and pre-allocating disjoint blocks - the same move that made inventory scale, applied to identity.

4. Waitlist, RAC, and the auto-upgrade cascade

The job: when confirmed seats run out, place bookings on RAC then WL; when a confirmed passenger cancels (or the chart is prepared), promote WL to RAC and RAC to confirmed - automatically, in order, without race conditions or double-promotion.

Naive approach: on each cancel, scan the bookings table for the “next” waitlisted PNR and update it

On a cancellation, run a query to find the lowest RAC/WL for that (train, date, class), promote it, repeat.

Where it breaks:

  • Concurrent cancels double-promote. Two cancels processed at once both find the same “highest WL” and both promote it, or both claim the same freed berth - an oversell or an inconsistent waitlist.
  • The “who is next” scan is a moving target under load. During chart prep or a cancellation storm, positions shift constantly; a naive re-scan sees stale positions and produces gaps or duplicates in the WL sequence.
  • No clear ordering guarantee. WL fairness requires strict position order (WL1 before WL2). Ad-hoc updates driven by whichever cancel arrives first can violate that if not serialized against the same key.

Treating promotion as independent table updates loses the ordering and atomicity that waitlist semantics demand.

Evolved approach: per-key ordered waitlist state, single-writer promotion, event-driven

Model RAC and WL as ordered structures owned by the same single-writer inventory key, and make promotion an atomic step on that owner, triggered by release events.

  • Ordered waitlist per key. For each (train, date, class, quota), keep rac_avail, the next WL position counter, and an ordered list/queue of RAC and WL PNRs (position N -> PNR). Because this lives on the same single-owner shard as the count, all mutations - book, release, promote - are serialized against each other. There is no cross-writer race by construction.
  • Promotion is one atomic transition on release. When a confirmed seat is released (cancel, payment timeout, or chart vacancy), the owner runs an atomic script: free berth -> pull the head RAC PNR, mark it CONFIRMED with that berth -> pull the head WL PNR, move it into the freed RAC slot -> decrement/adjust positions. This “WL -> RAC -> CONFIRMED” shift happens as one indivisible step, so no PNR is promoted twice and the order is exactly preserved.
  • The durable booking record is updated via events, not in the critical section. The atomic promotion produces an event promote(pnr -> CONFIRMED, berth S4-32); a consumer updates the SQL booking record and fires the SMS. The serialized owner only mutates the small in-memory ordered state; the slow persistence and notification are async, exactly as with the initial booking.
  • RAC semantics. RAC is “two passengers share one full berth” - a defined pool (rac_avail) separate from confirmed. RAC passengers can travel; they get a full berth if one frees up. The cascade treats RAC as the middle tier between WL and CONFIRMED, so a single cancellation ripples: CNF frees -> top RAC becomes CNF -> top WL becomes RAC.
  • Auto-upgrade at chart prep. The optional auto-upgrade scheme promotes confirmed passengers into empty berths of a higher class for free, to fill vacancies and open lower-class seats for waitlisted passengers. This runs in the chart preparation batch: with booking effectively frozen, the worker computes vacancies per class, applies upgrade eligibility top-down, then runs the WL/RAC cascade to consume the freed lower-class berths. Doing it in a controlled batch (not live) avoids fighting concurrent bookings and lets it optimize globally.
On RELEASE(key, freed_berth):        # atomic on the key's single owner
  if RAC queue non-empty:
      rac_pnr = RAC.pop_head()
      confirm(rac_pnr, freed_berth)   # RAC -> CONFIRMED
      if WL queue non-empty:
          wl_pnr = WL.pop_head()
          RAC.push(wl_pnr)            # WL -> RAC
      else:
          rac_avail += 1
  else:
      avail += 1; free berth bit      # nobody waiting, seat goes back to pool
  emit promotion events -> update SQL + notify (async)

Chart prep (batch, booking frozen):
  compute per-class vacancies
  apply auto-upgrade top-down (fill higher class from lower CNF)
  run cascade to consume freed lower-class berths for RAC/WL
  freeze final berth map -> chart

Why this is right:

  • Ordering and atomicity are guaranteed because promotion is a single-writer atomic transition on the same key that owns the count - the same serialization that prevents oversell also prevents double-promotion.
  • The cascade is exactly one step per release, so a cancellation storm is just many independent atomic transitions, each cheap and correct, rather than repeated full-table scans.
  • Fairness in the waitlist (WL1 promoted before WL2) falls out of the ordered queue and single-writer processing.
  • Auto-upgrade is a global optimization done when it is safe - in the chart batch, not racing live bookings - so it can fill higher classes and free lower-class seats without inconsistency.

The waitlist is not a separate subsystem bolted on; it is the other half of the same single-owner inventory key. Book decrements one side; cancel/promote shifts the ordered queues on the same owner. One serialization point covers both directions.

API Design & Data Schema

External APIs

REST/HTTPS over the booking tier (behind the virtual queue). All booking calls carry a queue token and an idempotency key.

GET /v1/search?from=NDLS&to=BCT&date=2026-07-23&class=3A
  -> 200 { "trains": [
       { "train_no":"12951", "name":"Mumbai Rajdhani",
         "classes": { "3A": { "quota":"TATKAL",
                              "status":"AVAILABLE", "avail":42 } },
         "as_of":"2026-07-22T10:00:03Z" } ] }        // may be 1-2s stale

POST /v1/queue/join            # before/at 10:00
  body: { "train_no":"12951", "date":"2026-07-23", "class":"3A" }
  -> 200 { "queue_token":"qt_9f3...", "position":48201, "eta_sec":120 }

GET  /v1/queue/status?token=qt_9f3...
  -> 200 { "state":"WAITING"|"ADMITTED", "position":1203 }

POST /v1/bookings              # only accepted if token ADMITTED
  headers: { Idempotency-Key: "usr42-12951-20260723-a1" }
  body: {
    "queue_token":"qt_9f3...",
    "train_no":"12951", "date":"2026-07-23",
    "class":"3A", "quota":"TATKAL",
    "passengers":[ {"name":"A Kumar","age":34,"gender":"M"},
                   {"name":"S Kumar","age":31,"gender":"F"} ],
    "payment_method":"UPI"
  }
  -> 200 {
    "pnr":"2641097835", "status":"HELD",
    "allocation":[ {"pax":"A Kumar","coach":"B4","berth":32,"type":"LOWER"},
                   {"pax":"S Kumar","coach":"B4","berth":33,"type":"MIDDLE"} ],
    "fare":4820, "hold_expires":"2026-07-22T10:08:00Z",
    "payment_url":"/v1/pay/txn_77c..." }
  -> 200 (sold out) { "pnr":"2641097840", "status":"WL", "wl_position":37, ... }

POST /v1/pay/{txn}/callback    # from gateway (async)
  -> flips booking CONFIRMED or releases the hold

GET  /v1/pnr/{pnr}             # authz: only owner; defeats enumeration
  -> 200 { "pnr":"2641097835", "status":"CNF", "chart_prepared":false, ... }

DELETE /v1/bookings/{pnr}      # cancel -> refund + triggers cascade
  -> 200 { "status":"CANCELLED", "refund":4100 }

Data model and store choices

1. Inventory state - in-memory, single-owner per key (Redis / sharded in-mem service). Strongly consistent per key, microsecond atomic ops. Reconstructable from a durable event log; not the only copy.

KEY  inv:{train}:{date}:{class}:{quota}          # e.g. inv:12951:2026-07-23:3A:TATKAL
  avail        int          # confirmed seats left
  rac_avail    int          # RAC slots left
  wl_next      int          # next WL position to hand out
  berth_bitmap bytes        # 1 bit per berth, 1 = free
  rac_queue    list<pnr>    # ordered RAC (head = highest priority)
  wl_queue     list<pnr>    # ordered WL  (head = WL1)
  Shard key: the whole inventory key -> one owner. Single writer per key.

2. Bookings / PNR - sharded SQL (Postgres), source of truth for a booking. Relational because a booking is a transactional entity with passengers, payment, and status that needs ACID and joins for lookups, refunds, and audits. Strong consistency on the record itself; the money and the PNR must be durable.

-- shard by PNR (hash), so lookups by PNR hit one shard
CREATE TABLE booking (
  pnr           CHAR(10) PRIMARY KEY,
  train_no      VARCHAR(8)  NOT NULL,
  journey_date  DATE        NOT NULL,
  class         VARCHAR(4)  NOT NULL,
  quota         VARCHAR(8)  NOT NULL,
  status        VARCHAR(12) NOT NULL,   -- HELD|CNF|RAC|WL|CANCELLED|PENDING_PAY
  wl_position   INT,                    -- null unless WL
  fare_paise    BIGINT      NOT NULL,
  payment_ref   VARCHAR(40),
  user_id       BIGINT      NOT NULL,
  hold_expires  TIMESTAMPTZ,
  created_at    TIMESTAMPTZ NOT NULL DEFAULT now(),
  idempotency_key VARCHAR(64) UNIQUE     -- dedupe retries: no double PNR
);
CREATE INDEX ON booking (user_id, created_at);           -- "my bookings"
CREATE INDEX ON booking (train_no, journey_date, class); -- ops / chart

CREATE TABLE passenger (
  pnr        CHAR(10) REFERENCES booking(pnr),
  seq        SMALLINT,               -- 1..6
  name       VARCHAR(60), age SMALLINT, gender CHAR(1),
  coach      VARCHAR(4),  berth INT, berth_type VARCHAR(6),
  status     VARCHAR(12),            -- per-pax (can differ: CNF + WL)
  PRIMARY KEY (pnr, seq)
);

3. Inventory event log - append-only (Kafka / log store). Every decrement, hold, release, promotion is an event. This is the durable truth used to rebuild the in-memory shard after a crash and to audit “how did this berth get allocated.” Append-only sequential writes, cheap and fast.

event: { seq, key, type:HOLD|CONFIRM|RELEASE|PROMOTE, pnr, berth,
         avail_after, ts }     # replay -> reconstruct inventory state

4. Availability read cache - Redis, eventually consistent. Denormalized per (train, date, class, quota) availability number, refreshed from the inventory event stream (CDC), short TTL. Serves the 500K read QPS; being 1-2s stale is fine because the booking commit re-checks the truth.

5. Reference data - read-mostly SQL + CDN. Train schedules, station lists, coach layouts, quota rules. Rarely changes, heavily cached at the edge.

Why this split: the contended write (decrement) demands in-memory strong consistency on a tiny key, so it lives in a single-owner in-memory store, not SQL. The durable booking (money, PNR, passengers) demands ACID and rich queries, so it lives in sharded SQL. The read storm demands cheap, stale-tolerant fanout, so it lives in a cache + CDN. Match each store to its access pattern instead of forcing one database to do all three and failing at all three.

Bottlenecks & Scaling

Where it breaks first, in order, and the fix for each.

1. The read storm on search/availability (~500K QPS) - breaks first. Millions refreshing availability would flatten the origin. Fix: CDN for static/search results and a Redis availability cache with a 1-2s TTL, refreshed from the inventory CDC stream. 500K read QPS collapses to a few thousand origin refreshes/sec at ~99% cache hit. Users accept slightly stale numbers because the commit re-checks.

2. The hot inventory key (one train-class-quota) under 200K attempts/sec. A single-owner key is a single serialization point. Fix: it does not need to be parallel - it needs to be fast and metered. The virtual queue paces arrivals to what the owner can drain; the owner does a microsecond in-memory atomic decrement, so it sustains ~100K ops/sec and sells out ~100 seats in under a second, then cheaply returns WAITLIST. Different trains are different keys on different shards, so unrelated hot trains scale out horizontally. The shard key is the full (train, date, class, quota) - the unit of contention and consistency at once.

3. The Tatkal spike overwhelming the front door. 3M users in one second. Fix: admission control / virtual waiting room (deep dive 2) flattens the burst before it reaches anything stateful, gives FIFO fairness, filters bots at the gate, and defuses retry amplification with truthful status pages + idempotency keys.

4. Payment tier and hold pressure. Every granted hold ties up a seat and a payment session for minutes; the gateway is slower than the decrement. Fix: holds with TTL so an abandoned checkout auto-releases the seat and immediately promotes a waitlisted passenger; async gateway callbacks so booking servers are not blocked on payment; idempotent payment handling so a duplicate callback confirms once. Size the admission rate partly to the payment tier’s capacity, not just inventory’s.

5. PNR generation as a global counter. A single sequence would bottleneck and be a SPOF. Fix: partitioned ID space with pre-allocated disjoint blocks minted locally (deep dive 3) - coordination once per 10,000 IDs, collisions impossible, no SPOF.

6. Booking DB write hot spot and connection limits. Even async, a burst of durable booking inserts can hammer one shard. Fix: shard the booking table by PNR hash so inserts spread across shards; keep the durable write off the synchronous decrement path (write after the hold is granted); use connection pooling / a write queue so a spike buffers instead of exhausting connections. Reads by PNR hit one shard; “my bookings” is served by the user_id index (or a per-user read replica).

7. The waitlist promotion cascade under a cancellation storm. Many concurrent cancels could double-promote. Fix: promotion is a single-writer atomic transition on the same inventory key (deep dive 4), so cancels serialize against bookings and each other - no double-promotion, strict WL order preserved. Persistence and notification are event-driven and async.

8. Single points of failure in the inventory owner. If the shard owning a hot key dies mid-Tatkal, bookings for that train stall. Fix: replicate each inventory shard (primary + sync replica) with the append-only event log as the durable truth; on failover the replica rebuilds avail/bitmap/queues from snapshot + log and resumes. Because the log is the source of truth, no hold or promotion is lost; a brief pause on one train is acceptable, an oversell is not - correctness over availability on the commit path.

9. Storage growth (~1.6 TB/year of bookings + logs, kept for years). Refunds, disputes, and audit forbid purging. Fix: partition booking tables by journey month and archive past-travel bookings to cheaper cold storage; the inventory event log rolls to object storage after the journey date. Snapshots bound how far a shard must replay to recover.

Shard keys, stated plainly: inventory and its waitlist shard by (train, date, class, quota) - simultaneously the unit of contention, ordering, and consistency. Bookings shard by PNR hash for even write spread and single-shard PNR lookup. The availability cache is keyed the same as inventory. Gateways and booking API servers are stateless (scale for load only, no sharding). PNRs are partitioned by zone/block for coordination-free generation. Never shard a single inventory key internally - one key is one writer by design, and that single-writer property is exactly what makes oversell impossible.

Wrap-Up

The trade-offs that define this design:

  • Strong consistency only on a tiny in-memory key; everything else async. The one thing that must be linearizable - the seat count and berth assignment - is a microsecond atomic op on a single-owner in-memory shard. Payment, durable persistence, notifications, and cache refresh all live outside that critical section. We trade a globally simple “one database does it all” model for a small, fast, correct core surrounded by eventually-consistent scale.
  • Meter the burst instead of trying to parallelize the unparallelizable. You cannot make a single hot key infinitely concurrent, so a virtual waiting room paces arrivals down to what the single-writer decrement can serve - which also buys fairness and kills retry storms. We trade instant access for stability and first-come fairness.
  • Availability is stale, the commit is truthful. Search and availability numbers are cached and eventually consistent; the booking commit re-checks the authoritative count. We trade perfectly live availability for a 500K-QPS read path that does not melt.
  • The waitlist is the same key, not a separate system. Book decrements one side; cancel/promote shifts ordered RAC/WL queues on the same single owner, so oversell prevention and double-promotion prevention are the same serialization. Auto-upgrade is deferred to the chart batch where it can optimize globally without racing live bookings.
  • Correctness over availability on writes. If an inventory shard is uncertain, it halts that key rather than risk an oversell; a briefly stuck train is recoverable, a double-booked berth is not.

One-line summary: a virtual waiting room flattens the 10am Tatkal burst into a paced stream, stateless booking servers mint collision-free PNRs from pre-allocated blocks and call a single-owner in-memory inventory shard that atomically decrements a tiny seat count and assigns a berth (or an ordered RAC/WL position) in microseconds, holds the seat through async payment with a TTL release, and runs the WL-to-RAC-to-CONFIRMED promotion cascade on that same key - so tens of thousands of seats are allocated exactly once each to millions of contending users, never oversold, with the durable booking in sharded SQL and an append-only event log as the source of truth for scale, recovery, and audit.