“Store identity for everyone and let them prove who they are” sounds like a users table with a photo column. Then you look at the numbers and the physics of biometrics, and it stops being CRUD. You are holding fingerprints, irises, and demographics for 1.4 billion people. Every single new enrollment has to be checked against the entire existing population to guarantee that this human does not already hold an identity under another name - a 1:N biometric match against 1.4 billion records, not a key lookup. And once enrolled, a person must be able to prove their identity at a ration shop, a bank, or a SIM counter in under two seconds, 50 million times a day, while a regulator watches every byte of how you store and move the most sensitive data a state can hold about a person.

The core of this problem is that identity must be unique (no duplicate humans), provable in real time (fast 1:1 authentication), and locked down (a biometric leak is permanent and catastrophic) - all at the scale of a country. Uniqueness is a brutal offline matching problem; authentication is a latency-and-availability problem; security is a regulatory and cryptographic problem. These three pull in different directions, and the whole design is about keeping them apart so each can be solved with the right tool. Let me build it properly.

Functional Requirements (FR)

In scope:

  • Enrollment. A resident presents demographics (name, date of birth, gender, address) and biometrics (10 fingerprints, 2 iris scans, a face photo) at an enrollment center. The system must verify this person is not already enrolled, assign a unique 12-digit Aadhaar number if they are new, and store the record. This dedup step is the heart of the system.
  • Deduplication (1:N). Before issuing a number, match the new biometrics against every existing enrolled person to catch a resident trying to enroll twice (fraud, benefit duplication). A false “new” (issuing two numbers to one person) breaks the core promise; a false “duplicate” (rejecting a genuine new person) denies them identity.
  • Authentication (1:1). Given an Aadhaar number plus a biometric sample (fingerprint or iris) or an OTP to the registered mobile, return a yes/no “is this person who they claim to be.” This runs 50M+ times a day with a sub-2-second budget.
  • eKYC. With resident consent, return their demographic fields (name, address, photo, DOB) to a requesting agency after a successful auth, so banks and telcos can onboard without paper.
  • Demographic and biometric updates. Residents move, change names, and children’s biometrics must be re-captured as they grow. Updates must re-run relevant checks.
  • Masking and privacy controls. Never return the raw biometric. Support a Virtual ID (a revocable token that maps to an Aadhaar number) so agencies need not store the real number.

Explicitly out of scope (name it so you own the scope):

  • The enrollment center hardware and operator workflow. Capture devices, operator login, and the client software are an upstream system; a request reaching the core carries already-captured, signed, encrypted biometric packets.
  • The matching algorithm internals. I treat the biometric matcher (fingerprint minutiae, iris code Hamming distance) as a well-defined black box with a score and a threshold. I design the system that runs it at national scale, not the feature extractor.
  • Downstream benefit systems. Ration distribution, bank account linking, and subsidy transfer (DBT) are separate systems that consume auth/eKYC; I expose the APIs, I do not build the welfare logic.
  • Legal/policy framework. Consent law, data-retention law, and who is allowed to request auth are policy; I build the technical controls that enforce whatever policy sets.

The decision that drives everything: enrollment and authentication are two completely different systems that happen to share a data store. Enrollment is an offline, compute-crushing 1:N dedup that can take minutes to hours and must be perfectly correct. Authentication is an online, latency-critical 1:1 check that must be fast and always up. Conflating them is the classic mistake; separating them is the whole design.

Non-Functional Requirements (NFR)

  • Scale: ~1.4 billion enrolled identities. Enrollment/update throughput historically ran at hundreds of thousands to ~1 million packets/day at peak rollout. Authentication is ~50 million requests/day and climbing, with strong daytime concentration.
  • Latency: authentication must return in under ~2 seconds end to end (often much less), because a person is standing at a counter. Enrollment dedup is not latency-critical - a new number can be issued hours later via an asynchronous pipeline. eKYC rides on auth, so it shares the 2-second budget.
  • Availability: authentication is critical infrastructure - target 99.9%+, because an outage means people cannot buy subsidized food or access their bank. Enrollment can tolerate hours of downtime; a center just queues packets.
  • Consistency: the uniqueness guarantee is strongly consistent in effect - two concurrent enrollments of the same person must not both get numbers. Authentication reads can tolerate slightly stale replicas of templates and demographics (an update propagating for a few seconds is acceptable). eKYC must reflect the latest demographic data within a short window.
  • Durability: an issued Aadhaar number and its biometric record must never be lost. This is a lifetime identity; losing it strands a citizen.
  • Security and regulation: biometrics are irrevocable - you cannot reissue a fingerprint after a breach. So the biometric store must be encrypted at rest, access-controlled, network-isolated, and audited, with the raw biometrics ideally never leaving a hardened vault. No agency may track a person across services by correlating requests - the system must not become a surveillance graph. Every access is logged for audit.

The tension to state up front: perfect uniqueness demands comparing every new enrollment against all 1.4 billion records, which is computationally galactic, while authentication demands millisecond 1:1 lookups and the strongest possible data protection. We resolve it by making dedup a massively parallel asynchronous batch that uses fast modalities (iris) and binning to avoid brute-forcing 1.4B comparisons, by making auth a cached, sharded 1:1 path that touches a tiny slice of data, and by putting the raw biometrics in a separate hardened vault that the online auth path reaches only through a narrow, audited interface. Three subsystems, three consistency models, one shared identity.

Back-of-the-Envelope Estimation (BoE)

Real numbers with the arithmetic, because they dictate the architecture.

Population and record sizes:

Enrolled identities:        ~1,400,000,000
Per person, biometrics:     10 fingerprints + 2 iris + 1 face photo.
  Raw/compressed images:    ~3-5 MB per person (UIDAI keeps raw images).
Per person, demographics:   name, DOB, gender, address, phone ~ 1 KB.
Per person, templates:      fingerprint minutiae (~0.5-1 KB each),
  iris codes (~512 B each), face template (~2 KB) ~ a few KB total.

Biometric image storage:  1.4B * 4 MB  = ~5.6 PB.
Demographic storage:      1.4B * 1 KB  = ~1.4 TB.
Matching templates:       1.4B * ~6 KB = ~8.4 TB of templates.

So the raw biometric vault is multi-petabyte cold-ish storage, while the templates used for matching are ~8 TB - large but shardable across a cluster and partly holdable in RAM. The crucial split: you match on small templates, you archive the big raw images separately.

Authentication load:

Auth requests/day:   ~50,000,000.
Average:             50M / 86,400s = ~580 req/sec.
Daytime concentration: most auth happens in ~12 waking hours and
  spikes in business bursts. Effective peak ~ 3-5x average
  -> ~2,000-3,000 auth/sec sustained peak, bursting higher.
Each auth: 1:1 template fetch + one match + audit write, sub-2s budget.

Auth is a ~3K/sec, millisecond-per-op, read-heavy problem - entirely tractable with sharding and caching. The hard part of auth is not raw QPS; it is the sub-2-second tail at 99.9% availability under regulatory security, and never leaking the biometric.

Deduplication load - this is the monster:

Peak enrollments:   ~1,000,000 packets/day = ~11.6/sec average arrival.
Naive 1:N dedup:    each new packet vs ALL 1.4B existing records.
  Comparisons/day = 1,000,000 * 1,400,000,000 = 1.4 * 10^15 /day.
Even at 1 microsecond per fingerprint compare, that is
  1.4 * 10^15 us = 1.4 * 10^9 seconds of CPU per day = ~16,000 CPU-days
  of work PER DAY. Brute force is hopelessly infeasible.

That number - 1.4 quadrillion comparisons a day - is the single fact that shapes the dedup design. You cannot brute-force it. You must (a) use iris, which is far more discriminative and faster to match 1:N than fingerprint, (b) bin/filter the gallery so each probe compares against a small candidate set, not all 1.4B, and (c) run it as a huge parallel offline batch (ABIS), not online.

Bandwidth and cache:

Auth payload (encrypted biometric packet): ~5-15 KB in, tiny yes/no out.
  3K/sec * 15 KB = ~45 MB/sec inbound - trivial bandwidth.
Hot cache: active residents authenticate repeatedly (ration, bank).
  Cache their demographic + template slice. Even 100M "active" identities
  * ~6 KB template = ~600 GB of hot templates across a distributed cache -
  shardable, and most auth hits a working set far smaller than 1.4B.

The takeaway: auth is bandwidth-cheap and cache-friendly; dedup is compute-galactic. The architecture spends its cleverness on making dedup survivable (iris + binning + massive parallel batch) and on making auth fast, available, and airtight - and keeps the two from ever sharing a hot path.

High-Level Design (HLD)

Two spines meeting at one data core. The enrollment/dedup spine is asynchronous and compute-heavy: packets land in a queue, a quality/validation stage filters them, and a horizontally scaled ABIS (Automated Biometric Identification System) cluster runs 1:N matching against the template store; only on “no duplicate found” does the enrollment service allocate a 12-digit number and commit the record. The authentication spine is synchronous and latency-critical: agency requests flow through an auth API gateway, which does a 1:1 match against a sharded template store (fronted by cache) and writes an audit record, returning yes/no in under 2 seconds. Both spines sit on top of a hardened biometric vault (raw images, encrypted, isolated) and a demographic store. Nothing on the auth path ever returns raw biometrics.

   ENROLLMENT (async, compute-heavy)        AUTHENTICATION (online, <2s)

  Enrollment center                         Agency (bank/telco/ration)
     │ signed, encrypted packet                 │ AUA/KUA request (VID + sample)
     ▼                                          ▼
 ┌──────────────┐                        ┌──────────────────────┐
 │ Ingest /     │                        │  Auth API Gateway     │  TLS, authN of
 │ Packet Queue │ (Kafka)                │  (stateless, N)       │  agency, rate-limit,
 └──────┬───────┘                        │  VID -> Aadhaar map   │  consent check
        ▼                                └───────┬──────────────┘
 ┌──────────────┐                                │ 1:1 request
 │ Validation / │  quality score, format,         ▼
 │ Quality      │  liveness, demographics     ┌──────────────────────┐
 └──────┬───────┘                            │  Template Cache       │  hot active
        ▼                                     │  (Redis, sharded)     │  residents
 ┌──────────────────────────┐                └───────┬──────────────┘
 │  ABIS cluster (1:N)       │                        │ miss
 │  iris-first filter + bin, │                        ▼
 │  fingerprint confirm.     │   reads      ┌──────────────────────┐
 │  Massively parallel.      │─────────────►│  Template Store       │  sharded by
 └──────┬───────────────────┘   templates  │  (sharded KV, 1:1)    │  Aadhaar/shard
        │ no-dup                            └───────┬──────────────┘
        ▼                                           │ match score vs threshold
 ┌──────────────────────────┐                       ▼
 │  Enrollment Service       │               ┌──────────────────────┐
 │  allocate 12-digit number │               │  Match + Decision     │ yes/no only
 │  commit record (strong)   │               └───────┬──────────────┘
 └──────┬───────────────────┘                        │
        │ writes                                      ▼
        ▼                               ┌──────────────────────────────┐
 ┌───────────────────────────────┐      │  Audit Log (append-only,      │
 │  DATA CORE                     │      │  tamper-evident, every access)│
 │  - Demographic store (SQL)     │      └──────────────────────────────┘
 │  - Template store (sharded KV) │◄──────────── read by auth
 │  - Biometric VAULT (encrypted, │
 │    isolated, raw images, PB)   │  reached only by ABIS + updates, never auth-return
 └───────────────────────────────┘

Authentication flow (the online, sub-2s path):

  1. An agency (an Authentication User Agency - AUA) captures a biometric or triggers an OTP, encrypts the sample with the system’s public key on the capture device, and sends an auth request carrying the Aadhaar number or Virtual ID plus the encrypted sample and a consent flag. The raw biometric is encrypted end to end from the capture device; the agency never sees it in the clear either.
  2. The auth gateway authenticates the agency (mutual TLS, licensed agency keys), rate-limits it, resolves a Virtual ID to the real Aadhaar number via the VID map, and checks consent. It then issues a 1:1 request keyed by the Aadhaar number.
  3. The template cache is checked for that person’s stored template slice. On hit (common for active residents), the stored template is in memory; on miss, it is fetched from the sharded template store (sharded by Aadhaar number) and cached.
  4. The matcher decrypts the incoming sample inside a protected boundary, compares it to the stored template, and produces a score. Score above threshold -> yes; below -> no. For OTP, the gateway instead verifies the one-time code sent to the registered mobile. The response is a plain yes/no (plus optional eKYC payload on consent) - never the stored biometric.
  5. An audit record (who asked, when, which Aadhaar, success/failure - but not the biometric) is written append-only. The yes/no returns to the agency, well inside 2 seconds.

Enrollment flow (the async, dedup-heavy path):

  1. A center captures demographics + 10 fingerprints + 2 iris + photo, signs and encrypts the packet, and uploads it. It lands in the ingest queue; the center’s job is done in seconds regardless of how long dedup takes.
  2. Validation checks packet integrity, biometric quality (a blurry fingerprint is rejected and re-capture requested), liveness, and demographic sanity. Bad packets bounce back; good ones proceed.
  3. The packet enters the ABIS cluster for 1:N dedup. ABIS does not compare against 1.4B records one by one; it uses iris codes for a fast, highly discriminative first pass, bins/filters the gallery to a small candidate set, and confirms with fingerprint matching. If any candidate matches above threshold, the packet is a duplicate - flagged, the resident is informed they already hold a number, and no new number is issued.
  4. On no duplicate, the enrollment service allocates a fresh unique 12-digit Aadhaar number and commits: demographics to the demographic store, templates to the template store, raw images to the vault, all referenced by the number. This commit is strongly consistent and durable.
  5. The new templates are added to the ABIS gallery so the next enrollment is deduped against this person too, and the template cache/replicas are updated for auth.

The structural insight: the only thing shared between the two spines is the data core, and even there the hot paths are disjoint - auth reads a tiny 1:1 slice from cache/template store and never touches the petabyte vault on return, while dedup grinds the whole template gallery offline and writes the vault once. Keep them apart and each is tractable; merge them and neither is.

Component Deep Dive

The hard parts, naive-first then evolved: (1) biometric deduplication (1:N) against 1.4 billion, (2) authentication at 50M/day under a 2-second budget, (3) the biometric vault and privacy/security model, (4) unique 12-digit number allocation.

1. Deduplication: proving uniqueness against 1.4 billion records

The job: for each new enrollment, decide “has this exact human already enrolled?” by matching their biometrics against the entire population, correctly enough that duplicates are caught and genuine new residents are not wrongly rejected.

Naive approach: match the new packet against every record, one by one

For each incoming enrollment, loop over all 1.4B stored fingerprint templates, run the matcher, and if any score exceeds the threshold, declare a duplicate.

for packet in new_enrollments:
    for record in all_1_400_000_000_records:      # the whole gallery
        score = fingerprint_match(packet.fp, record.fp)
        if score > THRESHOLD:
            flag_duplicate(packet, record); break

Where it breaks:

  • The compute is galactic. As the BoE showed, 1M packets/day against 1.4B records is 1.4 quadrillion comparisons a day - tens of thousands of CPU-days of work per day. No cluster you can afford runs this in real time, or even in a day.
  • Fingerprint 1:N is the wrong primary tool. Fingerprint matching is relatively expensive and less discriminative than iris for 1:N; running it against the full gallery maximizes both cost and the false-match rate. With 1.4B comparisons per probe, even a tiny per-pair false-match probability yields many spurious “duplicates,” wrongly rejecting real people.
  • It assumes a single matcher and single vendor. One algorithm has systematic blind spots; a resident whose fingerprints are worn (manual laborers, the elderly) may fail to match themselves or falsely match others. A monoculture matcher bakes in those errors at national scale.
  • It is synchronous in spirit. Trying to do this before issuing a number at the counter would make enrollment take hours while the resident waits. Dedup cannot sit on the enrollment-response path.

Brute-force 1:N is not “slow,” it is physically impossible at this scale, and it uses the weakest modality as its workhorse.

Evolved approach: iris-first filtering, binning, multi-ABIS, massively parallel and async

Make dedup an offline batch that shrinks the candidate set with fast, discriminative signals before any expensive comparison, and spread it across many matchers.

  • Iris as the primary 1:N filter. An iris code is a compact (~512-byte) binary string compared by Hamming distance - cheap, and iris is far more discriminative than fingerprint (two irises per person, extremely low false-match rate). Doing the first pass on iris slashes the candidate set massively and cheaply. Fingerprint then confirms on the handful of survivors, where its cost is affordable.
  • Binning / filtering to avoid full-gallery scans. Partition the gallery by coarse, stable features (fingerprint pattern class - whorl/loop/arch, iris code high-order bits, plus demographic blocking like age band and gender) so a probe only compares against the bin it falls into, not all 1.4B. Good binning turns a 1.4B-wide scan into a scan of millions or fewer per probe, several orders of magnitude cheaper, while keeping the true match in-bin with high probability.
  • Multi-ABIS (multiple matchers, voting). Run several independent biometric matchers (different vendors/algorithms) in parallel and combine their verdicts. This is exactly what a real national system does: no single algorithm’s error profile dominates, worn-fingerprint and edge-case residents are caught by another modality/vendor, and the fused false-match and false-non-match rates drop far below any single engine. A packet flagged as a potential duplicate by the fused score goes to manual adjudication before anyone is denied.
  • Massively parallel, horizontally scaled, offline. The gallery templates are sharded across an ABIS cluster; each probe fans out to the shards covering its bins, each shard matches in parallel, results fuse. Because it is asynchronous (the queue decouples arrival from processing), you size the cluster for throughput, not latency - a packet may take minutes to hours, which is fine, since the number is issued later and delivered to the resident afterward.
  • Append to the gallery on success. A newly confirmed-unique person’s templates are added to every relevant bin/shard so subsequent enrollments are deduped against them too. This write is the only point where dedup touches the live gallery, and it is batched.
Probe (new packet) dedup pipeline (async, parallel):
  1. extract iris codes + fp pattern class + demographic block
  2. candidate_bins = bins(iris_highbits, fp_class, age_band, gender)
  3. iris 1:N within candidate_bins (Hamming) -> short candidate list
  4. fingerprint confirm on candidates (expensive, but few)
  5. fuse scores across multiple ABIS engines (vendor A,B,C)
  6. if fused_score > dup_threshold -> DUPLICATE (-> manual review)
     else -> UNIQUE -> allocate number, append templates to gallery
Cost per probe: ~millions of cheap iris compares + a few fp confirms,
  not 1.4B fp compares. Orders of magnitude reduction.

Why this is right:

  • It replaces an impossible full scan with a tractable filtered scan. Iris Hamming distance plus binning turns 1.4B comparisons per probe into a small bounded candidate set, making national dedup actually computable on real hardware.
  • It uses the strongest modality where it matters. Iris leads (discriminative, cheap 1:N); fingerprint confirms (where its cost is paid only on a few candidates). The weak-at-scale operation is never run against the whole gallery.
  • Multi-ABIS crushes the error tails. Fusing independent engines drives false-accept and false-reject far below any single algorithm, and manual adjudication guards the one outcome you must never automate wrongly: denying a real person an identity.
  • Async + horizontal scale makes latency a non-issue. Dedup is sized for throughput; the resident is not blocked. The queue absorbs bursts; the cluster grows linearly with enrollment volume.

The principle: uniqueness at a billion-scale is won by shrinking the comparison set before comparing (filter with the cheap, discriminative signal first) and by fusing multiple independent matchers offline, never by brute-forcing the gallery on the hot path.

2. Authentication: 50M 1:1 checks a day under 2 seconds

The job: given an Aadhaar number (or Virtual ID) plus a sample, return yes/no fast, reliably, at ~3K/sec peak, without ever exposing the stored biometric.

Naive approach: one central auth server reading from the main biometric database

Point every agency at a central auth service that, per request, reads the person’s biometric straight from the main store, runs the match, and returns the result.

Where it breaks:

  • The 2-second budget is blown by a cold, central read. Fetching a full biometric record from a petabyte-scale store per request, over a shared link, adds unpredictable latency and tail spikes. A person at a counter cannot wait on a slow disk.
  • A single central service is an availability SPOF. If it is down, the entire country cannot authenticate - no ration, no bank onboarding. For critical national infrastructure this is unacceptable.
  • It couples auth to the vault. Reading raw biometrics on every auth means the most sensitive data is on the hottest, most externally exposed path - the exact opposite of what regulation demands. One compromised auth node could exfiltrate biometrics at scale.
  • No regional locality. Requests from across a large country all funnel to one place; network RTT alone threatens the budget.

A single central reader is slow, fragile, and a security liability - it puts the crown-jewel data on the busiest path.

Evolved approach: sharded template store + hot cache + regional replicas, match-only, yes/no out

Turn auth into a locality-friendly 1:1 lookup over compact templates (never raw images), fronted by cache and replicated regionally.

  • Match on templates, not raw biometrics. The online path stores and reads only the match template (minutiae/iris code, a few KB), never the raw image. Templates are enough to compute a match score and are far smaller, so the read is tiny and fast. The raw image vault is not on the auth path at all.
  • Shard the template store by Aadhaar number. The 1:1 read goes to exactly one shard, determined by hashing the number. 1.4B templates spread across shards means each shard holds a manageable slice, and a lookup is a single-shard point read. No full scan, ever, on auth.
  • Hot cache of active residents. Most auth comes from a working set far smaller than 1.4B (people repeatedly authenticate at ration shops and banks). A distributed in-memory cache holds the templates of active residents; a cache hit is a microsecond match. Even hundreds of GB of hot templates fit across a cache cluster. Misses fall through to the shard and warm the cache.
  • Regional replicas for locality and availability. Replicate the template store and cache across multiple regions/data centers. An agency’s request is served by the nearest region, cutting RTT and giving no-single-region SPOF: if one region fails, traffic shifts to another. Template data is read-mostly (it changes only on enrollment/update), so replication is cheap and staleness of a few seconds is harmless for auth.
  • OTP as a parallel, cheaper modality. Not every auth needs biometrics. An OTP to the registered mobile is a simple, fast factor: the gateway generates a time-boxed code, sends it via SMS, and verifies it - no biometric read at all. This offloads a large share of auth from the matcher and serves cases where biometric capture is awkward.
  • Yes/no out, audited. The response is a boolean (plus eKYC payload only with consent). Every request writes an append-only audit entry. The matcher runs inside a protected boundary; the decrypted sample lives only transiently in memory and is never persisted.
AUTH 1:1 (online, <2s):
  req = { aadhaar|VID, encrypted_sample|otp, agency_id, consent }
  1. gateway: authN agency (mTLS), rate-limit, VID->aadhaar, consent check
  2. shard = hash(aadhaar) % N ; region = nearest
  3. template = cache.get(aadhaar) or template_store[shard].get(aadhaar)
  4. if biometric: score = match(decrypt(sample), template); ok = score>TH
     if otp:       ok = verify_otp(aadhaar, code)
  5. audit.append(agency_id, aadhaar, ts, ok)   # no biometric stored
  6. return { ok, ekyc? }                        # yes/no only
Latency budget: cache hit match ~ single-digit ms; miss + shard read ~tens ms;
  network dominates - regional replicas keep it well under 2s.

Why this is right:

  • It fits the 2-second budget with headroom. A sharded point read of a small template, usually from an in-region cache, plus one match, is milliseconds of work; the 2 seconds is mostly network and capture, not the core.
  • It has no single point of failure. Sharding spreads load; regional replicas mean any one region or shard failing degrades gracefully instead of taking the nation offline.
  • It keeps biometrics off the hot, exposed path. Auth touches only templates, never the raw vault, and returns only yes/no - minimizing what a compromised node can leak.
  • It scales linearly and cheaply. 3K/sec over a sharded, cached, read-mostly store is comfortable; growth is handled by adding shards/replicas, and OTP siphons off load that never hits the matcher.

The move: make the online path a small, sharded, cached, replicated 1:1 template lookup, isolate it from the heavy vault, and return the minimum - a boolean - so auth is fast, available, and tight.

3. The biometric vault: storing the irrevocable

The job: hold 1.4 billion people’s raw biometrics (multi-petabyte) such that a breach is extraordinarily hard, access is minimal and audited, and no one can build a surveillance graph from the system’s usage.

Naive approach: one big encrypted table, biometrics beside demographics, read on demand

Store raw fingerprints, irises, demographics, and the Aadhaar number together in one database, encrypt the disk, and let the auth and enrollment services read whatever they need.

Where it breaks:

  • Raw biometrics on a queryable, networked store is a catastrophe waiting. A fingerprint cannot be reissued. If this store is reachable from the online auth path, a single compromised credential or SQL-injection-equivalent can exfiltrate irrevocable biometrics for millions. Disk encryption does not help once an authenticated service reads it.
  • Colocating biometrics with demographics enables trivial correlation. One breach yields name + address + fingerprint together - the worst possible bundle. And storing the real Aadhaar number everywhere lets any leak be joined across datasets.
  • “Read on demand” by the auth service means the vault is hot and exposed. The most sensitive data sits on the busiest external-facing path, maximizing both attack surface and blast radius.
  • No defense against internal misuse or usage tracking. If every service can read the vault and nothing isolates request metadata, the system itself becomes a record of who authenticated where - a surveillance database by accident.

Putting the irrevocable data in a normal, reachable, colocated store means a single failure is permanent and total.

Evolved approach: isolated encrypted vault, template/vault split, tokenization, least-privilege + audit

Treat the raw biometrics as a toxic asset to be quarantined, and let only a tiny, offline, heavily guarded interface touch them.

  • Physically and logically isolate the vault. Raw biometric images live in a network-isolated, encrypted store with no route from the public internet and no route from the online auth service. Only the ABIS/enrollment pipeline (offline) and tightly-scoped update jobs can reach it. The auth path cannot, by network policy, read a raw biometric - it only ever sees templates.
  • Split templates from raw images. The template store (used for matching) is separate from the raw image vault (used for re-enrollment/quality/legal). Auth reads templates; the vault is cold. A compromise of the auth tier yields templates at worst (still sensitive, but far smaller surface) and never the raw images.
  • Encrypt per record with managed keys / HSMs. Each record (or shard) is encrypted with keys held in hardware security modules; the matcher decrypts a sample only transiently inside a protected boundary and never writes plaintext biometrics to disk. Keys are rotated and access-gated so that even DB admins cannot read biometrics without the HSM-guarded path.
  • Tokenize the identifier - Virtual ID and reference keys. Agencies use a Virtual ID (VID), a revocable token mapping to the real Aadhaar number, so downstream systems never store the real number and cannot correlate a person across services by it. If an agency’s data leaks, it leaks VIDs, which can be revoked/rotated - unlike the number itself.
  • Least privilege, full audit, no central tracking. Every access to any identity data is logged append-only in a tamper-evident audit store (who, when, which record, why) - but the system is designed so that auth requests are not aggregated into a cross-service profile: the agency learns only yes/no, the central system does not expose a queryable “where did person X authenticate” view to anyone outside strict audit controls. Access is role-scoped: an auth node can match, it cannot browse; an operator can enroll, they cannot bulk-export.
Data separation:
  VAULT (raw biometrics, PB) ── encrypted, HSM keys, network-isolated,
     reachable ONLY by offline ABIS/enrollment + legal/update jobs.
  TEMPLATE STORE (few KB/person) ── sharded, replicated, read by AUTH.
  DEMOGRAPHIC STORE ── SQL, read for eKYC on consent.
  VID MAP ── revocable token -> Aadhaar number (agencies hold VIDs only).
  AUDIT ── append-only, tamper-evident, every access; not a tracking graph.
Rule: AUTH path touches TEMPLATE + DEMO (on consent) only. Never the VAULT.
      Agencies touch VID + yes/no + consented eKYC only. Never a biometric.

Why this is right:

  • The irrevocable data is off every hot and external path. The vault is cold and isolated; a breach of the busy auth tier cannot reach raw biometrics at all. Blast radius is minimized by construction.
  • Tokenization kills cross-service correlation. VIDs mean a leak in one agency does not expose the real number or let datasets be joined into a surveillance profile; revoking a VID contains the damage.
  • Defense in depth, not disk encryption alone. Isolation + per-record HSM-held keys + least-privilege roles + transient-only decryption means multiple independent failures are needed to expose biometrics.
  • It satisfies the regulator’s real fear. The design proves the system cannot easily become either a breach source for irrevocable data or a tracking database - the two outcomes regulation most wants prevented.

The principle for irrevocable data: store it as if you will be breached - isolate it, split it from everything that identifies a person, encrypt it with keys you control, keep it off every online path, tokenize the identifier, and audit every touch.

4. Allocating a unique 12-digit Aadhaar number

The job: when dedup clears a new resident, mint a unique 12-digit number with no collisions, resistant to trivial enumeration, at enrollment rate across many enrollment servers.

Naive approach: a single auto-increment sequence

Use one central counter: next number = last + 1, handed out sequentially.

Where it breaks:

  • Single counter is a SPOF and a bottleneck. Every enrollment across the country serializing on one sequence node means its outage halts all number issuance, and it caps throughput.
  • Perfectly sequential numbers are trivially enumerable. If numbers increment by one, anyone can guess valid neighbors, aiding fraud and scraping. A national ID number should not be guessable from another.
  • Races on “last + 1” across distributed enrollment servers can mint duplicates if two read the same last value - and a duplicate national ID is a correctness disaster.

A lone incrementing counter is fragile, guessable, and race-prone.

Evolved approach: partitioned issuance, check digit, non-sequential, coordination-free

Partition the number space, mint locally from disjoint blocks, and bake in validation.

  • Partition the 12-digit space into disjoint ranges per issuing zone/server. Each enrollment server (or region) is handed a block of candidate numbers; it mints from its block with a local counter and requests a new block when low. Blocks never overlap, so two servers can never mint the same number - no global lock on the hot path, coordination only once per block.
  • Build in a check digit (Verhoeff). The last digit is a checksum over the first eleven, so typos and random guesses are caught instantly without a database lookup. Real Aadhaar numbers use a Verhoeff check digit for exactly this. This also means a valid-looking number is not just any 12 digits - the space of valid numbers is thinned.
  • Avoid pure sequentiality. Numbers are drawn from many partitions in parallel and need not be issued in strict global order, so consecutive real enrollments do not get adjacent numbers. For stronger unguessability the issued number can be a formatted mapping over the raw block index, but the real protection against misuse is authorization on every lookup (you must auth as the person or be an entitled agency), not number secrecy.
  • Durable block hand-out. Block assignments are persisted, so a crashed server’s unused block is simply retired (a small gap), never reissued - uniqueness survives failures.
ID service: space partitioned into blocks B0,B1,...  (12-digit candidates)
Enrollment server (low-water): request a block -> durably marked consumed
Per new resident (after dedup clears):
   raw = next_in_block()                 # local, lock-free
   n11 = format11(partition_prefix, raw) # 11 digits
   number = n11 + verhoeff_check(n11)    # 12th digit = checksum
Uniqueness: disjoint blocks -> no collision, no global lock.
Validation: check digit rejects typos/guesses without a DB hit.

Why this is right:

  • No SPOF, no hot-path coordination. Local minting from pre-allocated disjoint blocks means any server with a block keeps issuing numbers even if the central ID service blips; coordination is once per block, not per number.
  • Collisions are impossible by construction. Disjoint blocks guarantee uniqueness without distributed locking.
  • The check digit adds cheap integrity. Verhoeff catches entry errors and random guesses instantly, improving data quality and reducing invalid-number traffic.
  • It is not naively enumerable and, crucially, does not rely on secrecy - real protection is that a number alone does nothing without auth or agency entitlement.

Same move as everywhere in this design: turn a global uniqueness requirement into local, coordination-free generation over a partitioned space, and add a checksum so correctness is self-verifying.

API Design & Data Schema

External APIs

HTTPS, mutual TLS, licensed agencies only. Biometric samples are encrypted on the capture device with the system’s public key; the agency relays, it does not decrypt.

POST /v1/auth
  headers: { Agency-Id, Agency-Signature }        # mTLS + signed request
  body: {
    "uid_or_vid": "VID-91823...",                 # Virtual ID preferred
    "auth_type": "FINGERPRINT|IRIS|OTP",
    "encrypted_bio": "base64(...)",               # present for bio types
    "otp": "483920",                              # present for OTP
    "consent": true, "timestamp": "2026-11-11T09:14:02Z",
    "txn_id": "au-77c1f..."
  }
  -> 200 { "txn_id":"au-77c1f...", "result": true,  "score_band":"HIGH" }
  -> 200 { "txn_id":"au-77c1f...", "result": false, "err":"BIO_MISMATCH" }
  // never returns the stored biometric

POST /v1/ekyc                                      # demographics on consent
  body: { "uid_or_vid":"VID-...", "auth_type":"OTP", "otp":"...",
          "consent":true, "txn_id":"ek-22a..." }
  -> 200 { "name":"...", "dob":"1991-04-02", "gender":"F",
           "address":"...", "photo":"base64(jpg)", "txn_id":"ek-22a..." }

POST /v1/otp/generate                              # send OTP to registered mobile
  body: { "uid_or_vid":"VID-...", "txn_id":"ot-10b..." }
  -> 200 { "sent": true, "masked_mobile":"XXXXXX4821" }

POST /v1/vid/generate                              # resident rotates their token
  body: { "uid":"...", "auth":{...} }
  -> 200 { "vid":"VID-91823...", "valid_until":"2027-11-11" }

POST /v1/enroll                                    # center -> async pipeline
  body: { "demographics":{...}, "encrypted_biometrics":"...",
          "operator_id":"...", "center_id":"...", "packet_sig":"..." }
  -> 202 { "enroll_ref":"en-5521...", "status":"QUEUED" }   # number issued later

GET  /v1/enroll/{enroll_ref}/status
  -> 200 { "status":"QUEUED|VALIDATING|DEDUP|ISSUED|DUPLICATE|REJECTED",
           "uid": "xxxx-xxxx-xxxx" }               # uid only when ISSUED

Data model and store choices

1. Demographic store - sharded SQL (Postgres), source of truth for identity fields. Relational: a person record with structured fields, updates, and audit needs ACID and queryability. Sharded by Aadhaar number.

-- shard by hash(aadhaar)
CREATE TABLE identity (
  aadhaar       CHAR(12) PRIMARY KEY,          -- 11 digits + Verhoeff check
  name          VARCHAR(120) NOT NULL,
  dob           DATE         NOT NULL,
  gender        CHAR(1)      NOT NULL,
  address       TEXT         NOT NULL,
  mobile_hash   CHAR(64),                       -- for OTP routing, hashed
  photo_ref     VARCHAR(80),                    -- pointer into vault
  status        VARCHAR(12)  NOT NULL,          -- ACTIVE|SUSPENDED|DECEASED
  created_at    TIMESTAMPTZ  NOT NULL DEFAULT now(),
  updated_at    TIMESTAMPTZ  NOT NULL DEFAULT now()
);
CREATE INDEX ON identity (mobile_hash);          -- OTP / update lookups
-- demographic dedup blocking (soft): name-phonetic + dob band, used by ABIS
CREATE INDEX ON identity (dob, gender);

2. Template store - sharded key-value, read by auth. Compact match templates, sharded by Aadhaar number, replicated across regions, fronted by cache. Strongly keyed 1:1 reads, read-mostly.

KEY  tmpl:{aadhaar}
  fp_templates   bytes[10]   # minutiae templates per finger
  iris_codes     bytes[2]    # ~512 B iris codes
  face_template  bytes       # ~2 KB
  version        int         # bumped on biometric update
  Shard: hash(aadhaar) % N. Replicas: per region. Cache: hot active set.

3. Biometric vault - isolated encrypted object store, raw images (PB). Not relational, not on the auth path. Per-record encryption, HSM-held keys, network-isolated. Reached only by ABIS/enrollment and legal/update jobs.

vault/{aadhaar}/fp_{1..10}.wsq      # raw fingerprint images, encrypted
vault/{aadhaar}/iris_{l,r}.jp2      # raw iris images, encrypted
vault/{aadhaar}/face.jpg            # raw photo, encrypted
  encryption: per-object key wrapped by HSM master key; audited access only.

4. ABIS gallery - sharded, binned template index for 1:N. Templates organized by bins (fp pattern class, iris high-order bits, demographic block) across many matcher nodes; iris-first, fingerprint-confirm; multi-vendor.

gallery shard -> { bin_id -> [ {aadhaar, iris_code, fp_minutiae}, ... ] }
  probe fans out to bins it falls in; parallel match; fuse multi-ABIS scores.

5. VID map - revocable token store (KV). vid -> aadhaar, with expiry/rotation. Agencies hold VIDs only.

6. Audit log - append-only, tamper-evident. Every auth/eKYC/enroll access: who, when, which record, result - never the biometric. Not exposed as a cross-service tracking query.

Why this split: the uniqueness/dedup workload needs a binned, sharded, multi-vendor matcher gallery (offline, compute-sized), so it is its own cluster. The auth workload needs small templates, sharded and cached and regionally replicated for sub-2s 1:1 reads, so templates live apart from raw images. The irrevocable raw biometrics need a cold, isolated, encrypted vault, so they are off every online path. The identity fields need ACID SQL for updates and eKYC. Match each store to its access pattern and its threat model, rather than one database holding biometrics on a hot public path - which would fail at latency, availability, and security at once.

Bottlenecks & Scaling

Where it breaks first, in order, and the fix for each.

1. Deduplication compute - breaks first and hardest. The 1.4-quadrillion-comparison brute force is impossible. Fix: iris-first filtering + binning to shrink each probe’s candidate set by orders of magnitude, fingerprint only to confirm survivors, multi-ABIS fusion for accuracy, and run it all as a massively parallel async batch sized for throughput. Add matcher nodes linearly with enrollment volume. The queue absorbs bursts; residents are never blocked.

2. Auth latency tail under the 2-second budget. A cold central read would blow it. Fix: sharded template store (1:1 point read by Aadhaar hash), hot in-memory cache of active residents, and regional replicas so the read is near the agency. Match on small templates, not raw images. OTP offloads a large share with no biometric read at all. The budget is spent on network/capture, not the core.

3. Hot shard / hot region on auth. Popular regions (dense cities, big banks) concentrate load. Fix: shard templates by Aadhaar number hash (uniform spread, not geographic skew), replicate each region’s working set, and scale replicas where traffic concentrates. Cache absorbs repeat authenticators (same person at the ration shop daily).

4. The biometric vault as attack surface. The irrevocable data is the highest-value target. Fix: network-isolate the vault off the auth path, split templates from raw images, per-record HSM-held encryption, least-privilege roles, and transient-only decryption. Auth can never reach raw biometrics by construction, so a hot-tier breach cannot exfiltrate them.

5. Cross-service tracking / privacy failure. The system could accidentally become a surveillance graph. Fix: Virtual IDs (revocable tokens) so agencies never hold the real number and cannot correlate a person across services; yes/no-only responses; audit that logs access without exposing a queryable “where did X authenticate” view outside strict controls.

6. Number issuance as a global counter. A single sequence would bottleneck and SPOF. Fix: partitioned disjoint blocks minted locally with a Verhoeff check digit - coordination once per block, collisions impossible, self-validating numbers, no central hot path.

7. Availability of auth as critical infrastructure. An outage strands citizens from food and banking. Fix: stateless auth gateways behind load balancers, multi-region active-active template/cache replicas with automatic failover, and graceful degradation (fall back to OTP if a biometric region is impaired). No single region or shard failing takes the nation offline.

8. Update propagation (address change, child biometric refresh). A stale template would reject a genuine person. Fix: writes go to the demographic/template source of truth and version-bump the template; replicas and cache invalidate on the version change. Eventual consistency of a few seconds is acceptable for auth; the version prevents matching against an outdated template after an update.

9. Storage growth (multi-PB vault, 8 TB templates, ever-growing audit). Nothing can be purged - it is lifetime identity and legally audited. Fix: the vault is cold object storage with tiering; templates stay on fast sharded stores (only a few KB/person); audit logs roll to append-only cold storage with tamper-evidence. Capacity grows with population, which grows slowly and predictably.

Shard keys, stated plainly: the template store, demographic store, and VID map shard by Aadhaar number hash - uniform spread and single-shard 1:1 auth reads. The ABIS gallery shards and bins by biometric features + demographic blocks - the unit of a dedup probe’s comparison set. The vault is keyed by Aadhaar number but isolated and cold. Auth gateways are stateless (scale for load, no sharding). Never put raw biometrics on a store the auth path can read, and never run dedup against the full gallery - both are the cardinal sins this design exists to avoid.

Wrap-Up

The trade-offs that define this design:

  • Enrollment and authentication are separate systems sharing a data core. Dedup is an offline, compute-sized, async batch that can take hours and must be perfectly correct; auth is an online, cached, sharded 1:1 check that must be fast and always up. We trade a single unified service for two subsystems tuned to opposite constraints - and that separation is what makes both tractable.
  • Shrink the comparison, do not brute-force it. Billion-scale uniqueness is won by iris-first filtering plus binning plus multi-ABIS fusion, turning 1.4 quadrillion comparisons into a bounded candidate scan. We trade algorithmic simplicity for feasibility and accuracy.
  • Store the irrevocable as if you will be breached. Raw biometrics live in an isolated, encrypted, cold vault off every online path; auth touches only small templates and returns only yes/no. We trade the convenience of one colocated database for a blast radius small enough to survive a compromise.
  • Tokenize identity to prevent tracking. Virtual IDs and yes/no responses keep the system from becoming a surveillance graph, satisfying the regulator’s deepest fear. We trade a little indirection for un-correlatability across services.
  • Fast, available, stale-tolerant reads; strongly-consistent uniqueness and issuance. Auth tolerates seconds-stale replicas; the uniqueness check and number allocation are correctness-absolute. We put strong consistency only where identity integrity demands it, and let the hot read path be eventually consistent and cached.

One-line summary: a national identity system splits into an offline, massively parallel ABIS pipeline that deduplicates every new enrollment against 1.4 billion people using iris-first filtering, binning, and multi-vendor fusion before minting a unique Verhoeff-checked 12-digit number from partitioned blocks, and an online, sharded, cached, regionally replicated 1:1 authentication path that matches a compact template in milliseconds and returns a bare yes/no in under two seconds - with the irrevocable raw biometrics quarantined in an isolated, encrypted vault that no auth request can ever reach, and Virtual IDs keeping the whole system from becoming a tracking graph.