“Let users log in” is the request that hides an entire distributed system. You need to store a password without ever storing the password, verify it in a way that is slow for attackers but fast enough at 100K logins a second, hand out a credential the user carries around, revoke that credential the instant an account is compromised, survive a leaked database dump, layer on a second factor, and let people sign in with Google without you ever seeing their Google password. Every one of those is a place the system breaks, and most of them are places where breaking means a breach, not a slow page.

The core tension: authentication is a read-mostly, latency-sensitive, brutally security-critical path. It gates every other request in the product, so it has to be fast and always up. But it is also the single juiciest target in the company - the place where being clever and being wrong gets 500M credentials dumped on a forum. So the design is a constant negotiation between throughput and paranoia. Let me build it properly.

Functional Requirements (FR)

In scope:

  • Signup. A user registers with an email (or phone) and a password. We verify the email, reject weak or breached passwords, and create the account. The email must be unique.
  • Login. A user presents credentials, we verify them, and issue a session credential (token) they use for subsequent requests. Failed attempts are rate-limited and locked out.
  • Session management. Issue, validate on every request, refresh, and - critically - revoke sessions (logout, logout-everywhere, forced revocation on password change or compromise).
  • Password reset. A user who forgot their password proves control of their email via a one-time, expiring, single-use token and sets a new password. Reset must invalidate existing sessions.
  • Multi-factor authentication (MFA). After the password step, optionally require a second factor: TOTP (authenticator app), SMS/email OTP, or WebAuthn/passkey. Support enrollment, verification, and recovery codes.
  • OAuth / social login. Sign in with Google/Apple/GitHub as an identity provider. Link a social identity to a local account, or create one. We act as an OAuth client, and optionally as an OIDC provider for our own first-party apps.

Explicitly out of scope (say it so you own the scope):

  • Authorization - roles, permissions, what a user is allowed to do once authenticated. This system answers “who are you,” not “what can you touch.” I mention where the token carries identity claims, but RBAC/ABAC is a separate system.
  • The full fraud/risk engine (device fingerprinting, impossible-travel scoring, bot detection). I design the hooks where risk signals plug into login, not the ML that produces them.
  • User profile management beyond the credential (display name, avatar, settings). Not an auth concern.
  • Enterprise SSO / SAML federation as a provider for third-party B2B tenants. I cover OAuth/OIDC; SAML is the same shape with uglier XML.

The decision that drives everything: a password is a liability we are forced to hold, so we minimize what we store, slow down verification deliberately, and make every credential we issue revocable. We never store a password, we store a slow one-way hash of it. We never trust a token we cannot kill. That defensive posture, not raw throughput, is the shape of this system.

Non-Functional Requirements (NFR)

  • Scale: 500M registered users, ~150M DAU. Peak 100K logins/sec (Monday morning login storm, a regional surge, or a botnet credential-stuffing run - the peak is not always legitimate). Token validation is far higher: every authenticated API request checks a token, easily 2-5M/sec across the product.
  • Latency: login end-to-end p99 under ~300ms including the deliberately slow password hash (the hash alone is ~100ms of that budget by design). Token validation must be p99 under ~5ms - it is on every single request, so it has to be almost free. Signup can be slightly slower (email send is async).
  • Availability: 99.99%+. Auth is the front door; if it is down, the entire product is down for everyone, including users already “logged in” if validation depends on it. This is the most availability-critical service in the company.
  • Consistency: mostly eventual, with hard exceptions. A new session appearing on a read replica a few ms late is fine. A revoked session that still validates is a security hole - revocation must be strongly and quickly consistent. Email-uniqueness at signup must be strongly consistent (no two accounts with the same email).
  • Durability: the credential store is sacred. Losing a user’s password hash locks them out permanently. Full replication, backups, and point-in-time recovery. But the store must also be breach-resilient - durability of a hashed credential is fine, durability of a plaintext anything is a crime.
  • Security posture (the real NFR here): assume the database will leak. Design so a full dump is expensive-to-useless for the attacker (strong per-user-salted slow hashes). Assume the network is hostile. Assume tokens get stolen. Every choice is “what happens when this specific thing is compromised.”

Back-of-the-Envelope Estimation (BoE)

Real numbers with the arithmetic, because they set the architecture.

Login QPS (the write-ish path):

150M DAU, assume ~1.2 logins/user/day (some log in on 2 devices, most
stay logged in via long-lived sessions and rarely re-auth)
150M * 1.2 = 180M logins/day
= 180,000,000 / 86,400 sec
≈ 2,100 logins/sec average
Peak (login storm + credential-stuffing, ~50x avg) ≈ 100,000 logins/sec

The 50x peak is deliberate: a credential-stuffing attack drives login QPS far above organic, and the system must survive it without falling over and without letting it succeed.

Token validation QPS (the read path, the real volume):

Every authenticated API request validates a token.
Assume 150M DAU * ~40 authenticated requests/user/day
= 6B validations/day
= 6,000,000,000 / 86,400 ≈ 70,000/sec average
Peak (bursty, ~30x) ≈ 2,000,000+/sec

Validation is ~20-30x more frequent than login. This is why token validation must be near-free - ideally no database or network hop at all. It dominates the request budget of the whole product.

Password hashing CPU (the expensive part):

A memory-hard hash (Argon2id / bcrypt cost ~12) targets ~100ms and
~tens of MB of memory PER verification, on purpose.
At 100K logins/sec * 100ms CPU = 10,000 CPU-seconds/sec of hashing.
=> ~10,000 dedicated hashing cores at peak, i.e. thousands of machines,
   or a horizontally-scaled hashing tier.

This is the single biggest compute cost in the system, and it is intentional - we are buying attacker-cost with our own CPU. It also means the login tier must scale independently of everything else, and credential-stuffing (which forces real hashes) is a genuine DoS-by-CPU risk that rate limiting must blunt before the hash.

Storage - user credential store:

Per user record:
  user_id          8 bytes (UUID or bigint)
  email            ~40 bytes
  email_norm       ~40 bytes (lowercased, for unique index)
  password_hash    ~100 bytes (Argon2id encoded: algo+salt+params+hash)
  status/flags     ~8 bytes
  mfa_enabled      1 byte
  created_at etc   ~24 bytes
  --------------------------------------------------
  ≈ 260 bytes, round to ~400 bytes with indexes/overhead
500M users * 400 bytes ≈ 200 GB

200 GB of core credential data. Tiny by storage standards, trivially fits a sharded relational cluster. The credential store is small; the challenge is never its size, it is its criticality.

Session store:

Assume server-side sessions (we will justify this choice later).
150M DAU * ~2 active devices = 300M live sessions
Per session: session_id(32) + user_id(8) + device/ip(~60) +
             issued/expiry(16) + flags(8) ≈ ~130 bytes
300M * 130 bytes ≈ 40 GB in a fast store (Redis), TTL-evicted.

~40-50 GB of Redis for live sessions. A handful of nodes. This is the store that gets hit 2M+ times/sec on validation, so it is replicated and read-scaled hard - or bypassed entirely for stateless tokens, the central trade-off of this design.

The headline: the credential store is small (200 GB) but sacred; the compute cost is dominated by deliberately-slow hashing (thousands of cores); and the real traffic is token validation at millions/sec, which is why the session/token strategy is the make-or-break decision. Login itself at 100K/sec is modest QPS but expensive per call and under active attack.

High-Level Design (HLD)

The spine: a stateless API gateway validates a token on every request cheaply (no DB hit in the common case). Unauthenticated requests hit an Auth Service that owns signup, login, reset, MFA, and OAuth. Credentials live in a sharded relational User/Credential store; hashing runs on a dedicated CPU-heavy tier; sessions/tokens live in a fast replicated store; and a revocation mechanism ties tokens back to a kill-switch so stolen or logged-out credentials die quickly.

                        ┌──────────────────────────────┐
                        │           Clients            │
                        │  (web, mobile, 3rd-party)    │
                        └───────────────┬──────────────┘
                                        │ every request carries a token
                        ┌───────────────▼──────────────┐
                        │        API Gateway / Edge     │
                        │  validate token (fast path),  │
                        │  check revocation, rate-limit │
                        └──────┬─────────────────┬──────┘
              token invalid /  │                 │ token valid
              login / signup   │                 └──────────▶ downstream product services
                        ┌──────▼───────────────────────┐
                        │        Auth Service           │
                        │  signup / login / reset /     │
                        │  mfa / oauth. Stateless.      │
                        └──┬────┬────────┬────────┬─────┘
                verify pwd │    │ issue  │ 2nd    │ oauth
                  ┌────────▼─┐  │ token  │ factor │ code exchange
                  │ Hashing  │  │        │        │
                  │  Tier    │  │  ┌─────▼─────┐  │  ┌────────────────┐
                  │(Argon2id,│  │  │ MFA Svc   │  │  │ External IdP    │
                  │ CPU-heavy)│ │  │ TOTP/OTP/ │  │  │ (Google/Apple/  │
                  └────┬─────┘  │  │ WebAuthn  │  │  │  GitHub)        │
                       │        │  └─────┬─────┘  │  └────────────────┘
        ┌──────────────▼──┐  ┌──▼────────▼──┐  ┌──▼───────────────┐
        │ User / Credential│  │ Session /    │  │ Revocation List  │
        │ Store (sharded   │  │ Token Store  │  │ (blocklist,      │
        │ SQL, salted hash)│  │ (Redis)      │  │  short-TTL Redis)│
        └──────────────────┘  └──────────────┘  └──────────────────┘
                       │                │
                ┌──────▼────────────────▼──────────────────────────┐
                │  Async: email/SMS sender (verify, reset, OTP),    │
                │  audit log (all auth events), risk/fraud signals  │
                └───────────────────────────────────────────────────┘

Login flow (the money path):

  1. Client POST /login with {email, password} to the Auth Service (through the gateway, which rate-limits per IP/account before any expensive work).
  2. Auth Service normalizes the email, looks up the user by email_norm in the credential store. If no user, it still runs a dummy hash to keep response time constant (prevent username enumeration by timing).
  3. It sends the presented password + stored salt to the Hashing Tier, which computes the Argon2id hash (~100ms) and compares in constant time. This is the deliberately slow, CPU-heavy step.
  4. On match: if the account has MFA enabled, issue a short-lived MFA challenge token (not a full session) and return “MFA required.” The client then submits the second factor; on success we proceed.
  5. Issue a session: create a session record (or a signed token) and return an access token (short-lived) + refresh token (longer-lived, revocable). Write an audit-log entry.
  6. Subsequent requests carry the access token; the gateway validates it locally (verify signature / cache lookup) and checks the revocation blocklist. No round-trip to the Auth Service on the hot path.

Signup flow:

  1. POST /signup with {email, password}. Validate password strength, check against a breached-password list (k-anonymity range query, do not send the full password). Reject weak/breached.
  2. Insert the user with a unique constraint on email_norm (strong consistency: two concurrent signups on the same email - exactly one wins). Mark email_verified = false.
  3. Generate a single-use, expiring email-verification token, store its hash, and async send the verification email. Return 201 immediately; the email send is off the request path.
  4. User clicks the link, we verify the token, flip email_verified = true.

Password reset flow:

  1. POST /forgot-password {email}. Always return the same generic “if that account exists, we sent a link” response (no enumeration). If the user exists, generate a single-use, short-TTL reset token, store only its hash, async-email the link.
  2. POST /reset-password {token, new_password}. Look up by token hash, check not expired/used, validate the new password, write the new hash, mark the token used, and revoke all existing sessions for that user (a reset is a “I may have been compromised” signal).

The key structural insight: validation is separated from issuance. Issuing a credential (login) is rare, expensive, and centralized in the Auth Service. Validating a credential is constant, must be near-free, and is pushed out to the edge/gateway. The entire session-vs-token debate is really a debate about how to make validation cheap without losing the ability to revoke.

Component Deep Dive

The hard parts, naive-first then evolved: (1) password storage & verification, (2) session vs stateless token and revocation, (3) MFA, (4) OAuth/social login, (5) surviving credential stuffing.

1. Password storage and verification

The job: store something that lets us verify a password later, such that a full database leak does not hand the attacker everyone’s password.

Naive approach: store the password (or a fast hash of it)

The first cut stores the password directly, or - one step better and still wrong - a fast hash like MD5 or SHA-256 of it.

# plaintext (catastrophic)
INSERT INTO users(email, password) VALUES(?, ?)

# fast hash (still catastrophic)
INSERT INTO users(email, password_hash) VALUES(?, SHA256(?))

Where it breaks:

  • Plaintext: the database leak is the breach. Every password, reusable on every other site the user has (password reuse is universal). Game over instantly.
  • Fast unsalted hash (MD5/SHA-256): SHA-256 is designed to be fast, which is exactly wrong here. A modern GPU computes billions of SHA-256/sec. An attacker with the dump brute-forces common passwords in seconds and uses rainbow tables (precomputed hash->password maps) because there is no salt. Two users with the same password get the same hash, so cracking one cracks both, and you can see how many people share a password.
  • No per-user salt means one precomputation attacks the whole table at once.

Fast hashes lose because our entire threat model is “the dump leaked” - and against an offline attacker with the dump, speed is the enemy.

Evolved approach: per-user salt + a slow, memory-hard hash, with constant-time compare and pepper

Use a deliberately slow, memory-hard password hashing function - Argon2id (preferred), or scrypt/bcrypt - with a unique random salt per user stored alongside the hash, tuned so one verification costs ~100ms and tens of MB of RAM.

# signup
salt = csprng(16 bytes)
hash = Argon2id(password, salt, mem=64MB, iterations=3, parallelism=4)
store: algo + params + salt + hash   (one encoded string, ~100 bytes)

# login
stored = load(user).password_hash          # encodes salt+params
computed = Argon2id(password, stored.salt, stored.params)
if constant_time_eq(computed, stored.hash): OK

Why each piece matters:

  • Per-user salt kills rainbow tables and means identical passwords produce different hashes. The attacker must attack each user separately; no shared precomputation.
  • Slow + memory-hard is the core defense. Argon2id needs tens of MB per guess, which defeats GPU/ASIC parallelism (they have limited fast memory per core). At ~100ms/guess, brute-forcing even a mediocre password becomes wildly expensive per account. We are literally trading our CPU at login time for the attacker’s cost at crack time.
  • Constant-time comparison prevents a timing side channel that could leak how many leading bytes matched.
  • Dummy hash on unknown user: if the email does not exist, still run a hash against a dummy value so login latency is identical whether or not the account exists - no username enumeration by timing.
  • Pepper (optional, strong): a secret key stored outside the database (in an HSM / KMS), mixed into the hash (e.g. HMAC the password before hashing). Now a database-only leak is useless without also stealing the pepper from the separate secret store. Defense in depth against exactly the leak we assume happens.

Rehashing on login: store the cost parameters with the hash. When you raise the cost (hardware got faster), detect old-parameter hashes on successful login and transparently rehash with the new params. The table upgrades itself as users log in.

The cost is real - this is the thousands-of-cores line in the BoE - but it is the point. We spend our compute to make an offline dump expensive-to-useless, which is the only thing that matters when (not if) the dump leaks.

2. Session management: stateless token vs server-side session, and revocation

The job: after login, hand the user a credential they present on every request, that we can validate in <5ms at 2M/sec, and that we can revoke instantly. Those last two fight each other.

Naive approach A: pure server-side session, DB lookup per request

Store a random opaque session_id in a table; on every request, look it up to get the user.

login:   session_id = random(32); db.insert(session_id -> user_id, expiry)
request: user = db.lookup(session_id)   # every single request

Where it breaks: at 2M validations/sec, a database (or even a single Redis) lookup on every request is a massive, latency-adding, availability-critical dependency. If the session store hiccups, the whole product logs everyone out. It is trivially revocable (delete the row) but the per-request lookup does not scale cheaply.

Naive approach B: pure stateless JWT, no server state

Issue a signed JWT containing {user_id, expiry, claims}. The gateway verifies the signature locally - no lookup at all. Blazing fast, infinitely scalable.

token = JWT.sign({user_id, exp}, secret)
request: JWT.verify(token, secret)   # local, no DB, ~microseconds

Where it breaks: you cannot revoke it. A stateless JWT is valid until it expires, by definition. User logs out? The token still works. Account compromised and you force logout? The stolen token still works until exp. Password changed? Old sessions live on. For a security system, “cannot revoke” is disqualifying. Long-lived JWTs are a loaded gun.

Evolved approach: short-lived stateless access token + revocable refresh token + a thin revocation blocklist

Split the credential into two, and get the best of both:

  • Access token: a short-lived (5-15 min) signed JWT. Validated locally at the gateway - signature check, expiry check, no DB hit. This is the near-free validation that handles the 2M/sec. Because it is short-lived, the revocation window is bounded to minutes even in the worst case.
  • Refresh token: a long-lived (days-weeks), opaque, server-stored, revocable credential. When the access token expires, the client presents the refresh token to the Auth Service, which checks it against the session store (is it still valid? not revoked?) and issues a fresh access token. This is a rare call (once per 5-15 min per device, not once per request), so a DB lookup here is fine.
Access token (JWT, 10 min)  --->  validated locally at gateway, no DB
        expires
Refresh token (opaque, 30d) --->  POST /refresh -> Auth Service checks
                                   session store -> new access token
                                   (revocable: delete session -> refresh fails)

Revocation now works on two levels:

  1. Logout / revoke a device: delete the refresh token / session row. Within one access-token lifetime (≤15 min) that device is fully locked out, because its next refresh fails. For most cases, minutes is acceptable.
  2. Immediate revocation (compromise, forced logout-everywhere, password change): for the cases where 15 minutes is too long, add a thin revocation blocklist the gateway checks. We do not look up every valid session (that is naive approach A again). Instead we keep a small, short-TTL blocklist of revoked token/session IDs (or a per-user min_valid_issued_at timestamp). The gateway checks: “is this token’s session in the blocklist, or issued before this user’s cutoff?” The blocklist only holds recently-revoked entries (TTL = access-token lifetime, after which the token expires anyway), so it stays tiny and cache-resident.
gateway validate(access_token):
  if not JWT.verify_sig_and_exp(token): reject          # local, ~µs
  if blocklist.contains(token.sid): reject              # small Redis/edge cache
  if token.iat < user_cutoff[token.uid]: reject         # optional per-user kill
  accept

Why this is right:

  • The 2M/sec hot path is local signature verification plus a hit against a small revocation cache - not a full session lookup. Cheap and horizontally trivial.
  • Refresh is where state and revocability live, and it is 20-30x rarer, so a real store lookup there is affordable.
  • Revocation is bounded to minutes for free (short access-token TTL) and immediate when needed (blocklist), without paying the per-request-lookup tax for the 99.99% of tokens that are fine.
  • Refresh-token rotation: each refresh issues a new refresh token and invalidates the old one. If a stolen refresh token is used, the legitimate client’s next refresh fails (token already used) - a detectable replay signal that lets us kill the whole session family. This catches refresh-token theft.

The trade-off is explicit: we accept a worst-case revocation delay of one access-token lifetime for the common case (in exchange for a stateless, near-free hot path), and pay for a small blocklist to make true-immediate revocation possible for the rare critical case.

3. Multi-factor authentication (MFA)

The job: after the password, require a second factor so a leaked/guessed password alone is not enough. Support enrollment, verification, and recovery.

Naive approach: SMS OTP as the only factor, checked inline

Generate a 6-digit code, SMS it, store it, compare on submit.

Where it breaks:

  • SMS is weak: SIM-swap attacks, SS7 interception, and it depends on a flaky, costly SMS provider on the login path. It is better than nothing but should not be the only or default factor.
  • Storing the raw code and comparing without rate limits invites brute force - 6 digits is a million combinations, trivially guessable if you allow unlimited attempts.
  • No fallback: phone lost = permanently locked out. No recovery path.
  • Checking inline blocks the login on the SMS provider’s latency and availability.

One weak factor, stored carelessly, with no recovery, is barely MFA.

Evolved approach: TOTP + WebAuthn as primary, SMS as fallback, with a challenge-token state machine and recovery codes

Model MFA as a second step in a login state machine, supporting multiple factor types, defaulting to strong ones:

  • TOTP (authenticator app): on enrollment, generate a random secret, show it as a QR code, store the secret encrypted (KMS-backed). Verification: the app and server both compute TOTP(secret, current_30s_window); accept a small window (±1 step) for clock skew. No network dependency, no SMS cost, resistant to SIM-swap.
  • WebAuthn / passkeys (strongest): public-key based. The authenticator (phone, security key) holds a private key; the server stores only the public key. Login signs a server challenge. Phishing-resistant (the credential is bound to the origin) and nothing secret is stored server-side to leak. This is the direction to push users.
  • SMS/email OTP (fallback only): for users without an authenticator, kept as a lower-assurance option, rate-limited hard.

The flow becomes a two-phase state machine:

1. POST /login {email, password}  -> password OK, MFA enabled
     issue short-lived (2 min) MFA_CHALLENGE token (NOT a session).
     It proves "password step passed, awaiting factor for user X."
2. POST /mfa/verify {challenge_token, factor_type, code|assertion}
     validate the challenge token, verify the factor:
       TOTP: compare computed codes, constant-time, rate-limited
       WebAuthn: verify signature over the challenge with stored pubkey
     on success -> issue real access + refresh tokens.

Key details:

  • The challenge token is not a session. It cannot access anything; it only unlocks the MFA verify step, and it expires in ~2 minutes. This prevents skipping the password step.
  • Rate-limit and lock the OTP verify hard (e.g. 5 attempts then lock the challenge) - a 6-digit code is only safe if guessing is bounded.
  • Recovery codes: at enrollment, generate ~10 single-use backup codes, show them once, store only their hashes. Losing the phone is not a permanent lockout; a recovery code (or a heavier identity-verification flow) restores access.
  • Encrypt TOTP secrets at rest with a KMS key - unlike WebAuthn pubkeys, a TOTP secret is sensitive (it lets you generate codes), so a DB leak of plaintext TOTP secrets defeats MFA.
  • Remember-this-device: optionally issue a long-lived, revocable “device trust” token so a known device is not re-challenged every login, trading a bit of security for UX on trusted devices.

Why this is right: strong phishing-resistant factors are the default, SMS is a clearly-labeled fallback not the foundation, the challenge-token machine makes the two steps unskippable, secrets that matter are encrypted, and recovery codes prevent the lockout that makes users disable MFA entirely.

4. OAuth / social login

The job: let a user “Sign in with Google” - authenticate via an external identity provider - without us ever seeing their Google password, and reconcile that external identity with a local account.

Naive approach: ask for the Google password, or trust an ID token blindly

The truly naive version asks the user for their Google credentials directly (never do this - it is phishing-by-design). The subtly-wrong version: the client logs into Google, gets a token, sends it to us, and we trust it.

Where it breaks:

  • Collecting the provider’s password is catastrophic and defeats the entire point of OAuth.
  • Trusting a client-supplied token without verification: an attacker sends us any Google ID token (or a forged one, or one issued for a different app). If we do not verify the signature, issuer, audience, and expiry, we accept a token that was never meant for us - “token substitution” / confused-deputy. Accepting a valid-Google-token-for-another-app lets an attacker log in as anyone.
  • No CSRF protection on the redirect (state parameter) lets an attacker splice their own auth into the victim’s session.

Trusting the outside world’s token without verifying it was minted for us is the classic OAuth footgun.

Evolved approach: Authorization Code flow with PKCE, server-side code exchange, strict token verification, and identity linking

Use the OAuth 2.0 / OIDC Authorization Code flow with PKCE, doing the sensitive exchange server-side:

1. Client -> our Auth Service: "start Google login"
   We generate `state` (CSRF token) + PKCE `code_verifier`/`code_challenge`,
   redirect the browser to Google's authorize endpoint with our client_id,
   redirect_uri, scope=openid email profile, state, code_challenge.
2. User authenticates WITH GOOGLE (we never see the password).
   Google redirects back to our redirect_uri with an authorization `code`
   and the `state`.
3. We verify `state` matches (CSRF), then our SERVER exchanges the code +
   code_verifier + our client_secret with Google's token endpoint for an
   ID token (JWT) + access token. The secret and exchange stay server-side.
4. We VERIFY the ID token rigorously:
     - signature against Google's published JWKS (rotated keys)
     - iss == accounts.google.com
     - aud == OUR client_id           <-- stops token-substitution
     - exp not passed, iat sane
     - nonce matches what we sent
5. Extract the verified `sub` (Google's stable user id) + email.

Then link the identity to a local account:

Table: federated_identities
  provider     STRING   -- 'google'|'apple'|'github'
  provider_sub STRING   -- stable id from the provider
  user_id      BIGINT   -- our local user
  UNIQUE(provider, provider_sub)

lookup (provider, sub):
  found      -> log in that user_id
  not found  -> if email matches an existing verified account:
                  prompt to LINK (after re-auth) - do not silently merge
                else: create a new local user, mark email_verified from
                      the provider's verified claim, link the identity.

Key details:

  • Verify aud == our client_id - this single check stops the most common OAuth login bypass. A token minted for another app is rejected.
  • PKCE protects the code from interception (mobile/SPA where a client secret cannot be kept), binding the code to the original requester.
  • state is CSRF protection on the redirect.
  • Do the code exchange server-side so client_secret never touches the client.
  • Account linking is deliberate, not automatic by email - silently merging on matching email lets an attacker who controls an unverified email hijack an account. Require the email to be provider-verified and re-authenticate before linking.
  • After verification, we issue our own access/refresh tokens - the Google token is used once to establish identity, then discarded. The user carries our session, not Google’s.

Optionally we act as an OIDC provider for our own first-party apps, issuing ID tokens the same way - same machinery, other direction.

Why this is right: the user’s provider password never reaches us, the sensitive exchange is server-side with the secret protected, every inbound token is verified to have been minted for us and by the real provider, and identity reconciliation is explicit so social login cannot be turned into account takeover.

5. Surviving credential stuffing (the login path under attack)

The job: the 100K/sec peak includes attackers replaying leaked username/password pairs from other sites. Each attempt forces a real ~100ms hash. This is both a security event and a CPU-DoS.

Naive approach: just check credentials and let the hash tier scale

Treat every login the same, verify the password, scale the hashing tier to absorb the load.

Where it breaks:

  • CPU meltdown: attackers can drive login QPS to any level. At 100ms/hash, a stuffing run trivially exhausts the (very expensive) hashing tier, starving legitimate logins - a denial of service that also costs a fortune in compute.
  • Successful takeovers: stuffing works because password reuse is universal. Some fraction of leaked pairs are valid on our site too. Without detection, attackers quietly take over real accounts.
  • A naive per-account lockout after N failures becomes a self-inflicted DoS: attackers spray one attempt across millions of accounts, locking out real users without ever tripping a per-account threshold, or intentionally lock a target out.

Treating hostile and organic login traffic identically means the attacker sets your CPU bill and your takeover rate.

Evolved approach: layered defense that rejects bad traffic before the expensive hash

Stack cheap filters in front of the expensive hash, and add detection:

  • Rate-limit before the hash, at multiple scopes. Per-IP, per-account, and per-IP-subnet token buckets checked at the gateway before touching the hashing tier. Cheap rejection (a Redis counter) protects the expensive resource. This is the CPU-DoS defense.
  • Progressive friction, not binary lockout. Instead of “lock after 5 fails” (which enables lockout-DoS), escalate: after a few failures require a CAPTCHA / proof-of-work, then temporary throttling. Legitimate users solve a CAPTCHA; bots at scale cannot cheaply. This defends without letting attackers lock real users out.
  • Breached-credential and reuse checks. At signup and password change, reject passwords found in known breach corpora (k-anonymity range query so we never send the full password). Fewer reused passwords = fewer stuffing wins.
  • Risk-based / adaptive auth. Score each attempt on device fingerprint, IP reputation, geo-velocity (impossible travel), and known-bad patterns. Low risk: normal login. High risk: force MFA / step-up, or block. A stuffing run from a datacenter IP range hitting thousands of accounts lights up every signal.
  • Detection and containment. Watch the aggregate: a spike in failed logins across many accounts from correlated IPs = stuffing in progress. Trip mitigations (tighten limits, force CAPTCHA globally, block subnets). For accounts that did get a valid login under attack conditions, force a step-up MFA or a password reset.
  • Constant-time on the unknown-user path (the dummy hash from deep dive 1) so attackers cannot even enumerate which usernames exist to refine their list.
login request:
  gateway: per-IP + per-account + subnet rate check   # cheap, pre-hash
      too many -> 429 / CAPTCHA challenge (no hash run)
  risk score (device, IP rep, geo-velocity)
      high -> step-up MFA or block
  ONLY NOW -> hashing tier verifies password (expensive)
  on success under elevated risk -> require MFA anyway

Why this is right: the expensive hash is protected by cheap pre-filters so an attacker cannot set our CPU bill; progressive friction stops bots without handing attackers a lockout weapon; breached-password rejection shrinks the attack surface at the source; and risk scoring plus aggregate detection turn a silent takeover campaign into a visible, containable event.

API Design & Data Schema

API

Core auth endpoints:

POST /api/v1/signup
Body: { "email": "[email protected]", "password": "..." }
  -> 201 { "user_id": "...", "email_verified": false }
  -> 409 email already registered
  -> 422 weak or breached password

POST /api/v1/login
Body: { "email": "[email protected]", "password": "..." }
  -> 200 { "access_token": "...", "refresh_token": "...", "expires_in": 600 }
  -> 200 { "mfa_required": true, "challenge_token": "...", "methods": ["totp","webauthn"] }
  -> 401 invalid credentials (generic, constant-time)
  -> 429 rate limited / CAPTCHA required

POST /api/v1/mfa/verify
Body: { "challenge_token": "...", "method": "totp", "code": "123456" }
  -> 200 { "access_token": "...", "refresh_token": "..." }
  -> 401 bad code   -> 423 challenge locked (too many attempts)

POST /api/v1/token/refresh
Body: { "refresh_token": "..." }
  -> 200 { "access_token": "...", "refresh_token": "<rotated>" }
  -> 401 refresh revoked/expired/replayed

POST /api/v1/logout          Body: { "refresh_token": "..." }   -> 204
POST /api/v1/logout-all      (auth) revoke every session for the user -> 204

POST /api/v1/forgot-password Body: { "email": "..." }
  -> 200 always (generic "if the account exists...")
POST /api/v1/reset-password  Body: { "token": "...", "new_password": "..." }
  -> 200 (also revokes all sessions)  -> 400 invalid/expired/used token

GET  /api/v1/verify-email?token=...       -> 200 verified
POST /api/v1/oauth/{provider}/start       -> 302 redirect to IdP (+state,PKCE)
GET  /api/v1/oauth/{provider}/callback    -> 200 our tokens (after verify+link)

Conventions: login returns 200 with either tokens or an MFA challenge; forgot-password always returns the same body (no enumeration); errors are generic and constant-time on the credential path; tokens are always sent over TLS; refresh tokens are HttpOnly; Secure; SameSite cookies for web clients to keep them out of JS.

Data stores

Different access patterns, different stores. Be explicit.

1. User / Credential Store - relational (PostgreSQL/MySQL), sharded by user_id. We want a hard unique constraint on email_norm (signup correctness demands strong consistency - no duplicate accounts), row-level integrity, and simple point lookups. This is a classic relational job; the data is small (200 GB) and the correctness guarantees matter more than exotic scale.

Table: users
  user_id        BIGINT/UUID  PRIMARY KEY
  email          STRING
  email_norm     STRING       UNIQUE           -- lowercased/normalized; unique index
  password_hash  STRING                        -- Argon2id encoded (algo+params+salt+hash)
  email_verified BOOL
  mfa_enabled    BOOL
  status         STRING       -- active|locked|disabled
  pwd_changed_at TIMESTAMP    -- feeds the "min_valid_issued_at" revocation cutoff
  created_at     TIMESTAMP
  Index: email_norm (unique), user_id (pk)
  Shard by: user_id  (login looks up by email_norm -> keep a global unique index
                      or a secondary email->user_id lookup table also sharded)

Login is by email, but sharding is by user_id. Resolve with a small email_norm -> user_id lookup table (sharded by a hash of email_norm) or a global secondary index, so the email lookup is single-shard, then the user row read is single-shard.

2. Session / Refresh-token Store - Redis (or a fast KV), sharded by session_id/user_id. Refresh-token validation and rotation happen here. Read on refresh (rare-ish), written on login/refresh/logout, TTL-evicted at token expiry.

Key: session:{session_id}
Value: { user_id, refresh_token_hash, device, ip, issued_at, expires_at, family_id }
  TTL: refresh-token lifetime (e.g. 30 days)
Index set: user:{user_id}:sessions -> {session_id...}   (for logout-all)
Shard by: session_id  (validation is single-key); secondary set per user_id.

Store only the hash of the refresh token, never the raw token - same reasoning as passwords, at lower cost.

3. Revocation Blocklist - Redis, short-TTL. The tiny set the gateway checks on the hot path. Holds recently-revoked session_ids (TTL = access-token lifetime, after which the JWT expires anyway) and per-user min_valid_issued_at cutoffs for logout-all / password-change.

Key: revoked:{session_id}   -> 1   TTL = access_token_lifetime (~15m)
Key: user_cutoff:{user_id}  -> issued_at_threshold   (bump on logout-all/pwd change)

4. MFA Store - relational or KV, per user. TOTP secrets (encrypted via KMS), registered WebAuthn public keys, hashed recovery codes.

Table: mfa_factors
  user_id      BIGINT
  type         STRING       -- totp|webauthn|sms
  secret_enc   BYTES        -- TOTP secret, KMS-encrypted (null for webauthn)
  public_key   BYTES        -- WebAuthn credential public key (null for totp)
  cred_id      STRING       -- WebAuthn credential id
  created_at   TIMESTAMP
Table: recovery_codes
  user_id   BIGINT
  code_hash STRING          -- single-use; delete/mark on use

5. Federated Identity Store - relational. UNIQUE(provider, provider_sub) -> user_id, as in deep dive 4.

6. Reset / Verification Tokens - KV, short-TTL, store only the hash. token_hash -> {user_id, purpose, expires_at, used}. Single-use, expiring, never store the raw token.

7. Audit Log - append-only, high-write (Kafka -> cold store). Every auth event (login success/fail, reset, MFA, revocation) for forensics and anomaly detection. Off the hot path, async.

Why relational for credentials/identity and KV/Redis for sessions/tokens: the credential and identity data need hard uniqueness and integrity constraints (unique email, unique federated identity) and are small - relational is the right tool. Sessions, revocation, reset tokens, and rate buckets are high-volume, key-scoped, TTL’d, and need microsecond reads with no cross-entity transactions - Redis/KV territory. We never need a distributed transaction across these; the one place we need strong consistency (email uniqueness, session revocation) we get from a unique index and a small strongly-consistent blocklist respectively.

Bottlenecks & Scaling

Where it breaks first, in order, and the fix for each.

1. Token validation at 2M+/sec (breaks first if done wrong). A per-request lookup against a central store would be a latency tax and a single point of failure that logs everyone out on a hiccup. Fix: short-lived signed JWT access tokens validated locally at the gateway/edge - signature + expiry check, no network hop. The only shared state on the hot path is a tiny revocation blocklist that is cache-resident and replicated to every edge. The 2M/sec path touches no database.

2. Password-hashing CPU under login peak / credential stuffing. The deliberately-slow hash is the biggest compute cost and an attacker-controllable DoS vector. Fix: a dedicated, independently-scaled hashing tier (autoscaled on CPU), and - crucially - cheap rate limiting and CAPTCHA/risk checks that reject bad traffic before the hash runs, so an attacker cannot force unbounded hashing. Legitimate peak is provisioned; attack peak is filtered pre-hash.

3. Credential-store hot shard on login-by-email. Login resolves email->user; a naive layout can concentrate load. Fix: shard the email_norm -> user_id lookup by a hash of the email, and the user row by user_id. Both lookups are single-shard and spread uniformly. Read replicas absorb the read-mostly credential lookups; the primary handles the rare writes (signup, password change).

4. Revocation consistency vs the stateless hot path. A revoked token that still validates is a security hole; a per-request full lookup kills performance. Fix: the two-tier model - short access-token TTL bounds the worst case to minutes for free, and the small short-TTL blocklist (revoked session IDs + per-user cutoff timestamps) gives true-immediate revocation for the critical cases, replicated to the edge. We pay for a tiny always-consistent set instead of a huge one.

5. Session-store hot keys (a viral/celebrity or a login storm). Many logins concentrating on one shard. Fix: shard sessions by session_id (naturally high-cardinality, uniform). The per-user session set (for logout-all) is the only per-user key and it is written rarely. Redis replicated with read replicas; the store is ~40 GB so it fits in memory with room to shard.

6. Email/SMS provider on the login path (reset, OTP). A slow/dead provider must not stall login or signup. Fix: everything provider-bound is async - verification and reset emails go through a queue, not the request path (signup returns 201 immediately). MFA that must be inline (SMS OTP) is a fallback factor behind rate limits, and TOTP/WebAuthn have no provider dependency at all, which is another reason to prefer them.

7. Credential stuffing as CPU-DoS and takeover. Covered in deep dive 5. Fix: layered pre-hash rate limiting (per IP/account/subnet), progressive CAPTCHA/proof-of-work (not binary lockout, to avoid lockout-DoS), breached-password rejection at signup, risk scoring, and aggregate detection with global mitigations.

8. The Auth Service / hashing tier as a single point of failure. If auth is down, the whole product is down. Fix: stateless Auth Service horizontally scaled across multiple availability zones/regions behind load balancers, active-active. The credential store is replicated with automated failover; Redis is replicated. Multi-region so a regional outage does not take down global login. Already-issued short-lived tokens keep validating locally even during a brief Auth Service blip (the hot path does not depend on the Auth Service being up), which contains the blast radius of an outage.

9. Secret and key management. Signing keys for JWTs, the pepper, KMS keys for TOTP secrets - if these leak or cannot rotate, it is a systemic breach. Fix: keys in an HSM/KMS, never in the database or app config. Rotate JWT signing keys with overlapping validity (publish a JWKS with key IDs so old and new tokens both verify during rotation). The pepper lives outside the DB so a DB-only leak cannot use it.

10. Audit-log volume and single points for forensics. Every auth event logged is high write volume. Fix: audit events go to an append-only pipeline (Kafka) off the hot path, then to cold analytical storage. This is where stuffing detection and forensics read from, so it must be durable but it is never on the latency path.

Shard keys, stated plainly: users table -> user_id; email lookup table -> hash(email_norm); sessions -> session_id (+ a per-user_id set for logout-all); revocation blocklist -> replicated tiny set to every edge; rate buckets -> (ip), (user_id), (subnet); audit log partitioned by time in the pipeline. Never shard the credential store by email string range (concentrates common prefixes) - hash it.

Wrap-Up

The trade-offs that define this design:

  • Store a slow, salted, memory-hard hash - never the password - and pepper it from a separate store. We spend real CPU (thousands of cores) to make an offline dump expensive-to-useless, because our threat model assumes the dump will leak. Speed at verification time is the enemy; we buy attacker-cost with our own compute.
  • Short-lived stateless access token + revocable refresh token + a thin blocklist. Validation is near-free and local (handling the 2M/sec that dominates traffic) while revocation still works - bounded to minutes for free via short TTLs, and immediate via a tiny always-consistent blocklist for compromise. We refused both the “DB lookup per request” and the “un-revocable JWT” extremes.
  • MFA defaults to phishing-resistant factors (WebAuthn/TOTP) with SMS as a labeled fallback, gated by an unskippable challenge-token state machine, with encrypted secrets and recovery codes so users are not locked out and do not disable it.
  • OAuth done server-side with strict token verification - Authorization Code + PKCE, verify signature/issuer/aud/nonce so a token minted for another app cannot log anyone in, and deliberate (not email-silent) account linking so social login is not an account-takeover vector.
  • Credential stuffing is filtered before the expensive hash - cheap layered rate limits, progressive CAPTCHA instead of lockout, breached-password rejection, and risk scoring - so an attacker cannot set our CPU bill or quietly take over accounts.

One-line summary: a stateless, edge-validated token layer over a deliberately-slow salted-hash credential store, with two-tier revocable sessions, phishing-resistant MFA behind a challenge state machine, server-side verified OAuth, and pre-hash abuse defenses - built on the assumption that the database will leak and every credential we issue must be killable.