Single Sign-On sounds like a login screen. It is not. It is a token factory that a hundred applications trust, sitting at the center of an organization, deciding on every single request across every single app whether the person holding a bearer string is who they claim to be and whether they are allowed to do the thing they are asking to do. You log in once, at one place, and then Gmail, Docs, Drive, Photos, YouTube, and three hundred internal tools all believe you without ever seeing your password. The magic is not the login. The magic is that one central authority mints credentials the other apps can verify, and can un-mint them the instant something goes wrong.

That is the whole tension. To scale to 100M token validations per second you want the apps to verify tokens locally, offline, without calling back to the identity provider - because a central check on every request is a central bottleneck and a central single point of failure. But if apps verify offline, how do you revoke a token before it expires? Offline verification and instant revocation are in direct opposition, and the entire design is how you buy both. This is not “design a login page.” This is “design the trust root for a fleet.” Let me build it properly.

Functional Requirements (FR)

In scope:

  • Single sign-on. A user authenticates once at the Identity Provider (IdP). Subsequent logins to other Relying Party apps (RPs) happen silently - no re-entering credentials - as long as the central session is alive.
  • Authorization Code flow with PKCE. The standard OAuth 2.0 / OIDC dance: an app redirects the user to the IdP, the user consents, the IdP hands back a short-lived authorization code, the app exchanges it server-side for tokens. PKCE protects public clients (SPAs, mobile).
  • Token issuance. Mint an access token (short-lived, presented to resource servers), an ID token (OIDC, identifies the user to the app), and a refresh token (long-lived, used to get new access tokens without re-login).
  • Token validation. Resource servers must validate an access token on every API call - signature, expiry, issuer, audience, and scopes - at enormous volume.
  • Scopes and consent. Tokens carry scopes (read:email, write:calendar). The user consents to the scopes an app requests. Resource servers enforce them.
  • Refresh and rotation. Access tokens expire fast; refresh tokens let the app get a new one silently. Refresh tokens rotate on use and are revocable.
  • Revocation and logout. Kill a specific token, kill all tokens for a user (compromise, password change), and single-logout across every app when the central session ends.
  • Discovery and key rotation. Publish signing keys via JWKS so resource servers can verify signatures, and rotate those keys without downtime.

Explicitly out of scope (say it so you own the scope):

  • The primary authentication itself - password hashing, MFA, passkeys, the login form. That is a separate system (I designed it elsewhere); here the IdP has a way to authenticate a human and produce a verified user_id. SSO is what happens after that first login: turning one authenticated session into tokens for many apps.
  • Fine-grained authorization / RBAC. Scopes are coarse permissions the token carries. What a user_id is actually allowed to do inside an app (row-level, tenant-level) is the app’s authorization system. I hand it identity plus scopes.
  • SAML. Enterprise SSO via SAML is the same shape with XML instead of JSON and POST-binding instead of redirects. I design OAuth 2.0 / OIDC; SAML maps onto the same session model.
  • User provisioning / SCIM. Creating and syncing user accounts into downstream apps is a separate directory-sync problem.

The one decision that drives everything: the IdP is the single trust root, and it issues self-describing tokens that apps verify offline, backstopped by a fast revocation channel. Every hard problem - scale, revocation, logout, key rotation - falls out of resolving “verify offline” against “revoke instantly.”

Non-Functional Requirements (NFR)

  • Scale: 1B registered users. Peak 100M token validations/sec across all resource servers of a hot app fleet (think every Google API call carrying a token). Actual login / token-issuance events are far rarer - people log in occasionally and then carry a token for hours. So the system is read-heavy by five to six orders of magnitude: issuance is thousands/sec, validation is 100M/sec.
  • Latency: token validation must be effectively free - p99 under 1ms, because it is on the critical path of every API request in every app. Login / token issuance can afford ~200-300ms (it is rare and interactive). Silent refresh p99 under 50ms.
  • Availability: 99.99%+ on issuance, and higher effective availability on validation because validation is offline - the IdP can be down and apps keep working until tokens expire. That decoupling is the point: the trust root being briefly unavailable must not take down the fleet.
  • Consistency: token issuance is eventually consistent (a new session on a replica a few ms late is fine). Revocation must be strongly and quickly consistent - a revoked token that still validates is a security hole. This is the one place I pay for strong consistency.
  • Durability: the signing keys and refresh-token store are sacred. Lose the signing key and you cannot issue tokens the fleet trusts (and worse, a leaked signing key lets an attacker mint valid tokens for anyone - key material is the crown jewel). Refresh tokens are long-lived credentials; losing them logs everyone out, leaking them is a breach.
  • Security posture (the real NFR): the token is a bearer credential - whoever holds it is the user for its lifetime. So tokens are short-lived, bound where possible, always revocable, and signed by keys that live in an HSM/KMS and never touch application memory. Assume tokens get stolen; design so a stolen token is low-blast-radius (short TTL, narrow scopes, revocable).

Back-of-the-Envelope Estimation (BoE)

Real numbers with the arithmetic, because they set the architecture - and the headline is the read/write asymmetry.

Issuance QPS (the rare, interactive path):

1B users, assume ~200M daily active
Each DAU triggers ~1 real login + ~5 silent token refreshes/day
(access tokens live ~1 hour, an active session refreshes several times)
Issuance events = 200M * 6 = 1.2B/day
= 1,200,000,000 / 86,400
≈ 14,000 issuance events/sec average
Peak (~10x, Monday morning login storm) ≈ 140,000/sec

Validation QPS (the real volume, the read path):

Every API call to every app carries a token that must be validated.
A hot fleet (all of Google's APIs) does ~100M API calls/sec at peak.
=> 100,000,000 token validations/sec

That is the number that shapes everything. Validation is ~7,000x more frequent than issuance. A design where every validation calls back to the IdP would need the IdP to serve 100M QPS - impossible and pointless. So validation MUST be local and offline. This single ratio forces self-describing (JWT) tokens.

Storage - refresh tokens and sessions:

Active refresh tokens: ~200M DAU * ~3 devices = 600M live refresh tokens.
Each record: token hash (32B) + user_id (8B) + client_id (8B) +
  scopes (~100B) + issued/expiry (16B) + device meta (~100B) ≈ 300B
600M * 300B ≈ 180 GB for the live refresh-token store.
Plus sessions: 200M central SSO sessions * ~200B ≈ 40 GB.

Small enough to live comfortably in a sharded, replicated store and be heavily cached. The refresh/session store is not the scaling problem - validation is.

Revocation list size (the crux):

We revoke rarely relative to issuance: compromised accounts, logouts,
password changes. Say 0.1% of daily issuance gets explicitly revoked
before natural expiry: 1.2B * 0.001 = 1.2M revocations/day.
Access tokens live ~1 hour, so a revoked access token only matters
for at most its remaining ~1 hour of life.
Revocations live for < 1 hour each => steady-state active revocations
≈ 1.2M/day * (1h / 24h) ≈ 50K entries at any moment.
50K entries * ~40B (jti + expiry) ≈ 2 MB.

The active revocation set is tiny - a couple of megabytes. That is the key insight that makes near-instant revocation of offline-verified tokens feasible: because access tokens are short-lived, the set of “revoked but not yet naturally expired” tokens is small enough to replicate to every resource server and hold in memory. Hold that thought - it is the deep dive.

Bandwidth:

JWKS (public keys) is a few KB, cached for hours by every RP - negligible.
Token issuance responses ~2KB each: 140K/sec * 2KB ≈ 280 MB/sec at peak.
Validation is offline => ~0 bandwidth to the IdP for the 100M/sec path.

The offline-validation design collapses the IdP’s bandwidth from “impossible” to “modest.” That is why it wins.

High-Level Design (HLD)

The system has three roles from OAuth: the Identity Provider (IdP / Authorization Server) that authenticates users and issues tokens, the Relying Party / Client (RP) which is the app the user wants to use, and the Resource Server which holds the API the app calls and validates tokens. The IdP is what we are designing; RPs and resource servers are the fleet that trusts it.

                                 ┌─────────────────────────────────────┐
                                 │        IDENTITY PROVIDER (IdP)       │
                                 │                                     │
  ┌────────┐   1. redirect to    │  ┌─────────────┐   ┌─────────────┐  │
  │        │──── /authorize ────▶│  │  Authorize  │   │   Session   │  │
  │  User  │                     │  │   Endpoint  │◀─▶│    Store    │  │
  │ Browser│◀── login + consent ─│  │  (+ PKCE)   │   │ (SSO cookie)│  │
  │        │                     │  └──────┬──────┘   └─────────────┘  │
  └───┬────┘   2. auth code      │         │ issues code               │
      │        (redirect back)   │         ▼                           │
      │                          │  ┌─────────────┐   ┌─────────────┐  │
      │ 3. code                  │  │    Token    │   │   Signing   │  │
      ▼                          │  │   Endpoint  │◀─▶│  Keys (HSM/ │  │
  ┌────────┐  4. code ──────────▶│  │ (code->tok) │   │  KMS + JWKS)│  │
  │Relying │◀── access+id+refresh│  └──────┬──────┘   └─────────────┘  │
  │Party   │                     │         │                           │
  │(RP app)│                     │  ┌──────▼──────┐   ┌─────────────┐  │
  │backend │  refresh ──────────▶│  │   Refresh / │   │  Revocation │  │
  └───┬────┘◀── new access ──────│  │  Revocation │──▶│   Log (CDC) │  │
      │                          │  │   Endpoint  │   └──────┬──────┘  │
      │ 5. call API              │  └─────────────┘          │         │
      │    with access token     └────────────────────────────┼────────┘
      ▼                                                        │ push revoked jti
  ┌─────────────────────────────────────────────┐             │
  │            RESOURCE SERVERS (fleet)          │◀────────────┘
  │  ┌───────────────┐   ┌──────────────────┐    │
  │  │ Verify JWT    │   │ Local revocation │    │  100M validations/sec,
  │  │ sig (JWKS)    │◀─▶│ bloom + set      │    │  offline, no IdP call
  │  │ + exp + aud   │   │ (in-memory)      │    │
  │  │ + scopes      │   └──────────────────┘    │
  │  └───────────────┘                           │
  └─────────────────────────────────────────────┘

The end-to-end request flow, step by step:

  1. User wants app X. The RP has no valid token, so it redirects the browser to the IdP’s /authorize endpoint with client_id, requested scope, redirect_uri, state (CSRF), and a PKCE code_challenge.
  2. IdP checks the central session. If the browser carries a valid SSO session cookie (set on a previous login), the user is already authenticated - no login form. This is the “single sign-on” moment: the second app never sees a login screen. If there is no session, the IdP shows the login form (delegated to the auth system), authenticates, and creates the SSO session.
  3. Consent. If the user has not previously granted this app these scopes, show a consent screen (“App X wants to read your email”). Record the grant so future logins are silent.
  4. Authorization code. The IdP generates a short-lived (~60s), single-use authorization code bound to the client, redirect_uri, and PKCE challenge, and redirects the browser back to the RP’s redirect_uri with code and state.
  5. Token exchange (back channel). The RP’s backend calls the IdP’s /token endpoint with the code, its client_secret (confidential clients) or the PKCE code_verifier (public clients). The IdP validates the code, verifies PKCE, and returns an access token (JWT, ~5-15 min TTL), an ID token (JWT, identifies the user), and a refresh token (opaque, long-lived).
  6. Calling APIs. The RP calls resource servers with Authorization: Bearer <access_token>. Each resource server validates the JWT locally: check signature against the cached JWKS public key, check exp, iss, aud, and required scope, and check the local revocation set. No call to the IdP. This is the 100M/sec path and it never touches the IdP.
  7. Silent refresh. When the access token expires, the RP backend calls /token with grant_type=refresh_token. The IdP validates the refresh token against its store, rotates it (issues a new one, invalidates the old), and returns a fresh access token - no user interaction.
  8. Revocation. On logout, password change, or compromise, the IdP writes to the revocation log, which streams revoked token IDs to every resource server’s in-memory set within a second or two, and deletes the refresh tokens from the store so refresh fails immediately.

The architectural spine: issuance is centralized and stateful (sessions, refresh tokens, keys), validation is decentralized and stateless (offline JWT verification), and a thin revocation stream bridges the two.

Component Deep Dive

1. Token validation at 100M/sec: opaque vs JWT

Naive approach: opaque tokens with introspection. Issue a random opaque string as the access token. Store its metadata (user, scopes, expiry) in a central database. On every API request, the resource server calls the IdP’s /introspect endpoint (or hits the token DB) to look up the token and get its claims.

This is clean and gives you instant revocation for free - delete the row, the next introspection fails. And at low scale it is genuinely the right call.

Where it breaks: at 100M validations/sec, introspection means the IdP (or its token DB) must serve 100M QPS. That is a central chokepoint of absurd size, it puts the IdP on the critical path of every single API call in the fleet, and it means if the IdP has a bad five seconds, every app in the company returns 401. You could cache introspection results at each resource server, but a cache hit that returns “valid” for a token that was revoked 3 seconds ago is exactly the revocation hole you were trying to avoid - so now caching TTL is a security parameter, and you are back to the offline-verification trade-off, just uglier.

The evolution: self-describing JWT access tokens, verified offline. The access token is a signed JWT carrying sub (user_id), aud (which resource servers it is for), scope, exp, iat, and a unique jti (token ID). It is signed with the IdP’s private key (asymmetric - RS256 or ES256, never a shared HMAC secret, because you do not want every resource server holding a key that can mint tokens; they only get the public key to verify).

Now a resource server validates entirely locally:

1. Parse the JWT, read the `kid` (key id) in the header.
2. Fetch the matching public key from JWKS (cached for hours).
3. Verify the RS256/ES256 signature. (~microseconds, CPU only)
4. Check exp (not expired), iss (trusted IdP), aud (this server).
5. Check the token's scopes cover what this endpoint requires.
6. Check jti against the local revocation set (deep dive #2).

Zero network calls to the IdP. The 100M/sec path is pure local CPU. The IdP can go down and every app keeps serving requests until tokens expire. This is why every large-scale OAuth provider uses JWT access tokens despite the revocation headache.

The price you pay: a JWT is valid until it expires, even if you want it dead sooner. You bought offline verification at the cost of instant revocation. The next deep dive buys revocation back.

The mitigation baked into the design: keep access tokens short-lived (5-15 minutes). A stolen token is only useful for minutes, and the “revoke before natural expiry” window you must actively cover is tiny. Long-lived power lives in the refresh token, which is checked centrally on every refresh.

DimensionOpaque + introspectionSigned JWT (offline)
Validation costNetwork call to IdPLocal CPU, sub-ms
IdP load at 100M/sec100M QPS (fatal)~0
RevocationInstant (delete row)Delayed (until expiry or revoke-list)
IdP outage impactWhole fleet 401sFleet keeps working
Token sizeSmall (~40B)Larger (~800B-1KB)

Verdict: JWT access tokens, short TTL, plus the revocation channel below.

2. Instant revocation of offline-verified tokens

Naive approach: just wait for expiry. If access tokens live 15 minutes, a revoked token is dead in at most 15 minutes. For a lot of use cases that is acceptable. But “your account was compromised, and the attacker keeps full access for 15 more minutes” is not acceptable for a serious IdP, and neither is “you clicked logout but the token still works.” We need revocation in seconds, not minutes.

Naive fix: put every resource server back on a central check. That is introspection again - 100M QPS to the IdP. Rejected for the reasons above.

The evolution: a small, replicated, in-memory revocation set pushed to every resource server. Remember the BoE: because access tokens are short-lived, the set of currently-revoked-but-not-yet-expired tokens is tiny - on the order of tens of thousands of entries, a couple of megabytes. That is small enough to hold in RAM on every resource server and keep continuously updated.

The mechanism:

Revocation happens (logout / compromise / password change):
  1. IdP writes {jti, exp} to the revocation log (an append-only,
     durable, replicated log - Kafka or equivalent).
  2. Every resource server subscribes to the revocation topic.
  3. On each event it adds jti -> exp to an in-memory set + a bloom
     filter front (bloom answers "definitely not revoked" in O(1)
     for the 99.99% common case; only a possible-hit consults the set).
  4. Entries auto-evict when their exp passes (a revoked token that
     naturally expired no longer needs tracking - self-cleaning set).

Validation adds one step: after signature+exp+aud+scope pass,
  check the bloom filter; if maybe-present, check the set; if the
  jti is there, reject with 401. Cost: nanoseconds, in-memory.

Propagation latency is the streaming lag - typically under 1-2 seconds to fan out to the whole fleet. So effective revocation is “seconds,” achieved without any resource server ever calling the IdP on the hot path.

What about the gap during propagation? For the highest-risk revocations (account compromise), the IdP also immediately deletes the user’s refresh tokens, so no new access tokens can be minted, and it can push a user_id + revoke-all-before-timestamp epoch entry so resource servers reject any token for that user issued before the epoch - covering tokens whose individual jti you may not have enumerated. The set therefore holds two kinds of entries: specific jti bans and per-user epoch cutoffs.

Refresh tokens are revoked differently - centrally, and that is fine. Refresh is a rare event (thousands/sec, not 100M/sec) and always calls the IdP’s /token endpoint. So refresh tokens are opaque and checked against the central store on every use - delete the row and refresh fails instantly, no streaming needed. This is the elegant split: the high-volume access token is a self-verifying JWT with a streamed revocation set; the low-volume refresh token is opaque and centrally checked. Each token type uses the strategy that fits its access pattern.

3. Refresh token rotation and replay detection

Naive approach: a long-lived refresh token you reuse forever. The RP stores one refresh token and re-uses it for months to mint access tokens. Simple. But a refresh token is a long-lived, high-value credential - if it leaks (logs, a compromised device, a stolen backup), the attacker has indefinite access and you cannot tell the theft happened.

The evolution: rotation with reuse detection. Every time a refresh token is used, the IdP issues a new refresh token and invalidates the old one (single-use). The RP must store the latest. This bounds a leaked token’s usefulness to one use, but the real value is detecting theft:

Refresh tokens form a chain, all tagged with a family_id.
Normal use: RT1 -> (RT2 issued, RT1 marked used) -> RT3 -> ...
If a STOLEN RT1 is replayed AFTER it was already rotated:
  the IdP sees a "used" token being presented again
  => this is a replay. Someone has an old copy.
  => revoke the ENTIRE family_id (invalidate the whole chain).
Both the attacker and the legitimate user are logged out; the user
re-authenticates cleanly, the attacker is locked out. Theft detected.

This turns refresh-token rotation from a mild hardening into an active theft-detection system. The refresh-token store therefore keeps family_id, used flag, and expiry, and the write on rotation must be atomic (a transaction or compare-and-swap) so two concurrent refreshes cannot both succeed and desync the chain.

4. Signing keys and rotation without downtime

Naive approach: one signing key, hardcoded public key in every resource server. Works until you need to rotate (routine hygiene, or a suspected key compromise). Rotate the key and every resource server that has not been redeployed rejects every token - a fleet-wide outage.

The evolution: JWKS with kid and overlapping keys. The IdP publishes its public keys at a well-known JWKS endpoint (/.well-known/jwks.json), each with a kid. Every JWT header names the kid it was signed with. Resource servers fetch and cache JWKS (with a short-ish TTL, minutes to hours) and pick the key by kid.

Rotation is then graceful and overlapping:

1. Generate new keypair K2. Publish BOTH K1 and K2 public keys in JWKS.
2. Keep signing with K1. Wait for JWKS caches to pick up K2
   (past the cache TTL) - now every RS knows both keys.
3. Switch issuance to sign with K2 (tokens now carry kid=K2).
4. Old tokens signed with K1 still verify (K1 still in JWKS).
5. After the max access-token TTL passes, no live token uses K1.
   Remove K1 from JWKS.

No downtime, and the same procedure is your break-glass for a compromised key (skip the wait, force-refresh JWKS, and pair it with a mass revocation epoch). Private keys live in an HSM or cloud KMS and never enter application memory - the token endpoint asks KMS to sign, it does not hold the key. That way a compromise of the IdP application servers does not hand the attacker the ability to mint tokens.

API Design & Data Schema

Standard OAuth 2.0 / OIDC endpoints. Concrete shapes:

GET /authorize
  ?response_type=code
  &client_id=app_x
  &redirect_uri=https://appx.com/cb
  &scope=openid%20email%20read:calendar
  &state=<csrf_random>
  &code_challenge=<base64url(sha256(verifier))>
  &code_challenge_method=S256
  -> 302 redirect to login/consent, then back to
     https://appx.com/cb?code=<authz_code>&state=<csrf_random>

POST /token          (Authorization Code exchange)
  grant_type=authorization_code
  code=<authz_code>
  redirect_uri=https://appx.com/cb
  client_id=app_x
  client_secret=<secret>            (confidential clients)
  code_verifier=<pkce_verifier>     (public clients)
  -> 200 {
       "access_token": "<JWT>",
       "token_type": "Bearer",
       "expires_in": 900,
       "refresh_token": "<opaque>",
       "id_token": "<JWT>",
       "scope": "openid email read:calendar"
     }

POST /token          (silent refresh)
  grant_type=refresh_token
  refresh_token=<opaque>
  client_id=app_x
  -> 200 { new access_token, new rotated refresh_token, expires_in }

POST /revoke         (RFC 7009)
  token=<access_or_refresh_token>
  token_type_hint=refresh_token
  -> 200 (idempotent)

POST /introspect     (RFC 7662 - for opaque/legacy consumers only)
  token=<token>
  -> 200 { "active": true, "sub": "...", "scope": "...", "exp": ... }

GET  /.well-known/openid-configuration   -> discovery document
GET  /.well-known/jwks.json              -> public signing keys (JWKS)
GET  /userinfo   (Bearer access token)   -> { sub, email, name, ... }
POST /logout     (RP-initiated / front-channel single logout)

An access token JWT payload:

{
  "iss": "https://idp.example.com",
  "sub": "user_9f3a...",          // stable user id
  "aud": ["calendar-api", "mail-api"],
  "azp": "app_x",                 // authorized party (the client)
  "scope": "openid email read:calendar",
  "iat": 1754179200,
  "exp": 1754180100,              // iat + 900s
  "jti": "at_7c1e..."             // unique id, used for revocation
}

Data stores - SQL vs NoSQL, chosen per table:

Users and clients live in a relational DB (Postgres, sharded/replicated) - strong consistency on client registration, referential integrity between clients and their grants, and relatively low volume.

clients (SQL)
  client_id        VARCHAR   PK
  client_secret_hash TEXT              -- confidential clients only
  type             ENUM(confidential, public)
  redirect_uris    TEXT[]              -- exact-match allowlist (anti-open-redirect)
  allowed_scopes   TEXT[]
  grant_types      TEXT[]
  created_at       TIMESTAMP

consents (SQL)   -- "user granted app_x these scopes"
  user_id    BIGINT   ─┐  composite PK (user_id, client_id)
  client_id  VARCHAR  ─┘
  scopes     TEXT[]
  granted_at TIMESTAMP
  INDEX (user_id)                      -- list/revoke a user's app grants

Refresh tokens and sessions are high-churn, key-lookup, sharded by an opaque key - a NoSQL / KV store (DynamoDB, Cassandra, or a sharded Redis with durability) fits better than SQL:

refresh_tokens (KV, sharded by token_hash)
  token_hash    BYTES    PK            -- store the HASH, never the token
  family_id     UUID                   -- rotation chain / theft detection
  user_id       BIGINT
  client_id     VARCHAR
  scopes        TEXT[]
  used          BOOLEAN                -- single-use; replayed-if-true => revoke family
  expires_at    TIMESTAMP  (TTL index) -- auto-expire
  device_meta   JSON
  INDEX (user_id)     -- "log out everywhere": delete all of a user's tokens
  INDEX (family_id)   -- revoke a whole chain on replay

sso_sessions (KV, sharded by session_id)
  session_id    BYTES    PK            -- the value in the SSO cookie (hashed)
  user_id       BIGINT
  created_at    TIMESTAMP
  last_seen     TIMESTAMP
  amr           TEXT[]                 -- how they authed (pwd, mfa, ...)
  expires_at    TIMESTAMP (TTL)
  INDEX (user_id)     -- single logout: find all sessions for a user

authorization_codes (KV or in-memory KV, ~60s TTL)
  code_hash     BYTES    PK
  client_id, redirect_uri, user_id, scopes, code_challenge
  expires_at    (TTL, single-use)

Authorization codes are ephemeral (60s, single-use) - a TTL KV store (or Redis) is ideal; losing them on a node failure just makes a login retry, no durability crisis. The revocation log is a durable append-only stream (Kafka), not a queryable DB - it exists to be replayed/tailed by resource servers, not queried.

The choice pattern: SQL where I need integrity and joins (clients, consents), KV where I need sharded key-lookup at churn (tokens, sessions, codes), a log where I need fan-out (revocations).

Bottlenecks & Scaling

Where it breaks first: token validation. Already solved by design - offline JWT verification pushes the 100M/sec entirely onto the resource-server fleet, which scales horizontally with the apps themselves. The IdP never sees that traffic. This is the whole reason the architecture chose self-describing tokens.

Token issuance and the token endpoint. ~140K/sec at peak. The token endpoint is stateless per-request (validate code, sign JWT, write refresh token) so it scales horizontally behind a load balancer. The signing step calls KMS/HSM - cache and pool those connections, and use a signing key that can be exercised at high QPS (KMS is the potential bottleneck; ES256 signatures are cheap, and you can shard signing across KMS key versions if needed).

Sharding the token / session stores. Shard key = token_hash / session_id (the natural lookup key), which spreads uniformly by construction (hashes are uniform). The secondary access pattern - “all tokens/sessions for a user” (logout-everywhere) - does not follow the shard key, so keep a secondary index on user_id or a user_id -> [token_ids] mapping so a bulk revoke does not scatter-gather every shard. Consents and clients shard by user_id and client_id respectively.

Hot keys - the celebrity/service-account problem. A shared service account or a hugely popular OAuth client (the “app_x” everyone uses) makes its clients row and consent lookups hot. Fix: cache client metadata aggressively at the token endpoint (clients change rarely), and for a machine client doing millions of client-credentials grants, issue longer-lived tokens or a dedicated issuance lane so it does not hammer one shard.

Replication and the consistency split. The user/client SQL DB runs primary + read replicas; reads (client lookup during issuance) hit replicas, the rare writes (client registration) hit the primary. Refresh-token writes need read-your-writes on rotation, so route a token’s reads and writes to the same shard primary. The revocation path is the one place I force strong, fast consistency - the revocation log is durably replicated before the /revoke returns 200, so a caller who revokes is guaranteed the entry is committed and streaming.

Single points of failure.

  • Signing keys. The crown jewel. In HSM/KMS, multi-region, never in app memory. A leaked signing key is catastrophic - it lets an attacker mint valid tokens for any user - so key access is audited and rotatable (the graceful rotation above is also the incident response).
  • The IdP itself. Made non-critical for the hot path by offline validation: the IdP can be down and the fleet keeps serving requests with existing tokens. Only new logins and refreshes fail during an IdP outage. Run the IdP multi-region, active-active, so even that degradation is rare.
  • The revocation stream. If it lags or dies, revocations do not propagate - a security risk, not an availability one. Monitor its lag hard; on total failure, resource servers can fall back to a shorter cache TTL / mandatory re-check for high-value scopes, trading latency for safety.

Async processing. Consent-screen analytics, login-anomaly scoring, audit-log writes, and email notifications (“new sign-in to your account”) all go on a queue off the issuance hot path. The only synchronous writes on issuance are the refresh-token record and the auth-code consumption; everything else is fire-and-forget.

Cross-region. Users are global; route them to the nearest IdP region. The SSO session and refresh tokens should be readable in the region the user lands in - replicate sessions cross-region (async is usually fine; a session appearing a few hundred ms late just means one extra login). JWKS is global and cached everywhere. Revocation must fan out to all regions’ resource servers, so the revocation log is globally replicated.

Wrap-Up

The trade-offs, stated plainly:

  • Offline JWT validation over central introspection. Bought sub-ms, IdP-independent validation at 100M/sec and made the IdP non-critical for the hot path - at the cost of tokens that are not instantly revocable. Paid that cost down with short TTLs plus a streamed revocation set.
  • Split token strategy: self-verifying JWT access tokens, opaque centrally-checked refresh tokens. Each token type uses the model that fits its access pattern - high-volume access tokens verify offline, low-volume refresh tokens check centrally and revoke instantly.
  • Short access-token TTL + refresh rotation with reuse detection. Kept the blast radius of a stolen access token to minutes and turned refresh rotation into active theft detection.
  • Strong consistency only on revocation; eventual everywhere else. Paid for consistency exactly where a stale read is a security hole, and nowhere else.
  • Signing keys in HSM/KMS with overlapping JWKS rotation. The single most valuable secret in the system is never in app memory and can rotate with zero downtime.

One-line summary of the final design: a central Identity Provider mints short-lived, asymmetrically-signed JWT access tokens (verified offline by the whole fleet at 100M/sec) plus rotating opaque refresh tokens (checked centrally), and backstops offline verification with a tiny in-memory revocation set streamed to every resource server - buying both fleet-scale validation and near-instant revocation, which is the one trade-off SSO is really about.