Everyone draws the API gateway as a single box labelled “gateway” between the client and the microservices and moves on. Then the interviewer starts pulling threads: this one box is on the hot path of every request in the entire company, so what happens to latency when it does auth, rate limiting, routing, and circuit breaking on each of millions of requests per second? Where does the per-client rate-limit count live when you have 200 stateless gateway nodes behind a load balancer and the same customer’s requests are sprayed across all of them? What does the gateway do when one downstream service starts timing out - does it drag every caller down with it? Now it is a real problem.
The gateway is deceptive because it looks like plumbing but it is the single most load-bearing component you own. It is the one choke point every external request passes through, which makes it the perfect place to enforce cross-cutting policy (auth, quotas, routing, observability) and simultaneously the scariest single point of failure in the system. The whole design is a fight between two goals that pull in opposite directions: centralize policy so you enforce it once, and stay fast and available so you do not become the bottleneck or the outage. The hardest specific sub-problem, and the one I will spend the most time on, is enforcing a global per-client quota across a stateless fleet without paying for consensus on every request.
Let me do it properly.
Functional Requirements (FR)
In scope:
- Single entry point. All external (north-south) traffic enters through the gateway. Clients never address microservices directly; they hit one set of gateway endpoints.
- Routing. Match an incoming request (host, path, method, headers) to the correct upstream service and forward it, with path rewriting and version routing (
/api/v2/*-> payments-v2). - Authentication and authorization. Verify the caller’s identity (API key, JWT, OAuth token, mTLS) and reject unauthenticated or unauthorized requests at the edge, before any upstream is touched.
- Rate limiting and quotas. Enforce per-client request limits and quotas globally across the whole fleet, by tier and by endpoint.
- Circuit breaking and resilience. Detect failing upstreams and stop sending them traffic (fail fast), with retries, timeouts, and load shedding, so one sick service does not cascade.
- Request/response transformation. Header injection/stripping, protocol translation (public REST/JSON to internal gRPC), request/response rewriting.
- Observability. Emit structured logs, metrics, and distributed-tracing spans for every request so the gateway is the single pane of glass for edge traffic.
Explicitly out of scope (say this out loud so you own the scope):
- The identity provider itself. The gateway verifies tokens; issuing them, the login flow, user stores, and OAuth dance are a separate auth service. We consume its public keys, we do not run it.
- L3/L4 DDoS and volumetric attack scrubbing. That belongs at the network edge (anycast, SYN cookies, scrubbing centers) in front of the gateway. We handle L7 application traffic.
- Full WAF / signature-based threat detection. We do policy enforcement, not a payload-inspection security engine (that can sit as a filter in front of or inside the gateway, but it is its own system).
- Service-to-service (east-west) traffic. Internal mesh calls go through sidecars, not the edge gateway. We mention the split but the design target is north-south ingress.
- Business logic. The gateway is policy and routing, never domain logic. The moment it knows what a “payment” is, you have built a distributed monolith.
The one decision that drives everything: the gateway runs inline on the critical path of every single request. Every millisecond of latency and every failure mode it introduces is multiplied by total system QPS. So the design is dominated by “how do I enforce rich cross-cutting policy while staying fast, stateless, and available enough that I am not the bottleneck or the SPOF.”
Non-Functional Requirements (NFR)
- Scale: the fleet fronts ~2M requests/sec at peak across all services. Tens of millions of distinct clients (API keys / user IDs). Thousands of distinct routes mapping to hundreds of upstream services.
- Latency: the gateway must add under ~2-3ms at p99 on top of upstream latency. It is pure overhead - every bit of it is tax on every call in the company. The common “authenticated, under limit, route found” path must be decided with zero or near-zero blocking network calls.
- Availability: 99.99%+ (four nines). This is non-negotiable because the gateway is the single front door - if it is down, every service is unreachable regardless of how healthy the services are. This drives a stateless, horizontally scaled, multi-AZ fleet with no single instance in the critical path.
- Consistency: config (routes, rules, keys) is read-heavy and must be strongly consistent on change but served from local memory on the hot path. Rate-limit counters are eventually consistent with a bounded overshoot - we explicitly trade perfect global accuracy for latency.
- Durability: the gateway itself is stateless and holds no durable data. Config is durable in a control-plane store; counters are ephemeral in-memory state we are happy to lose on a crash (worst case: a client briefly gets a few extra requests).
The tension to name explicitly: four-nines availability plus single-digit-millisecond overhead plus rich centralized policy. You cannot get all three by doing everything synchronously inline, which is why the design pushes config into local memory, counters into an async-reconciled hybrid, and failure handling into circuit breakers that fail fast rather than block.
Back-of-the-Envelope Estimation (BoE)
Let me ground the design in numbers.
Request rate (the gateway’s own load):
Peak system traffic: 2,000,000 requests/sec
The gateway processes every request inline.
= 2M gateway passes/sec
Per request the gateway does, in order:
parse + route match -> auth (verify token) -> rate-limit check
-> forward to upstream -> observe (log/metric/trace)
Sizing the fleet:
A single modern gateway node (tuned, async I/O) handles
~20,000-50,000 req/sec doing real L7 work (parse, JWT verify, route, proxy).
Take a conservative 15,000 req/sec/node to leave headroom.
2,000,000 / 15,000 ≈ 135 nodes at peak.
Run ~200 nodes across 3 AZs for headroom + AZ-failure survival
(lose one AZ, the remaining ~133 still carry 2M).
The fleet is large and must be stateless so you can add/remove nodes freely and so the load balancer can spray requests across all of them without affinity.
Auth cost (why we cache):
JWT with asymmetric signature (RS256/ES256):
signature verify ≈ 50-200 microseconds CPU per request, no network call
(we hold the issuer's public keys (JWKS) in memory, refreshed periodically).
Opaque-token introspection (call the auth service per request):
1 network RTT (~0.5-2ms) + load on the auth service of 2M req/sec.
2M req/sec against an introspection endpoint is a self-inflicted DDoS.
That contrast is the whole reason we prefer stateless JWT verification (local, sub-millisecond, no network) over per-request introspection, and cache introspection results when we must use opaque tokens.
Rate-limit state (memory for counters):
Distinct active clients in a window: ~10M
Per client per rule, token-bucket state: (tokens_float, last_ts) ≈ 16 bytes
+ key overhead ≈ 40-60 bytes
≈ ~80 bytes/client/rule in Redis
10M clients * ~80 bytes ≈ 800 MB; a few rules each -> ~2-4 GB total.
Tiny. The entire quota state fits in RAM on a small sharded Redis cluster. This is a coordination-and-latency problem, not a storage problem.
Redis op rate if we checked counters naively (why we do not):
2M limit-checks/sec against Redis.
One Redis node does ~100K-200K simple ops/sec comfortably.
2M / 150K ≈ 14 shards just to hold the op rate,
and every request would pay one RTT (~0.3-0.5ms in-datacenter).
That RTT-per-request number is the ballgame for the quota problem. It is what pushes us to decide the common case in-process and reconcile asynchronously.
Bandwidth: the gateway proxies full request and response bodies. If average payload is ~4 KB and we handle 2M req/sec, that is 2M * 4KB * 2 (req+resp) ≈ 16 GB/sec of proxied traffic flowing through the fleet - spread over 200 nodes that is ~80 MB/sec/node, fine, but it means the gateway must stream bodies, not buffer them fully in memory, or it will OOM on large payloads.
Config size: thousands of routes + millions of API keys’ metadata. Routes are small (a few MB) and fit in every node’s memory. Full key material does not - we verify signatures with a handful of public keys, and look up key metadata (tier, status) from a cache, not by holding all keys in every node.
High-Level Design (HLD)
An API gateway is not one service you route to; it is a chain of inline filters every request passes through before reaching business logic. The architecture splits into a data plane (the stateless fleet that processes live traffic on the hot path) and a control plane (manages config - routes, auth policy, rate rules - and pushes it to the data plane, off the hot path). This split is the single most important structural idea: the thing on the critical path must be dumb, fast, and stateless; the thing that changes config can be slow and consistent.
┌──────────────────────────────┐
│ Clients │
│ (apps, partners, browsers) │
└───────────────┬────────────────┘
│
┌───────────────▼────────────────┐
│ L3/L4 LB + Anycast (DDoS edge) │
└───────────────┬────────────────┘
│ spray (no affinity)
┌──────────────────────────────┼──────────────────────────────┐
│ │ │
┌────────▼─────────┐ ┌─────────▼────────┐ ┌──────────▼───────┐
│ Gateway Node 1 │ │ Gateway Node 2 │ ... │ Gateway Node N │
│ ┌───────────────┐ │ │ (stateless) │ │ (stateless) │
│ │ FILTER CHAIN │ │ └──────────────────┘ └──────────────────┘
│ │ 1 route match │ │ each node runs the same ordered chain:
│ │ 2 authN/authZ │ │ ──────────────────────────────────────────────
│ │ 3 rate limit │ │ request parse
│ │ 4 transform │ │ -> ROUTE match (host/path/method -> upstream)
│ │ 5 circuit brk │ │ -> AUTH (verify JWT w/ cached JWKS, or introspect)
│ │ 6 forward │ │ -> RATE (local bucket + async Redis reconcile)
│ │ 7 observe │ │ -> TRANSFORM (headers, REST->gRPC)
│ └──────┬────────┘ │ -> CIRCUIT BREAKER (is upstream healthy?)
└────────┼──────────┘ -> FORWARD (stream to upstream, retry/timeout)
│ allow -> OBSERVE (log/metric/trace span)
▼
┌──────────────────────────────────────────────────────────────────────┐
│ Upstream Microservices │
│ payments orders users search ... (each behind its own LB) │
└──────────────────────────────────────────────────────────────────────┘
Shared data-plane dependencies (consulted off the common path):
┌─────────────────────────────┐ ┌──────────────────────────────────┐
│ Redis Cluster (quota state) │ │ Auth: JWKS endpoint + token │
│ sharded by client_id │ │ introspection (cached) │
└─────────────────────────────┘ └──────────────────────────────────┘
CONTROL PLANE (off the hot path):
┌──────────────────────────────────────────────────────────────────────┐
│ Admin API -> Config Store (etcd/Consul) ──push──> all gateway nodes │
│ routes, rate rules, auth policy, upstream registry; hot-reloaded │
│ Service discovery feeds live upstream endpoints into the registry │
└──────────────────────────────────────────────────────────────────────┘
Request flow (the hot path):
- A request hits the L3/L4 load balancer (anycast, absorbs volumetric attacks) and is sprayed to any gateway node - no session affinity, because nodes are stateless.
- The node parses the request and runs route matching: find the upstream service for
(host, path, method)from the in-memory route table. No match ->404. This is a local lookup, no network. - Auth filter: extract the credential. For a JWT, verify the signature locally using the issuer’s public key (JWKS) held in memory, check expiry/issuer/audience/scopes. Invalid ->
401; valid but not allowed for this route ->403. For opaque tokens, consult the introspection cache, falling back to the auth service only on a miss. - Rate-limit filter: build the client+rule key, decide against the node’s local token bucket; consult sharded Redis only at boundaries / on a cadence. Over limit ->
429withRetry-After. (This is the hard part, detailed below.) - Transform filter: strip hop-by-hop headers, inject identity headers (
X-User-Id,X-Tier) for the upstream, translate protocol if needed (public REST/JSON to internal gRPC). - Circuit-breaker + forward: check the breaker for the target upstream; if open, fail fast with
503(or serve a fallback). If closed, forward with a timeout and bounded retries, streaming the body. Record the success/failure into the breaker’s stats. - Observe: emit a structured access log, metrics (latency, status, upstream), and a distributed-tracing span, then stream the response back to the client.
The key architectural insight is that steps 2, 3, and 4 are all designed to be decided in-process from local memory on the common case. The gateway only reaches out over the network for the thing it cannot avoid (forwarding to the upstream) and, occasionally, for quota reconciliation and auth cache misses. Everything else is local, because at 2M req/sec a network call per filter per request would be fatal.
Component Deep Dive
Four hard parts: (1) the stateless data-plane / control-plane split and config push, (2) auth at the edge without a round trip per request, (3) global rate limiting across a stateless fleet - the headline problem, and (4) circuit breaking so a sick upstream does not cascade. I will walk each from naive to scalable.
1. Stateless Data Plane + Control-Plane Config Push
Approach A: Config baked into the gateway binary (the naive start)
Ship routes and rules in a config file deployed with the gateway. Where it breaks: every route change, every new API key tier, every rate-limit tweak now needs a full fleet redeploy of 200 nodes. A new upstream version means a code push to the front door of the whole company. Change velocity dies, and worse, a bad config is a bad deploy that can take the gateway down - the single most dangerous thing to break.
Approach B: Gateway reads config from a store per request (the overcorrection)
Make the gateway stateless by having it fetch the matching route/rule from a central config store on each request. Where it breaks: you just put a network call (and a shared dependency) on the hot path of every request. At 2M req/sec the config store is now a 2M-ops/sec bottleneck and a new SPOF, and you added an RTT to every call. Stateless is right; per-request fetch is wrong.
Approach C: Control plane pushes config, data plane serves from memory (the answer)
Split responsibilities:
- Control plane: an admin API writes routes, rate rules, auth policy, and the upstream registry into a strongly consistent store (etcd / Consul). It validates config (reject a route that points nowhere, a rule with a nonsense limit) before accepting it - the gateway is too important to let a malformed config in. Service discovery feeds live upstream endpoints into the registry as instances come and go.
- Data plane (gateway nodes): on startup, load the full config into memory. Then watch the config store for changes and hot-reload deltas into an in-memory structure, with zero per-request network calls. The route table is a prefix/trie match in memory; rules are a map lookup in memory.
Admin changes a route
-> Control plane validates -> writes to etcd (strongly consistent)
-> etcd watch fires -> each gateway node pulls the delta
-> node atomically swaps its in-memory route table (read-copy-update)
-> new requests use new routes; no redeploy, no per-request lookup
This gives the best of both: nodes are stateless (any node can serve any request; config is derived, not owned) and the hot path is pure local memory. Config propagation is eventually consistent across the fleet over a few seconds, which is fine - a route that takes 3 seconds to reach all 200 nodes is acceptable; a route that needs a redeploy is not. The one thing we make strongly consistent is the write to etcd, so there is a single source of truth and no split-brain config.
The stateless property is what unlocks availability: no node holds unique state, so losing a node loses nothing, the load balancer needs no affinity, and autoscaling is trivial (new nodes just load config and join).
2. Auth at the Edge Without a Round Trip per Request
The gateway must authenticate every request, but auth is exactly where a naive design adds a network call per request.
Approach A: Opaque token + introspection per request (the naive, correct-but-slow start)
Client sends an opaque bearer token. The gateway calls the auth service’s /introspect endpoint to ask “is this token valid, and who is it?” on every request. Where it breaks: 2M req/sec of introspection calls is a self-inflicted DDoS on the auth service, and every request pays a full RTT before it can even be routed. Correct, but it makes auth the bottleneck of the entire platform.
Approach B: Stateless JWT verification (the standard answer)
Use signed JWTs (RS256/ES256). The auth service signs a token with its private key at login; the gateway verifies the signature with the auth service’s public key, which it holds in memory (fetched from a JWKS endpoint and refreshed every few minutes). Verification is pure local CPU - no network call. The token itself carries the identity, scopes, tier, and expiry as claims, so the gateway reads everything it needs from the verified payload.
Login (once): client -> Auth service -> signs JWT with PRIVATE key -> client
Every request: client -> Gateway: verify JWT signature with PUBLIC key (in memory)
check: exp (not expired), iss, aud, scopes for this route
-> identity extracted from claims, zero network calls
Why this wins: verification is ~50-200 microseconds of local CPU, scales with the fleet (no shared dependency on the hot path), and the auth service only sees login traffic, not every API call.
The JWT revocation problem (name it, do not hand-wave): a stateless JWT is valid until it expires, so you cannot instantly revoke one. Fixes: keep access-token TTL short (5-15 min) so a leaked token self-expires quickly, pair with longer-lived refresh tokens handled by the auth service, and for immediate kill-switch needs maintain a small revocation list / deny-set of jti (token IDs) or user IDs in a fast cache (Redis / bloom filter) that the gateway checks - this is a tiny set (only actively revoked tokens) and a cache hit is sub-millisecond, versus introspecting every token. You pay the cache lookup only to gain instant revocation, and you can skip it for low-risk routes.
Approach C: Hybrid for opaque tokens - cache introspection
Some ecosystems require opaque tokens (no claims leaked to the client, server-side session). Keep introspection but cache the result keyed by token hash with a short TTL (e.g. 30-60s). First request for a token introspects once; the next thousand within the TTL hit the local/Redis cache. This collapses 2M introspections/sec down to roughly the rate of distinct new tokens, turning a per-request network call into a per-token one. Trade-off: a revoked token stays valid for up to the cache TTL - tune TTL against your revocation-latency requirement.
Decision: prefer stateless JWT (local verification, short TTL, small revocation deny-set for instant kill). Fall back to cached introspection when opaque tokens are mandated. Never introspect per request uncached.
3. Global Rate Limiting Across a Stateless Fleet (the core hard part)
This is the headline problem: a client is limited to, say, 10,000 req/min globally, but that client’s requests are sprayed across 200 stateless gateway nodes by a load balancer with no affinity. Each node sees only a slice of the client’s traffic, so no single node can enforce the global limit alone. And we cannot afford consensus (a synchronous agreement across nodes) on every request.
Naive: per-node local counters
Each gateway node keeps its own in-memory counter per client. Where it breaks: with 200 nodes and a 10,000/min limit, if each node allows 10,000 the client gets 2,000,000/min globally (200x the limit). If you divide the limit by node count (10,000/200 = 50 per node), a client whose traffic happens to concentrate on a few nodes gets throttled at 50 while the global budget is 10,000 - wildly wrong in both directions because load-balancer spray is never perfectly even. Local-only cannot enforce a global limit. You need shared state.
Better: a shared counter in Redis with atomic ops
All nodes read and write the same counter in a central Redis, sharded by client. The hard requirement is atomicity: the read-refill-decrement of a token bucket must be one indivisible operation, or two nodes racing on the same client both read “1 token left,” both decrement, and both allow - a lost-update race that blows the limit. Redis is single-threaded per shard, so a Lua script runs start-to-finish with no interleaving:
-- token_bucket.lua KEYS[1] = "rl:{rule_id}:{client_id}"
-- ARGV: capacity, refill_rate_per_sec, now, requested(=1)
local capacity = tonumber(ARGV[1])
local refill_rate = tonumber(ARGV[2])
local now = tonumber(ARGV[3])
local requested = tonumber(ARGV[4])
local state = redis.call("HMGET", KEYS[1], "tokens", "ts")
local tokens = tonumber(state[1])
local last = tonumber(state[2])
if tokens == nil then tokens = capacity; last = now end
-- lazy refill based on elapsed time, capped at capacity
tokens = math.min(capacity, tokens + (now - last) * refill_rate)
local allowed = 0
if tokens >= requested then
tokens = tokens - requested
allowed = 1
end
redis.call("HSET", KEYS[1], "tokens", tokens, "ts", now)
redis.call("PEXPIRE", KEYS[1], math.ceil(capacity / refill_rate * 1000) + 1000)
return { allowed, tokens }
This gives exact global accuracy - one atomic op per request, no race - but it pays a full Redis RTT on every one of 2M requests/sec and needs ~14 shards just for the op rate. It is correct but it drags the gateway’s latency floor up by the RTT and makes Redis a hard dependency on the hot path. We can do better on the common case.
Best for scale: local token bucket + async Redis reconciliation (the hybrid)
The insight: we do not need perfect accuracy, we need a bounded overshoot. So decide the common case locally and reconcile asynchronously, so no consensus happens per request.
Two proven patterns, both of which keep the gateway stateless-enough (local state is a soft cache, not the source of truth):
Batched reservations (the practical default). A node does not claim one token at a time from Redis; it atomically claims a batch (say 50 tokens) from the global bucket in one round trip, then serves those 50 requests from a local in-process counter with zero further Redis calls. When its local reservation runs low, it claims another batch. This cuts Redis traffic by the batch factor (50x fewer round trips -> ~14 shards of op rate becomes trivial) and keeps the common decision in nanoseconds. The cost is a bounded overshoot: a node holding an unused reservation when demand shifts can let the global count run slightly over. Batch size is the knob - bigger batches mean less Redis load but more potential overshoot.
Sync-and-correct. Each node serves from its local bucket and every T milliseconds pushes its local consumption to Redis and pulls back the authoritative global remaining, then corrects its local view toward the truth. Between syncs the global count can drift by at most
(num_nodes * per-node consumption in T). You bound the error by choosing T; shorter T means tighter accuracy and more Redis chatter.
Per-request (common case): decide from LOCAL bucket -> 0 network calls
On reservation low / every T: 1 atomic Redis op to claim batch / reconcile
Net Redis load: ~(2M / batch_size) ops/sec instead of 2M ops/sec
Accuracy: global count within a bounded overshoot, tuned by batch_size / T
This is the honest interview answer to “global quota without consensus per request”: perfect global accuracy would need a synchronous atomic op (effectively consensus on the counter) every request; instead each stateless node decides locally and reconciles asynchronously, accepting a small bounded overshoot (a few percent) to cut coordination by one to two orders of magnitude. State the knob and that it directly trades accuracy against latency and Redis load. The gateway stays stateless in the sense that matters - the local bucket is a disposable optimization, and Redis (sharded, replicated) holds the real counter - so losing a node loses only its unused reservation, a few tokens.
Redis failure - fail open or fail closed? If the client’s Redis shard is unreachable, the node cannot reconcile. Policy is per-rule: fail open (allow, relying on the local bucket as a coarse ceiling) for general public traffic - a rate-limiter outage must not become a full API outage - and fail closed (reject) for security-critical rules like /login brute-force protection. Run each shard as primary + replica with fast failover so this path is rarely hit at all.
4. Circuit Breaking so a Sick Upstream Does Not Cascade
The gateway forwards to hundreds of upstreams. When one gets slow or starts erroring, the naive gateway keeps sending it traffic and waiting on timeouts.
Approach A: Forward everything, rely on timeouts (the naive start)
Just proxy every request and set a timeout. Where it breaks: when payments starts hanging, every request to it sits waiting for the full timeout (say 5s) before failing. The gateway’s worker threads/connections pile up blocked on the dead service; at 2M req/sec those blocked workers exhaust the gateway’s own connection pool, and now the gateway cannot serve requests to healthy services either. One sick upstream has cascaded into a gateway-wide outage. This is the classic cascading failure.
Approach B: The circuit breaker (the answer)
Wrap each upstream in a circuit breaker - a small state machine per upstream (per gateway node) that tracks recent success/failure and short-circuits calls to a failing service:
failures exceed threshold
(e.g. >50% of last N, or >X in window)
CLOSED ─────────────────────────────────> OPEN
│ ▲ │
│ │ probe succeeds │ after cooldown
│ │ │ (e.g. 10s)
│ └──────────── HALF-OPEN <───────────────┘
│ │ send a few trial requests
└── normal flow └── probe fails -> back to OPEN
CLOSED: forward normally, count failures.
OPEN: fail fast - return 503 / fallback instantly, do NOT call the upstream.
HALF-OPEN: let a trickle of probes through; if they succeed, close; if not, re-open.
- CLOSED (normal): requests flow; the breaker counts failures (timeouts, 5xx, connection refused) over a rolling window.
- OPEN (tripped): once failures cross the threshold, the breaker opens and the gateway fails fast - returns
503or a cached/default fallback immediately without calling the upstream. This is the key move: it stops blocked workers from piling up and frees the gateway to keep serving healthy routes. - HALF-OPEN (recovering): after a cooldown, let a small number of trial requests through. If they succeed, close the breaker; if they fail, re-open and wait again. This probes for recovery without flooding a still-sick service.
Companions to the breaker, all at the gateway:
- Timeouts, aggressive and per-upstream, so a slow call fails fast instead of holding a worker.
- Retries with backoff + jitter, but only on idempotent methods and with a tight retry budget - naive retries on a struggling service create a retry storm that finishes it off. Cap retries and shed rather than retry when the breaker is near tripping.
- Bulkheads: isolate connection pools per upstream so exhaustion on one upstream’s pool cannot starve others - the gateway keeps separate resource budgets per downstream, so a flooded
paymentspool does not drain thesearchpool. - Load shedding: when the gateway itself is overloaded (CPU/queue depth high), drop or
503the lowest-priority traffic early to protect its ability to serve the rest.
The breaker state is per-node and per-upstream, which is fine - each node independently observes and reacts to the health of each upstream it talks to; you do not need a global breaker, because each node’s local view of an upstream’s health converges fast and a per-node breaker avoids a shared-state dependency on the failure path (the worst time to need a network call is when things are already failing).
API Design & Data Schema
Client-facing behavior (the gateway is transparent proxy + headers)
To clients, the gateway is the API - they call normal endpoints and the gateway adds policy via headers and status codes:
Any request -> gateway:
Authenticated, under limit, routed (proceeds to upstream), response carries:
X-RateLimit-Limit: 10000
X-RateLimit-Remaining: 8423
X-RateLimit-Reset: 1800000000
X-Request-Id: 7f3c... # correlation id, also the trace id
Rejections (never reach the upstream):
401 Unauthorized - missing/invalid/expired token
403 Forbidden - valid token, not allowed for this route/scope
404 Not Found - no route matches
429 Too Many Requests - over quota; Retry-After: 12
503 Service Unavailable - circuit open / upstream down; Retry-After: 5
Admin / control-plane API (managing the gateway)
# Routes
PUT /admin/v1/routes/{route_id}
{
"match": { "host": "api.example.com", "path": "/v2/payments/*", "methods": ["POST"] },
"upstream": "payments-v2",
"rewrite": { "strip_prefix": "/v2" },
"protocol": "grpc", # translate incoming REST/JSON -> gRPC
"auth": { "required": true, "scopes": ["payments:write"] },
"rate_rule": "paid-tier-default",
"timeout_ms": 800,
"retries": { "max": 2, "on": ["5xx", "connect_error"], "idempotent_only": true },
"circuit_breaker": { "error_threshold_pct": 50, "window": 100, "cooldown_ms": 10000 }
}
Response 200: { "route_id": "payments-v2-write", "version": 42 }
# Rate rules
PUT /admin/v1/rate-rules/{rule_id}
{
"algorithm": "token_bucket",
"capacity": 10000,
"refill_rate_per_sec": 166.6, # ~10000/min sustained
"key_by": "api_key", # api_key | user_id | ip
"on_store_failure": "fail_open" # fail_open | fail_closed
}
# Upstream registry (usually fed by service discovery, not by hand)
GET /admin/v1/upstreams -> list upstreams + healthy endpoints
GET /admin/v1/routes -> list routes
DELETE /admin/v1/routes/{route_id} -> remove a route
All of this is written to the control-plane store, validated, and pushed to the data plane. Nothing here is on the hot path.
Data store choices
The gateway touches three kinds of state, each with a different store, each for a reason:
Config (routes, rate rules, auth policy, upstream registry) - strongly consistent KV (etcd / Consul)
Route:
route_id STRING PK
match JSON -- host, path pattern, methods
upstream STRING -- FK to upstream registry
rewrite JSON
auth JSON -- required, scopes
rate_rule STRING -- FK to rate rule
timeout_ms INT
retries JSON
circuit_breaker JSON
version INT
Rate rule:
rule_id STRING PK
algorithm ENUM -- token_bucket | sliding_counter | leaky_bucket
capacity INT
refill_rate FLOAT
key_by ENUM -- api_key | user_id | ip
on_store_failure ENUM -- fail_open | fail_closed
Config is read-heavy, tiny, changes rarely, and must have a single source of truth with no split-brain - exactly the profile for a strongly consistent config store (etcd/Consul) that pushes to the fleet. It is not SQL because we need low-latency watch/push to 200 nodes, not relational queries, and the volume is trivial.
Rate-limit counters - in-memory KV (Redis), sharded, non-durable
Key: rl:{rule_id}:{client_id} -- e.g. "rl:paid-default:apikey_abc123"
Type: Hash
Fields: tokens FLOAT -- current tokens (fractional, lazy refill)
ts FLOAT -- last refill timestamp (epoch seconds)
TTL: ~ (capacity / refill_rate) + slack, so idle buckets self-evict
Access: single atomic Lua script (read + refill + decrement + expire)
Shard: hash(client_id) % num_shards
Counter state is tiny, ephemeral, accessed by exact key at extreme rate with atomic read-modify-write. That is the exact profile for in-memory Redis, sharded, deliberately non-durable (we are happy to lose sub-minute counters on a crash). SQL is wrong here: durable disk transactions add latency and durability we do not want, and no relational engine serves atomic per-key decrements at this rate.
Token revocation deny-set - fast cache (Redis / bloom filter)
Set: revoked_jti (or revoked_user_id) -- only actively revoked tokens
TTL: each entry expires when the underlying token would have expired anyway
Check: O(1) membership test; bloom filter in front to skip the lookup for the
overwhelming majority of tokens that are not revoked
Small set (only live revocations), checked cheaply, lets a stateless-JWT design still support instant revocation. SQL vs NoSQL verdict overall: config in a strongly consistent KV, hot counters and deny-sets in in-memory KV, nothing relational - the gateway has no joins, no range queries, and no durable domain data; it has tiny config and tiny hot ephemeral state.
Bottlenecks & Scaling
Where it breaks first, in order, and the fix for each:
1. Redis round-trip on every rate-limit check (breaks first at this scale). At 2M req/sec, one RTT per check saturates Redis op capacity and taxes every request. Fix: the hybrid local-bucket design - batched reservations or periodic sync so the common case is decided in-process and Redis is consulted per batch, not per request. This is the single most important scaling move; it trades a bounded overshoot for a 10-50x drop in Redis load and removes Redis from the common-case latency path.
2. Single Redis node as bottleneck / SPOF. One node cannot hold the op rate and its loss blinds the fleet’s quota enforcement.
Fix: shard Redis by client_id (hash(client_id) % num_shards). Each client’s state lives on exactly one shard, so an atomic Lua op touches one shard and load spreads evenly. Client IDs are high-cardinality and uniformly hashed, so there is no natural hot shard. Run each shard primary + replica with Sentinel/Cluster failover.
Shard key choice: client_id - every quota check is keyed by client, so the client is the natural partition key and each check is a single-shard, single-key atomic op. Sharding by rule_id would be wrong: a few popular rules would create hot shards and pile every client on one node.
3. The gateway fleet itself as the bottleneck / SPOF (the scariest one). The gateway fronts everything, so if the fleet cannot keep up or has a bad deploy, the whole platform is down. Fix: keep the data plane stateless so it scales horizontally and autoscales on CPU/QPS; run 200 nodes across 3 AZs so losing a full AZ still carries peak; put a dumb L3/L4 LB in front (which is itself simpler and more reliable than the L7 fleet); and roll config and binary changes gradually (canary a few nodes, watch error rate, then proceed) because a bad config at the front door is a company-wide outage. No single node holds unique state, so node loss is a non-event.
4. Auth as a hidden per-request network call. Introspecting every token DDoSes the auth service and adds an RTT to every call. Fix: stateless JWT verification with in-memory JWKS (local CPU, no network), short token TTL, and a small revocation deny-set checked from cache only when needed. Cache introspection results per token when opaque tokens are mandated. The auth service then sees login traffic, not API traffic.
5. Cascading failure from a sick upstream. A slow/erroring service fills the gateway with blocked workers and takes down healthy routes too. Fix: per-upstream circuit breakers (fail fast when open), aggressive per-upstream timeouts, bulkhead connection-pool isolation so one upstream cannot starve others, retry budgets with backoff+jitter (never a retry storm), and load shedding of low-priority traffic when the gateway itself is overloaded.
6. Hot client / hot key. A single viral or abusive client concentrates traffic on one Redis key and one route.
Fix: the local token bucket already absorbs most of a hot client’s traffic in-process between syncs, so Redis sees only the per-batch reconcile, not every request. For a genuinely enormous single key, shard one logical bucket into K sub-buckets (client_id:0..K-1, each holding capacity/K) across shards, trading a little accuracy for spread. If keying by IP, beware shared-NAT: a whole office behind one NAT shares a limit - prefer keying by API key/user.
7. Config propagation lag and bad config. A pushed config reaches nodes over a few seconds; a malformed one could brick the front door. Fix: validate config in the control plane before accepting it (reject routes to nowhere, nonsense limits), version it, push deltas with atomic in-memory swap (read-copy-update so in-flight requests finish on the old table), and support fast rollback to the previous version. Eventual (few-second) propagation is acceptable for routes; the write to etcd is strongly consistent so there is one truth.
8. Large-payload buffering / slowloris. Buffering full bodies at 2M req/sec with 4 KB+ payloads OOMs nodes; slow clients tie up connections. Fix: stream request and response bodies through the gateway rather than fully buffering, cap max body size, and set aggressive idle/read timeouts to drop slow-client (slowloris) connections that hold resources without sending data.
9. Clock skew across nodes. Token-bucket lazy refill uses timestamps; disagreeing node clocks drift the refill math.
Fix: on the atomic Redis path, compute elapsed time from Redis server time (Lua TIME) as the single clock authority; keep nodes NTP-synced for the local-bucket path and keep sync intervals short so drift stays bounded.
Wrap-Up
The trade-offs that define this design:
- Stateless data plane + control-plane config push over baked-in or per-request config. Nodes serve routes, rules, and auth policy from in-memory config that the control plane pushes and hot-reloads, so the hot path has zero config network calls and the fleet is trivially horizontally scalable and failure-tolerant. We accept a few seconds of config propagation lag to get this.
- Stateless JWT verification over per-request introspection. Local signature checks with cached public keys keep auth sub-millisecond and off the shared auth service, at the cost of needing short token TTLs plus a small revocation deny-set to regain instant revocation.
- Local token bucket + async Redis reconciliation over consensus-per-request. This is the central hard problem - a global quota across a stateless fleet. Perfect accuracy would need a synchronous atomic op (effectively per-request consensus on the counter); we instead decide the common case from a disposable local bucket and reconcile asynchronously, accepting a small bounded overshoot (tuned by batch size / sync interval) to cut coordination by one to two orders of magnitude. The real counter lives in an atomic, Lua-scripted, client-sharded, replicated Redis.
- Per-node circuit breakers over blind forwarding. Fail fast on sick upstreams, with timeouts, bulkheads, bounded retries, and load shedding, so one bad service cannot cascade into a gateway-wide outage. Breaker state is per-node to avoid a shared dependency on the failure path.
- Four-nines availability bought with statelessness and multi-AZ redundancy, not with a smarter single box. The gateway is the scariest SPOF in the system, so the whole design is about making no single node matter and making a bad config a canaried, reversible event rather than a fleet-wide one.
One-line summary: an API gateway is a stateless, horizontally scaled, multi-AZ fleet of filter-chain nodes fed config by a control plane, verifying JWTs locally, enforcing global per-client quotas via local token buckets reconciled asynchronously against a client-sharded Redis, and protecting itself with per-node circuit breakers - trading a few seconds of config lag and a few percent of quota overshoot for single-digit-millisecond overhead at millions of requests per second.
Comments