“Design a multi-tenant SaaS platform” sounds like “build an app and add a tenant_id column.” Say that in an interview and the follow-ups arrive immediately. “One tenant is a 50,000-seat bank with a compliance team that will audit whether their rows can ever be read by another tenant; the next is a 3-person startup on a free trial. Both run on the same servers and the same database cluster. How do you guarantee the bank’s data is isolated?” “Tenant A ships a CSV import that fires 40,000 writes in a minute - how does tenant B not notice?” “Each tenant has its own feature flags, SSO config, custom fields, data-residency region, and rate-limit tier. Where does that config live and how is it loaded on every request without a round trip?” “A single tenant grows to 80% of your largest shard. What now?”
The system is defined by one tension: you are running one codebase and one pool of infrastructure, but every tenant must behave as if they own a private, isolated, independently-configured deployment - and the cheapest way to share (one big table with a tenant_id column) is exactly what a security-conscious enterprise customer will not accept without proof. The hardest single decision is the data model: schema-per-tenant (or database-per-tenant) for hard isolation, versus a shared schema with row-level security for density and operability. Everything else - rate limiting, config, noisy-neighbor control, onboarding - hangs off that choice. Let me do it properly.
Functional Requirements (FR)
In scope:
- Tenant lifecycle. Provision a new tenant (sign-up / enterprise onboarding), assign it a unique
tenant_id, allocate its storage, seed defaults, and tear it down (offboard, export, delete) on churn. Provisioning must be fast enough for self-serve sign-up (seconds) and support bulk enterprise onboarding. - Strict data isolation. Every read and write is scoped to exactly one tenant. A query can never return another tenant’s rows, even if application code forgets a
WHEREclause. This is the central correctness and compliance requirement, not a nice-to-have. - Per-tenant configuration. Each tenant has its own settings: feature flags, branding/theme, SSO/identity provider, custom fields and schema extensions, data-residency region, and plan tier. Config is read on nearly every request and changes rarely.
- Per-tenant rate limits and quotas. Each tenant (and plan tier) has request-rate limits, concurrency budgets, and resource quotas (storage, API calls/month, seats). Limits are enforced so one tenant cannot starve the shared pool - the noisy-neighbor control.
- Shared application + per-tenant customization. One deployed codebase serves everyone, but behavior varies by tenant config (which features are on, which fields exist, which region data lives in).
- Tenant-scoped admin and observability. Per-tenant usage metering (for billing), per-tenant logs/metrics (so support can debug one tenant), and per-tenant audit trails.
Explicitly out of scope (say this to control scope):
- The product’s actual business domain. This is a platform for hosting tenants; I am not designing a specific CRM/ticketing/analytics feature set. The “entities” are generic tenant-owned records.
- The full billing/invoicing system. I will design usage metering (the input to billing) but not the payment/dunning pipeline.
- Identity internals (OAuth/OIDC/SAML protocol handling). I assume an auth service that, given a request, yields an authenticated principal and their
tenant_id; I will not re-derive SAML. - Front-end/SPA concerns, CDN for static assets, and email delivery. I note where they attach.
The one decision that drives everything: the isolation model (schema-per-tenant vs row-level security on a shared schema) determines your blast radius, your per-tenant cost, your onboarding speed, and your query performance - so the design is about choosing and defending that model, then building rate limiting, config, and noisy-neighbor control on top.
Non-Functional Requirements (NFR)
- Scale: ~5,000 enterprise tenants plus ~50,000 smaller/self-serve tenants. Extremely skewed: most tenants are tiny (hundreds of records, dozens of seats), a handful are enormous (millions of records, tens of thousands of seats). Assume ~2M DAU across all tenants, ~500M tenant-owned records total. Read-heavy, call it ~15:1 read:write.
- Latency: p99 read < 150ms, p99 write ack < 300ms. Config lookup must add near-zero latency (it is on every request), so it must be cache-resident. Tenant routing (resolving which shard/region a request belongs to) must be sub-millisecond.
- Availability: 99.95% for the shared platform. Critically, the blast radius of a failure must be bounded - a bad shard, a poisoned tenant, or a runaway migration should take down one group of tenants, not all of them. Availability is measured per-tenant, not just globally.
- Consistency: read-your-writes within a tenant (a user edits a record, reloads, sees the edit) - so the per-tenant primary store is strongly consistent. Cross-tenant aggregations (platform-wide analytics, billing rollups) can be eventually consistent.
- Durability: a tenant’s data must never be lost or leaked. Isolation is a durability-adjacent property here: “never lost” includes “never served to the wrong tenant.” Per-tenant backups must be restorable independently (a single tenant can request a point-in-time restore without touching others).
- Isolation (the defining NFR): the three isolation axes - data (tenant A cannot read tenant B’s rows), performance (tenant A’s load cannot degrade tenant B’s latency), and config/blast-radius (tenant A’s bad migration/flag cannot break tenant B) - are all first-class. A multi-tenant platform is defined by how well it enforces these.
Back-of-the-Envelope Estimation (BoE)
Real numbers, because they dictate the architecture.
Tenants and records:
55,000 tenants total, 500M records → ~9K records/tenant average,
but power-law: median tenant ~300 records, largest tenants ~20M records.
This skew is the whole problem - you cannot give all 55K tenants a
dedicated database (55K idle Postgres instances is absurd), and you
cannot put a 20M-record whale on the same shared table as 10,000
tiny tenants without the whale's scans hurting everyone.
Writes:
2M DAU * ~30 writes/day (edits, creates, imports averaged) =
60M writes/day
60M / 86,400 ≈ 700 writes/sec average
Peak (3x, business-hours + batch imports) ≈ 2,100 writes/sec
Bulk imports are spiky: one enterprise onboarding can inject
1-10M records in an hour → a deliberate burst that must be
throttled per-tenant so it does not become the platform's peak.
Reads:
2M DAU * ~450 reads/day (page loads, list views, API polling) =
900M reads/day
900M / 86,400 ≈ 10,400 reads/sec average
Peak ≈ 31,000 reads/sec
Each read must first resolve tenant → shard/region (routing) and
load tenant config. If those hit the DB, that is 31K extra QPS of
pure overhead at peak - which is why both are cache-resident.
Config lookups:
Config (flags, SSO, limits, custom-field schema) is read on EVERY
request: ~31K + 2.1K ≈ 33K req/sec at peak each needs config.
Config per tenant ~5 KB. 55K tenants * 5 KB ≈ 275 MB total.
The ENTIRE config set fits in memory on every app node - so config
is loaded once, cached locally, and refreshed on change. Zero
per-request DB cost. This is a key realization.
Storage:
Per record ~2 KB average (structured fields + some JSON).
500M * 2 KB ≈ 1 TB of tenant data, sharded across a cluster.
Per-tenant metadata (config, schema, keys, audit) ~ negligible by
comparison (~tens of GB total).
Audit log: ~2 writes/record-change * 500M lifetime ≈ 1B audit rows
* ~300 B ≈ 300 GB, append-only, ages to cold storage.
Backups: full + incrementals, per-tenant restorable. Budget ~3x
primary for retained backups ≈ 3 TB in object storage.
Rate-limit counter traffic:
Every request decrements a per-tenant token bucket: 33K ops/sec
against a counter store. Must be O(1), in-memory, sharded by
tenant_id (Redis) - cannot be a DB row update per request.
Cache size (hot working set):
Hot records people are actively viewing: ~20M records * ~2 KB
≈ 40 GB. Plus the full 275 MB config set resident everywhere.
Comfortable in a Redis tier + local in-process config cache.
The takeaway: traffic peaks are modest (~2K writes/sec, ~31K reads/sec). The architecture is not dictated by raw QPS - it is dictated by isolation under skew: 55K tenants with a 60,000:1 size ratio sharing one pool, where config and routing must be free, rate limits must be O(1) per request, and the data model must isolate a compliance-audited whale from 10,000 free-trial tenants on the same hardware.
High-Level Design (HLD)
Four concerns dominate: tenant routing (resolve a request to the right shard/region, cheaply), the tenant data store (isolated per tenant under the chosen model), per-tenant policy (config + rate limits loaded/enforced on every request), and the control plane (provisioning, migrations, metering, backups - the per-tenant operations). The data plane (serving requests) is kept dumb and fast; the control plane holds the per-tenant complexity.
┌───────────────────────────┐
│ Clients │
│ (web, mobile, public API)│
└─────────────┬─────────────┘
│ (JWT carries tenant_id)
┌─────────────▼─────────────┐
│ Edge / API Gateway │
│ - authN → principal │
│ - resolve tenant_id │
│ - per-tenant rate limit │
│ - region/shard routing │
└──┬────────────────────┬────┘
request │ │ config/limit lookup
┌─────────────▼──────────┐ ┌─────▼──────────────────┐
│ Application Tier │ │ Policy / Config Cache │
│ (stateless, one │◀──│ (in-proc + Redis; │
│ codebase, all tenants)│ │ flags, SSO, limits, │
│ - sets tenant context │ │ custom-field schema) │
│ - enforces quotas │ └─────────▲──────────────┘
└──┬──────────────────────┘ │ push on change
│ tenant-scoped query │
┌──────────▼───────────────────────┐ ┌───────┴──────────────┐
│ Tenant Router / Directory │ │ Control Plane │
│ tenant_id → {region, shard, mode}│───▶│ - Provisioning svc │
└──────────┬───────────────────────┘ │ - Migration runner │
│ │ - Metering/billing │
┌─────────────┼─────────────────┐ │ - Backup/restore │
│ │ │ │ - Tenant lifecycle │
┌──▼──────┐ ┌───▼──────┐ ┌──────▼───────┐ └───────────────────────┘
│ Shard 1 │ │ Shard 2 │ │ Dedicated DB │
│ (pooled: │ │ (pooled: │ │ (whale tenant│ ┌───────────────────────┐
│ RLS, │ │ RLS, │ │ schema- or │ │ Event Bus (Kafka, │
│ many │ │ many │ │ db-per- │◀──│ partitioned by │
│ tenants)│ │ tenants)│ │ tenant) │ │ tenant_id) │
└────┬─────┘ └────┬─────┘ └──────┬───────┘ └───────┬───────────────┘
│ │ │ │
└─────────────┴────────────────┴───────────────────┤
emit record_changed / usage events │
┌───────▼──────────┐
│ Metering + Audit │
│ + Analytics sink │
│ (OLAP, cold store)│
└───────────────────┘
Request flow (read or write):
- Client calls the API with a JWT whose claims include
tenant_id(set at login by the auth service). The Edge/API Gateway validates the token and extractstenant_id- the gateway never trusts atenant_idfrom the request body or a query param; it comes only from the signed token. - The gateway looks up the tenant’s policy (rate-limit tier, region) and routing (
tenant_id → {region, shard, isolation_mode}) from the Tenant Router + Policy Cache, both cache-resident (sub-ms). It applies the per-tenant rate limit (token bucket). If over limit,429immediately, before any work. - The request is routed to an Application node in the tenant’s home region. The app establishes a tenant context (sets
tenant_idfor the whole request lifecycle) and loads tenant config from the local in-process cache. - The app issues a tenant-scoped query to the tenant’s data store. How isolation is enforced here (RLS session variable vs schema selection vs dedicated DB) is the deep dive. Strongly consistent read of the tenant’s primary for read-your-writes.
- The store emits
record_changedandusageevents (via a transactional outbox) onto Kafka partitioned bytenant_id. Consumers meter usage (for billing), write the audit log, and feed cross-tenant analytics.
Control-plane flow (provisioning a tenant):
POST /tenants→ Provisioning service allocates atenant_id, decides placement (which pooled shard, or a dedicated DB for a known-whale enterprise), writes the routing entry, seeds default config, and - under RLS - simply starts writing rows with the newtenant_id; under schema-per-tenant, runsCREATE SCHEMA+ table DDL. Returns201with the tenant’s endpoint.- Config changes (flags, limits) are written to the control plane, which pushes an invalidation onto the bus so every app node refreshes its local config cache within seconds.
The key split: the data plane is stateless and uniform (one codebase, cache-resident policy, O(1) rate limiting), and all per-tenant variability - placement, isolation mode, migrations, metering, backups - is owned by the control plane and the Tenant Router.
Component Deep Dive
The hard parts, naive-first then evolved: (1) the isolation model - schema-per-tenant vs row-level security (the central decision), (2) per-tenant rate limiting and noisy-neighbor control, (3) per-tenant config and custom schema, (4) tenant placement, skew, and migrations. Isolation threads through all four.
1. The isolation model: schema-per-tenant vs row-level security
This is the decision the whole design turns on. Every tenant’s data must be isolated, but the three obvious models trade off isolation strength against density, operability, and query performance in opposite directions.
Approach A: database-per-tenant (the maximal-isolation naive answer)
Give every tenant its own database (or its own server). Hard physical isolation; a tenant’s data is in its own files; noisy neighbors are impossible; a per-tenant restore is trivial.
Where it breaks:
- Cost and density. 55,000 databases means 55,000 connection pools, 55,000 sets of background processes (vacuum, WAL, checkpointer), 55,000 things to monitor. A median tenant has 300 records; running a whole database engine for 300 rows is wildly wasteful. Idle overhead alone sinks the economics.
- Connection explosion. An app node that may serve any tenant needs a connection to any of 55,000 databases. Even with pooling, you cannot hold 55K warm pools per node. You end up with connection churn that dominates latency.
- Fan-out operations die. A platform-wide schema migration (ship a new column for everyone) is now 55,000 migrations, each of which can fail independently. Cross-tenant analytics means querying 55,000 databases. Operability collapses at this count.
Database-per-tenant is correct for a handful of whales, not for the long tail.
Approach B: shared schema, shared table, tenant_id column, filtered in application code (the naive dense answer)
One set of tables; every row has a tenant_id; the app adds WHERE tenant_id = ? to every query. Maximal density, trivial migrations (one ALTER TABLE), one connection pool.
Where it breaks:
- One forgotten
WHEREleaks everything. Isolation depends on every query, in every code path, forever, remembering the filter. One missed clause in a reporting endpoint, one ORM default scope bypassed, one hand-written admin query - and tenant A sees tenant B’s data. This is the single most common multi-tenant breach, and “we promise the app always filters” is exactly what a compliance auditor will reject. - Shared indexes and hot tables. A whale with 20M rows and the free-trial tenants share the same B-tree. The whale’s scans pollute the shared buffer cache; its bulk import bloats the shared table and its indexes;
VACUUMon the giant table affects everyone. - No per-tenant restore. Point-in-time restoring one tenant means extracting its rows from a shared table while everyone else keeps writing - there is no clean boundary to restore.
Density without an isolation guarantee is a liability.
Approach C: shared schema with database-enforced Row-Level Security, pooled onto shards, whales broken out to dedicated databases (the answer)
Combine the two with the isolation enforced by the database, not the application, and use placement to handle skew.
Row-Level Security (RLS) moves the tenant_id filter out of application code and into the database as a mandatory policy:
-- Every tenant-owned table carries tenant_id and enables RLS.
CREATE TABLE records (
tenant_id BIGINT NOT NULL,
record_id BIGINT NOT NULL,
data JSONB NOT NULL,
created_at TIMESTAMPTZ NOT NULL DEFAULT now(),
PRIMARY KEY (tenant_id, record_id)
);
ALTER TABLE records ENABLE ROW LEVEL SECURITY;
ALTER TABLE records FORCE ROW LEVEL SECURITY; -- applies even to table owner
-- The policy: a session can only see rows for its current tenant.
CREATE POLICY tenant_isolation ON records
USING (tenant_id = current_setting('app.current_tenant')::BIGINT);
On each request the app sets the session’s tenant once, inside the transaction, and then issues normal queries - the database appends the tenant predicate automatically:
BEGIN;
SET LOCAL app.current_tenant = '8812'; -- from the signed JWT, never the body
SELECT * FROM records WHERE record_id = 42; -- DB rewrites to also require tenant_id=8812
-- a query that FORGETS tenant_id still returns ONLY tenant 8812's rows
COMMIT;
Why this is the answer:
- Isolation is a database invariant, not an application convention. Even a buggy or hand-written query cannot cross tenants, because the policy is applied by the engine on every table access. This is the property you can hand an auditor: “the database itself forbids cross-tenant reads.”
FORCE ROW LEVEL SECURITYensures even the table owner is subject to it. - Density of the shared-schema model is preserved. One connection pool per shard, one set of tables, one migration to add a column for everyone. Thousands of small tenants pack onto a shard efficiently.
- The leading column
tenant_idmakes isolation also a performance win. WithPRIMARY KEY (tenant_id, record_id)and all secondary indexes prefixed bytenant_id, a tenant’s rows are physically clustered and every index seek starts by narrowing to one tenant - so one tenant’s data volume barely affects another’s lookup. The RLS predicate aligns with the index, so it is close to free.
But RLS alone does not fix skew or blast radius, so layer placement on top:
| Tenant class | Isolation mechanism | Where it lives |
|---|---|---|
| Long tail (small) | Shared schema + RLS | Pooled onto shared shards, many per shard |
| Mid-size | Shared schema + RLS | Fewer per shard, or own shard |
| Whale / regulated | Dedicated database (or dedicated schema) | Its own instance, physical isolation |
The Tenant Router stores each tenant’s isolation_mode. A tenant can be promoted: a free-trial tenant starts pooled under RLS; when it grows (or signs an enterprise contract demanding physical isolation or data residency), the control plane migrates it to a dedicated database and flips its routing entry. The application code does not change - it always sets tenant context and queries normally; only the connection target and whether RLS is “the guard” or “redundant with physical isolation” differ.
Why not schema-per-tenant as the default? A separate Postgres schema per tenant (same database, CREATE SCHEMA tenant_8812) gives stronger logical isolation than RLS and keeps one instance, but: the system catalog groans at tens of thousands of schemas and their table copies (per-schema table bloat in pg_catalog), search_path switching per request is error-prone, and a platform migration must loop over every schema. Schema-per-tenant is a reasonable middle option for hundreds of mid-size tenants, but RLS is denser for the 50,000-tenant long tail and dedicated DBs are cleaner for the handful of whales. So the production answer is a spectrum keyed by tenant size and compliance need, with RLS as the dense default.
denser / cheaper ◀─────────────────────▶ stronger isolation
shared table RLS on shared schema-per- database-per-
(app filter) schema (default) tenant tenant (whales)
[rejected: [long tail, [some mid-size, [regulated /
leaks] 50K tenants] hundreds] largest, dozens]
2. Per-tenant rate limiting and noisy-neighbor control
The isolation NFR has a performance axis: tenant A’s traffic or batch job must not degrade tenant B. RLS isolates data; it does nothing for load. Rate limiting and quotas are the mechanism.
Approach A: a global rate limiter and shared resources (the naive answer)
One global limit on the platform, first-come-first-served access to the shared DB and worker pool.
Where it breaks: a single tenant’s CSV import (40,000 writes/minute) or a tenant’s runaway integration polling the API in a loop consumes the shared connection pool, the shared worker threads, and the shared DB IO. Every other tenant’s p99 spikes. A global limit does not help - the noisy tenant is “within” the global limit while locally monopolizing resources. This is the noisy-neighbor problem, and it is the second most common multi-tenant failure after data leaks.
Approach B: per-tenant token buckets at the edge + per-tenant work isolation (the answer)
Enforce limits per tenant at the earliest point, and isolate background work by tenant.
Rate limiting at the gateway with a per-tenant token bucket in Redis, keyed by tenant_id and tier:
key: ratelimit:{tenant_id}
bucket: { tokens, last_refill_ts, capacity, refill_rate } # tier-derived
on each request (atomic Lua script in Redis, O(1)):
refill tokens by (now - last_refill_ts) * refill_rate, cap at capacity
if tokens >= 1: tokens -= 1 ; allow
else: return 429 with Retry-After = time to next token
- Token bucket per tenant, tier-parameterized. A free tenant gets capacity 60/min, an enterprise tenant 10,000/min. The bucket’s capacity and refill rate come from the tenant’s plan in the Policy Cache. Enforced at the edge, before any DB work, so a throttled tenant costs almost nothing.
- Concurrency budgets, not just rate. Beyond requests/sec, cap concurrent in-flight expensive operations per tenant (e.g. at most N simultaneous report generations). A semaphore per tenant prevents one tenant from occupying the whole worker pool even if its request rate is legal.
- Background work is per-tenant queued and throttled. Bulk imports, exports, and heavy jobs go onto a work queue partitioned by
tenant_idwith a per-tenant concurrency limit, so a 10M-row import drains at a bounded rate and never becomes the platform’s write peak. The 40,000-writes/minute import is admitted at, say, 2,000/min for that tenant - slower for them, invisible to everyone else. - Resource isolation at the DB. Pooled shards cap connections per tenant (a tenant cannot grab the whole pool). Whales on dedicated DBs are physically isolated, so their load cannot touch anyone. For pooled tenants, statement timeouts and per-tenant query-concurrency limits in the app tier stop a pathological query from pinning a shard.
Request ──▶ [Gateway]
├─ token bucket (tenant tier) ── over? ─▶ 429 Retry-After
├─ concurrency semaphore (tenant) ─ full? ─▶ 429/queue
└─ allow ─▶ App ─▶ per-tenant connection cap on shard
Bulk job ──▶ [Per-tenant work queue, bounded concurrency] ─▶ workers
The guarantee this buys: a tenant can exhaust its own budget and get throttled, but cannot reach into the shared pool and degrade a neighbor. Fairness is enforced at every shared resource - gateway, worker pool, connection pool - keyed by tenant_id.
3. Per-tenant config and custom schema
Each tenant has its own feature flags, SSO config, branding, rate-limit tier, data-residency region, and - the hard part - custom fields / schema extensions (one tenant adds a cost_center field, another adds risk_rating). This config is read on essentially every request and must cost nothing.
Approach A: read tenant config from the database on each request (the naive answer)
On every request, SELECT * FROM tenant_config WHERE tenant_id = ?.
Where it breaks: at 33K req/sec peak, that is 33K extra DB reads/sec purely to answer “what are this tenant’s settings,” doubling DB load for data that changes maybe once a week. And if custom fields are modeled as real columns per tenant, you hit the same schema-explosion wall a shared table does (Postgres caps at ~1,600 columns; the union of all tenants’ custom fields is tens of thousands).
Approach B: cache-resident config + semi-structured custom fields (the answer)
Config is small and rarely changes, so hold all of it in memory. The entire config set is ~275 MB (BoE). Each app node keeps an in-process cache of tenant config, backed by Redis and the control-plane store. Reads are a local map lookup - zero network, zero DB. On a config change, the control plane writes the authoritative row and publishes an invalidation on the bus; every node refreshes that one tenant’s entry within seconds. Config is read-mostly and tolerant of a few seconds of staleness, which is exactly the workload a push-invalidated local cache is best at.
Table: tenant_config (control plane, authoritative)
tenant_id BIGINT PRIMARY KEY
plan_tier TEXT -- free | pro | enterprise
region TEXT -- data residency home region
flags JSONB -- { "new_ui": true, "api_v2": false }
sso JSONB -- { "type":"saml", "idp_url":"...", "cert":"..." }
rate_limits JSONB -- { "rps": 10000, "concurrency": 50 }
branding JSONB -- theme, logo object-key
custom_fields JSONB -- field catalog (see below)
updated_at TIMESTAMPTZ
Custom fields are stored semi-structured, not as per-tenant columns. The tenant’s records carry a data JSONB column; the tenant’s custom_fields catalog in config declares each field’s name, type, and validation. The app validates incoming writes against the catalog and stores values inside data, so adding a custom field is a catalog edit (a config write), never a DDL migration on a shared table:
custom_fields catalog (per tenant, in tenant_config):
[ { "key":"cost_center", "type":"string", "required":false },
{ "key":"risk_rating", "type":"enum", "options":["low","med","high"] } ]
record.data example: { "cost_center":"CC-200", "risk_rating":"high", ... }
For tenants that need to filter/sort on custom fields at scale, index them the way the Jira-style design does: project records into a per-tenant search index (ES/OpenSearch, routed by tenant_id) where each declared custom field becomes a typed indexed field, so custom-field queries are indexed lookups rather than JSON scans. The relational store stays the consistent source of truth; the index serves custom-field search.
This keeps the two properties that matter: config lookups are free (cache-resident), and the schema is per-tenant dynamic without any per-tenant DDL on shared tables.
4. Tenant placement, skew, and migrations
55K tenants with a 60,000:1 size ratio share a finite set of shards. Placement decides which tenant lives where, and migrations/onboarding must not break the blast-radius guarantee.
Approach A: hash tenant_id to a fixed shard (the naive placement)
shard = hash(tenant_id) % N. Simple, stateless routing.
Where it breaks: hashing is size-blind. A whale hashes to a shard that also holds 9,000 small tenants, and now that shard is hot while others idle. You cannot move the whale without changing the hash (which reshuffles everyone). Skew plus fixed hashing guarantees hot shards, and the whole point of multi-tenancy is defeated when one shard carries a whale’s entire load.
Approach B: a directory-based router with size-aware placement and live migration (the answer)
Make routing a lookup, not a hash, so any tenant can live anywhere and move independently.
Table: tenant_routing (control plane, cached at the edge)
tenant_id BIGINT PRIMARY KEY
region TEXT -- home region (data residency)
isolation_mode TEXT -- pooled_rls | dedicated_schema | dedicated_db
shard_id TEXT -- which instance / cluster
status TEXT -- active | migrating | suspended
migrate_to TEXT NULL -- target shard during a live move
- Size-aware placement. The Provisioning service places a new small tenant on the least-loaded pooled RLS shard; a known enterprise onboarding (whale, or data-residency/compliance requirement) is placed directly on a dedicated DB in the required region. Placement is a decision, not a hash, so it can account for current shard load and tenant class.
- Promotion / live migration. When a pooled tenant grows past a threshold (rows, QPS, or a contract change), the control plane migrates it to a dedicated DB: snapshot the tenant’s rows (trivial because RLS/the
tenant_idprefix gives a clean boundary -WHERE tenant_id = ?), replicate the tail, setstatus=migratingso writes dual-write or briefly pause, flipshard_id, verify, then drop the old rows. Only that one tenant is touched; everyone else is unaffected. This is the payoff of a leading-tenant_idkey: a single tenant is a cleanly extractable, independently restorable unit. - Routing is cache-resident. The edge caches the routing table (55K small rows) so resolving
tenant_id → shardis a sub-ms local lookup, refreshed on change via the bus. - Blast-radius-bounded migrations. A platform schema migration (add a column everywhere) is rolled out shard by shard, not globally at once. A bad migration breaks one shard’s tenants, is caught, and is rolled back before reaching the rest. Dedicated-DB whales are migrated individually with extra care. The migration runner is part of the control plane and tracks per-shard version, so shards can even run mixed versions briefly behind a backward-compatible app.
The result: skew is handled by placing each tenant according to its size and compliance class, any tenant can move without disturbing others, and operational changes are scoped to a shard so the blast radius is always a subset of tenants, never all of them.
API Design & Data Schema
API
Tenant-facing APIs never take tenant_id as a parameter - it comes from the signed token. Control-plane APIs take it explicitly and require platform-admin auth.
# --- Tenant-facing (tenant_id from JWT, enforced by RLS) ---
GET /api/v1/records?filter=<...>&cursor=<...>&limit=50
Response 200:
{ "records": [ { "record_id": "...", "data": {...} }, ... ],
"nextCursor": "..." } // cursor paging (last-seen key)
Errors: 401, 403, 429 (rate limited, with Retry-After)
POST /api/v1/records
Body: { "data": { "name": "Acme deal", "cost_center": "CC-200" } }
// custom fields validated against the tenant's field catalog
Response 201: { "record_id": "...", "created_at": "..." }
Errors: 400 invalid custom field/type, 401, 403, 429
PUT /api/v1/records/{id} Body: { "data": { ... } } -> 200
GET /api/v1/me/config -> tenant's flags, branding, custom-field schema
# --- Control plane (platform-admin auth; tenant_id explicit) ---
POST /admin/v1/tenants
Body: { "name": "Acme Bank", "plan": "enterprise",
"region": "ap-south-1", "isolation": "dedicated_db" }
Response 201: { "tenant_id": "8812", "endpoint": "...", "status": "active" }
PATCH /admin/v1/tenants/{id}/config Body: { "flags": {...}, "rate_limits": {...} }
-> 200, pushes invalidation to all app nodes
POST /admin/v1/tenants/{id}/migrate Body: { "to": "dedicated_db", "region": "..." }
-> 202 Accepted (async live migration)
POST /admin/v1/tenants/{id}/backup -> 202 (per-tenant snapshot)
POST /admin/v1/tenants/{id}/restore Body: { "point_in_time": "..." } -> 202
GET /admin/v1/tenants/{id}/usage?period=2026-11 -> metered usage for billing
DELETE /admin/v1/tenants/{id} -> 202 (export + schedule deletion)
Writes are idempotent via a client Idempotency-Key header so a retried create does not duplicate a record. Paging is cursor-based (last-seen (tenant_id, record_id) tuple) so deep pages are stable and cheap - and the cursor is tenant-scoped, so it can never page into another tenant.
Data store choices
Different access patterns, so more than one store. Be explicit.
1. Tenant data store - relational with RLS, pooled onto shards, whales on dedicated DBs (Postgres / a NewSQL like CockroachDB/Spanner). Tenant records need per-tenant strong consistency (read-your-writes), transactional writes, and - above all - database-enforced isolation. RLS gives the isolation invariant; the leading tenant_id key gives clustering and clean per-tenant extraction; sharding + dedicated DBs for whales handle skew. NewSQL is attractive when you want transparent horizontal scale and multi-region homing without hand-rolled sharding, at some latency cost.
records (tenant_id, record_id, data JSONB, created_at, updated_at)
-- PK (tenant_id, record_id), RLS enabled+forced,
-- all secondary indexes prefixed by tenant_id
audit_log (tenant_id, seq, actor, action, before, after, ts)
-- append-only, clustered by tenant_id, ages to cold store
outbox (tenant_id, event_id, payload, published_at NULL)
-- transactional outbox for record_changed + usage events
2. Control-plane store - relational (its own small, highly-available DB). Holds tenant_routing, tenant_config, plan definitions, migration versions, and provisioning state. This is the brain; it is small (tens of thousands of rows) but must be rock-solid and is the source of truth the caches hydrate from.
3. Policy / config cache - in-process + Redis. The full config and routing set (~275 MB) resident on every app node for zero-cost per-request lookups, backed by Redis, invalidated via the bus. Rebuildable from the control-plane store.
4. Rate-limit counters - Redis. Per-tenant token buckets and concurrency semaphores, keyed by tenant_id, updated by an atomic O(1) script per request. Ephemeral by nature.
5. Per-tenant search index - Elasticsearch / OpenSearch. For tenants that filter/sort on custom fields at scale; each ticket/record is a document routed by tenant_id, custom fields typed from the tenant’s catalog. Derived, rebuildable from the tenant data store.
6. Event bus - Kafka, partitioned by tenant_id. Carries record_changed and usage events for metering, audit, and analytics. Partitioning by tenant_id preserves per-tenant ordering and keeps a tenant’s event processing isolated.
7. Backups + analytics - object storage + OLAP. Per-tenant snapshots/WAL for independent restore; cross-tenant analytics and billing rollups in a separate OLAP store fed off the bus (eventually consistent, never on the serving path).
Why RLS-on-shared-schema as the default, dedicated DBs for whales, cache-resident control plane: the long tail wants density with a database-enforced isolation guarantee (RLS), the whales want physical isolation and residency (dedicated DBs), and the per-request policy/routing wants to be free (cache). Partitioning everything by tenant_id makes each tenant a clean, isolated, movable unit - the foundation of all three isolation axes. A single shared table with app-level filtering would be denser still but cannot be defended to an auditor and offers no per-tenant blast-radius control - the classic mistake.
Bottlenecks & Scaling
Where it breaks first, in order, and the fix for each:
1. Cross-tenant data leaks (the primary correctness risk). App-level WHERE tenant_id filtering fails the first time a query forgets the clause.
Fix: enforce isolation in the database with FORCE ROW LEVEL SECURITY; set the tenant via a session variable from the signed token, never from request input. The DB rewrites every query to its tenant, so a forgotten filter cannot leak. Covered in deep dive 1.
2. Noisy neighbors degrading shared latency. One tenant’s import or runaway integration monopolizes the connection pool, worker threads, and DB IO.
Fix: per-tenant token buckets and concurrency semaphores at the edge, per-tenant connection caps on shards, and bulk work on per-tenant throttled queues. Fairness enforced at every shared resource, keyed by tenant_id. Covered in deep dive 2.
3. Config/routing lookups doubling DB load. Reading tenant settings and shard from the DB on every request. Fix: the entire config + routing set (~275 MB) is cache-resident in every app node, push-invalidated on change. Per-request policy lookup becomes a local map read - zero DB cost. Covered in deep dive 3.
4. Schema explosion from per-tenant custom fields. A real column per tenant’s custom field blows past the column limit and requires DDL per change.
Fix: store custom fields in a data JSONB column validated against a per-tenant catalog; index them in a per-tenant search index for filter/sort. Adding a field is a config edit, not a migration. Covered in deep dive 3.
5. Skew and hot shards. A whale hashed onto a shared shard swamps its neighbors; fixed hashing cannot move it.
Fix: directory-based, size-aware placement; promote whales to dedicated DBs via live per-tenant migration; the leading tenant_id key makes a tenant a cleanly extractable unit. Covered in deep dive 4.
6. Blast radius of migrations and bad config. A global schema migration or a bad flag rollout breaks every tenant at once. Fix: roll migrations shard by shard with per-shard versioning; roll config/flags out progressively (canary tenants first); dedicated-DB whales migrated individually. A failure is always scoped to a subset of tenants. Covered in deep dive 4.
7. Connection-pool pressure from many tenants per node. An app node serving any tenant cannot hold a warm pool to every shard.
Fix: pool per shard, not per tenant (RLS means all tenants on a shard share one pool, differentiated only by the session variable); cap per-tenant connections within the pool; use a connection proxy (PgBouncer-style) in transaction-pooling mode so SET LOCAL tenant context is scoped to the transaction and connections are reused safely across tenants.
8. Per-tenant backup/restore on a shared store. Restoring one tenant from a shared table is messy.
Fix: the tenant_id prefix gives a clean extraction boundary - a per-tenant logical snapshot is WHERE tenant_id = ?; whales on dedicated DBs restore natively. Point-in-time restore for one tenant never touches another.
9. Usage metering accuracy under load. Billing must not drop or double-count usage events.
Fix: emit usage via the transactional outbox (same transaction as the write), process with at-least-once delivery and idempotent aggregation keyed by event_id, into an OLAP rollup. Metering lags reality by seconds, which billing tolerates; it is never on the serving path.
10. Single points of failure and multi-region. A central router or a single region is a global choke point; enterprises demand data residency. Fix: app/gateway tiers are stateless behind load balancers, scaled horizontally; the control-plane DB, Redis, Kafka, and shards run RF≥3; the routing/config caches mean the edge keeps serving even if the control plane blips. Each tenant is homed in a region (residency is a compliance requirement), so its data store, cache, and index live in that region with no cross-region hot path; cross-region is for DR replication, not live serving.
Wrap-Up
The trade-offs that define this design:
- Isolation is a spectrum keyed by tenant size and compliance, with RLS as the dense default. The long tail of 50,000 tenants runs on a shared schema with database-enforced Row-Level Security - isolation becomes an engine invariant, not an application convention you hope every query honors. The handful of whales and regulated tenants get dedicated databases for physical isolation and residency. A single shared table with app-level filtering is denser but indefensible to an auditor and offers no blast-radius control; database-per-tenant for everyone is secure but economically absurd at 55K tenants.
- Partition everything by
tenant_id. The leading key clusters each tenant’s data, aligns with the RLS predicate (so isolation is nearly free), and makes a tenant a cleanly extractable, movable, independently restorable unit - the foundation of all three isolation axes. - Noisy neighbors are controlled at every shared resource. Per-tenant token buckets and concurrency semaphores at the edge, per-tenant connection caps, and throttled per-tenant bulk queues mean a tenant can exhaust its own budget but never reach into the pool and degrade a neighbor.
- Per-request policy is free. The entire config + routing set is cache-resident in every app node, push-invalidated on change, so resolving a tenant’s shard, limits, flags, and custom-field schema costs a local map read, not a DB round trip.
- Blast radius is always bounded. Directory-based placement, shard-by-shard migrations, progressive config rollout, and region-homed tenants ensure a bad change, a hot shard, or an outage hits a subset of tenants - never all of them.
One-line summary: a shared, stateless data plane where every request carries a signed tenant_id that drives database-enforced Row-Level Security on tenant_id-partitioned shards (whales promoted to dedicated DBs), with cache-resident per-tenant config/routing, per-tenant token-bucket rate limiting, and a control plane that places, migrates, meters, and backs up each tenant as an isolated unit - so thousands of enterprise tenants share one platform while staying isolated in data, performance, and blast radius.
Comments