Everyone thinks serverless is simple. “You upload a function, we run it when an event comes in, you pay per request - it is a glorified cron plus an HTTP handler.” That mental model survives exactly until you ask the two questions that actually define the system. First: when a request arrives for a function that has zero running instances - because it has not been called in an hour and you scaled it to nothing - where does the code run, and how fast can you get a sandboxed, isolated execution environment up and warm? That is the cold start, and if it takes two seconds the product is dead on arrival. Second: you are not hosting one customer’s function, you are hosting millions of distinct functions from hundreds of thousands of tenants, almost all idle at any instant, and you have a finite fleet of machines. How do you keep enough environments warm to make cold starts rare, without dedicating a machine to every function (you would need millions of idle machines and go bankrupt), while fairly sharing those warm environments so one customer’s traffic spike does not starve everyone else? That is the real system: a warm-pool multiplexing problem over a shared, oversubscribed fleet.
A serverless functions platform - the thing under AWS Lambda, Google Cloud Functions, Azure Functions, Cloudflare Workers, or an internal FaaS - is deceptive because the happy path (a warm function gets an event, runs for 40ms, returns) is trivial, and the entire actual difficulty is in the states around it: the cold start when nothing is warm, the scale-to-zero economics that make the business work, the autoscaling from zero to thousands of concurrent instances in seconds when a function goes viral, the placement decision of which host should run this invocation, and the isolation that lets you pack hundreds of different tenants’ code onto one machine without them seeing each other. Get the warm pool and cold start wrong and you have either a slow product or a bankrupt one.
Let me do it properly.
Functional Requirements (FR)
In scope:
- Deploy a function. A developer uploads code (a handler plus dependencies) for a supported runtime (Node, Python, Go, Java, a custom container), with config: memory size, timeout, environment variables, concurrency limits, and IAM/role. We version it, build a deployable artifact, and make it invocable. This is the entry point.
- Trigger a function on events. A function runs in response to an event: a synchronous HTTP/API call (caller waits for the response), an asynchronous event (a message dropped on a queue, an object uploaded to storage, a scheduled timer), or a stream (records from a log/stream processed in batches). The platform routes the event to the function and invokes it.
- Execute in an isolated, ephemeral environment. Each invocation runs in a sandbox that cannot see the host, other tenants, or the network beyond what the function’s role allows. The environment is short-lived; the function runs to completion or to its timeout and returns a result (or an error).
- Scale automatically, including to zero. Concurrency scales with load: zero instances when idle (you pay nothing), and up to thousands of concurrent instances within seconds when traffic spikes - with no capacity planning by the developer. Scaling to zero is a first-class requirement, not an afterthought; it is what makes serverless serverless.
- Meter and bill per invocation. Record per-invocation duration (to the millisecond), memory allocated, and count, because billing is
GB-secondsplus requests. Metering must be accurate - it is the revenue - and auditable. - Return structured results and logs. For sync calls, the function’s return value or error, plus the status. For all calls, structured logs, metrics (duration, errors, throttles, cold-start count), and traces are captured and shipped to the observability layer.
Explicitly out of scope (say it so you own the scope):
- Long-running or always-on services. We run functions that start, do a unit of work, and finish within a bounded timeout (say up to 15 minutes). We are not hosting a persistent web server, a stateful daemon, a WebSocket connection that lives for hours, or a background worker that never exits. Those are a container/VM product (a PaaS or orchestrator), with different lifecycle and billing. We note where long-lived connections hook in but do not build them.
- Durable in-function state. A function instance is ephemeral and stateless by contract - anything it writes to local disk or memory vanishes when the environment is reclaimed, and the next invocation may land on a completely different instance. State lives in external stores (a database, object storage, a cache). We do not promise sticky sessions or persistent local state between invocations (we exploit reuse as an optimization, but never as a guarantee).
- The developer’s build pipeline / source control. We accept a deployable artifact (a zip or a container image) and config. CI, tests, and source management live above us.
- Guaranteed ordering across invocations. For stream/async triggers we offer at-least-once delivery and batch processing, but we do not globally order concurrent invocations of the same function. Ordering within a shard/partition is the strongest guarantee we make.
- Perfect elimination of cold starts. We make cold starts rare and fast, not nonexistent. A function that has been idle for an hour, or that scales from N to N+1 concurrent instances, will sometimes pay a cold start. We minimize frequency and latency and offer provisioned warm capacity as an opt-in, but we are honest that scale-to-zero and zero-cold-start are in tension.
The one decision that drives everything: the platform does not dedicate capacity to functions, it multiplexes a shared fleet of warm execution environments across millions of mostly-idle functions, and the product’s whole quality rests on how often an incoming event finds a warm environment versus paying a cold start. Every other choice - the event routers, the placement service, the scaling controller, the isolation technology - exists to make warm hits frequent and cold starts fast and cheap.
Non-Functional Requirements (NFR)
- Scale: on the order of millions of distinct functions across hundreds of thousands of tenants, the vast majority idle at any instant; tens of billions of invocations/day in aggregate; a single hot function able to burst to thousands of concurrent instances in seconds. The defining shape is extreme fan-out of tiny, short, spiky workloads - most functions run for tens to a few hundred milliseconds.
- Latency: a warm invocation adds near-zero platform overhead (single-digit ms of routing on top of the function’s own runtime). A cold start must be sub-second - target p50 cold start well under 500ms and p99 under ~1s for interpreted runtimes - because cold starts are user-visible on the synchronous path. End-to-end sync p99 (warm) should be the function’s duration plus a few ms.
- Availability: 99.95%+ on accepting and dispatching invocations. The event-ingest and routing layer must not be a single point of failure; a dead host must not drop in-flight async events (they are redelivered). Sync calls degrade gracefully (throttle with a clear 429, do not hang).
- Isolation / multi-tenancy (a core NFR): one tenant’s function cannot read or write another tenant’s memory, filesystem, or environment; cannot see or signal other tenants’ processes; and cannot reach the network except where its own role permits. Hundreds of distinct tenants may be packed on one host, so the isolation boundary between co-located functions must be strong (VM-grade), not just a shared-kernel container.
- Elasticity / scale-to-zero: idle functions consume zero billable compute and near-zero platform resource (just metadata). Scaling from zero to serving, and from N to many, happens automatically within seconds, with no pre-provisioning required from the developer.
- Fairness: no single function or tenant can monopolize the warm pool or the fleet. A tenant whose function suddenly bursts to 10,000 concurrent must not evict every other tenant’s warm environments or exhaust shared capacity. Per-tenant concurrency limits and fair capacity allocation are mandatory, not optional.
- Billing accuracy: per-invocation duration metered to the millisecond, memory accounted, counts exact. Over-bill and you lose trust; under-bill and you lose money. Metering is on the critical path and must be precise and tamper-resistant (the tenant’s own code cannot influence it).
Back-of-the-Envelope Estimation (BoE)
Let me ground the design in numbers.
Functions, invocations, and the idle shape:
Distinct functions: 10,000,000 deployed across all tenants
Invocations: 20,000,000,000 /day (20B)
Average rate: 20B / 86,400s = ~230,000 invocations/sec steady
Average function duration: ~100 ms (most are short; tail runs to minutes)
Concurrency (Little's law): 230,000/s * 0.1s = ~23,000 concurrent executions
steady-state. But that is the AVERAGE; the fleet must handle bursts
far above it, and the 23,000 are spread over a long tail of functions.
The first non-obvious number: of 10M functions, only a tiny fraction are active in any given second. If 23,000 executions are concurrent on average and each function runs ~100ms, the set of functions actually executing right now is small - maybe tens of thousands - while the other ~9.97M are idle. You cannot keep a warm environment per function (10M warm environments, even tiny, is millions of machines of idle RAM - economically impossible). The whole game is deciding which functions to keep warm, for how long, on a shared fleet.
The cold-start blast radius:
Cold-start rate: suppose 5% of invocations are cold (a good system; a
bad one is far higher). 5% * 230,000/s = ~11,500 cold starts/sec steady,
with huge spikes when many idle functions wake at once (a 9am cron storm).
Cold start cost: creating an isolated environment + loading the runtime +
loading the function code + (for some) initializing dependencies
= 100ms-1s depending on technique; this is the dominant controllable cost.
Fleet sizing and packing:
Per host: say 128 GB RAM, 32 vCPU.
Function memory: average configured size ~256 MB (ranges 128MB-10GB).
Packing: 128 GB / 256 MB = ~500 environments per host by memory,
but most are IDLE warm environments holding RAM, not running. A host
might hold ~300-500 warm micro-environments and run ~30-60 at once
(CPU-bound), heavily oversubscribing CPU because idle functions use none.
Concurrent executions: ~23,000 steady / (say 40 active per host)
= ~600 hosts actively executing, but you keep MORE hosts holding
warm pools for the long tail. Total fleet is sized for warm-pool
capacity + burst headroom, not just average concurrency.
Storage, metering, bandwidth:
Function artifacts: 10M functions * avg ~20 MB (code + deps, some much larger
container images) = ~200 TB of artifacts in object storage, immutable,
content-addressed, cached near workers. Pulled on cold start, never on
the warm path.
Metering records: 20B invocations/day, each a small record (function id,
tenant, start, duration_ms, memory, billed GB-s, cold-start flag) ~100 B
-> 20B * 100 B = ~2 TB/day of metering/billing data. Aggregated and rolled.
Logs/metrics: far larger (function stdout/traces); sampled, batched,
shipped async to the observability pipeline, never on the invocation path.
Routing metadata: function config + current placement (which hosts hold warm
instances) is small per function but read on EVERY invocation - must be a
fast, cached, highly-available lookup.
The headline shape: millions of mostly-idle functions, tens of billions of tiny spiky invocations, where the scarce resources are warm-environment RAM on a shared fleet and cold-start latency. Size the fleet for warm-pool capacity plus burst headroom (not average concurrency), make cold starts rare (keep the right functions warm) and fast (snapshot/pre-init), oversubscribe CPU hard because idle functions use none, cache artifacts and routing metadata off the hot path, and meter every millisecond because that is the revenue.
High-Level Design (HLD)
The system splits into an event/invoke frontend (API gateway + async event sources + stream pollers) that receives triggers and turns them into invocation requests, a routing / placement service that for each invocation decides where it runs - finding a warm instance or asking for a new one, a scaling controller (the “placement and scaling brain”) that owns per-function instance counts, decides when to create and when to reap environments, and enforces per-tenant limits, a worker fleet of hosts each running a local worker agent that manages a pool of isolated micro-sandboxes (Firecracker-style microVMs), an artifact store (function code/images, content-addressed, cached on workers), a metadata store (function config, versions, placement), and a metering/billing pipeline plus an observability pipeline fed asynchronously.
TRIGGERS
┌──────────────┬──────────────────┬────────────────────┐
│ HTTP/API │ async event │ stream / schedule │
│ (sync, caller│ (queue, storage │ (log shards, cron) │
│ waits) │ event -> queue) │ │
└──────┬───────┴────────┬─────────┴─────────┬──────────┘
│ 1. invoke │ (buffered, retried) │ (polled in batches)
▼ ▼ ▼
┌──────────────────────────────────────────────────────────┐
│ INVOKE FRONTEND (stateless, multi-AZ) │
│ authn/authz, per-tenant throttle, pick function+version │
└───────────────────────────┬──────────────────────────────┘
2. "run function F" │ (reads function config from metadata)
▼
┌──────────────────────────────────────────────────────────┐
│ PLACEMENT / ROUTER + SCALING CONTROLLER │
│ - find a WARM idle instance of F on some host (route) │
│ - none free? within concurrency limit? -> COLD START: │
│ pick a host with capacity, tell its agent to create │
│ - over per-tenant/function limit -> THROTTLE (429) │
│ - owns desired instance count, warm-pool TTL, reaping │
└───────────┬───────────────────────────────┬──────────────┘
reads/writes placement │ 3. dispatch to host+slot
┌──────────▼─────────┐ ▼
│ METADATA STORE │ ┌──────── WORKER FLEET (packed, oversubscribed) ───────┐
│ (config, versions, │ │ host (128 GB, 32 vCPU), worker AGENT │
│ placement map) │ │ ┌──── warm pool for many DISTINCT functions ────┐ │
└────────────────────┘ │ │ microVM(F1, warm idle) microVM(F2, running) │ │
┌────────────────────┐ │ │ microVM(F1, running) microVM(F3, warm) │ │
│ ARTIFACT STORE │──▶│ │ ... each: own guest kernel, cgroups, role, │ │
│ (code/images, │ │ │ seccomp, no cross-tenant visibility │ │
│ content-addressed)│ │ └───────────────────────────────────────────────┘ │
└────────────────────┘ │ 4. run handler(event) -> result; meter ms+MB │
└──────────────┬───────────────────────────────────────┘
│ 5. result (sync) / ack (async)
▼ + async: logs, metrics, metering
┌───────────────┬──────────────────┐
│ METERING/BILL │ OBSERVABILITY │
└───────────────┴──────────────────┘
Request flow - a synchronous HTTP invocation (the latency-critical path):
- An HTTP trigger hits the invoke frontend (behind the API gateway). It authenticates, authorizes, checks the tenant’s and function’s concurrency/rate limits, and resolves which function + version to run (from cached metadata). If over the limit, it returns
429immediately - back-pressure at the edge. - The frontend asks the placement/router for an execution slot for function
F. The router consults the placement map: is there a warm, idle instance ofFon some host right now? - Warm hit (the common, fast case): the router hands back a
(host, slot)for an idle warm microVM ofF. The frontend dispatches the event there. The worker agent marks that microVM busy, runs the handler with the event payload, and streams back the result. Platform overhead is a few ms; total latency is the function’s own duration. The microVM is returned to the warm pool (idle) afterward, ready for the next call. - Cold miss: no warm instance, but the tenant is under its concurrency limit, so the scaling controller picks a host with spare capacity, tells its agent to create a new microVM for
F(boot/restore the sandbox, pull the artifact from local cache or the artifact store, load the runtime, run the function’s init code), then run the handler. This is the cold start; the whole design works to make it rare and sub-second. The new instance stays warm after, added to the pool for subsequent calls. - The worker meters the invocation (billed duration to the ms, memory, cold-start flag), returns the result on the sync path, and asynchronously ships logs/metrics/metering to the pipelines (never blocking the response).
Request flow - an asynchronous / event-source invocation: a message on a queue, an object-storage event, or a stream record. The event is durably buffered (an internal queue per function or a shared bus); a poller pulls batches and invokes F through the same placement path, but the caller does not wait - delivery is at-least-once with retries and a dead-letter queue after max attempts. Here cold starts are hidden from the end user (latency is not user-facing), so the controller can be lazier about keeping these functions warm. Scheduled (cron) triggers and stream shards fan in the same way.
The key architectural insight: invocation is a placement problem, not a compute problem. The frontend never runs code; the scaling controller owns the global view of “how many instances of each function exist and where,” and makes two decisions per function - route an event to an existing warm instance, or create a new one (cold start) within limits, or throttle. All compute danger and all packing live on the worker fleet, where many distinct tenants’ microVMs are multiplexed on one oversubscribed host. The warm pool - instances kept alive between invocations - is the thing the whole system optimizes: keep it big enough that routes mostly hit warm, small enough that you are not renting idle RAM for 10M functions.
Component Deep Dive
Four hard parts: (1) the cold start - making a from-nothing environment ready in sub-second time; (2) the warm pool - deciding which functions to keep warm, for how long, and reaping idle ones (scale-to-zero); (3) placement and fair multiplexing - which host runs an invocation, and sharing a packed fleet fairly across millions of functions and many tenants; and (4) autoscaling a single function from zero to thousands of concurrent instances fast. I will walk each from naive to the real design.
1. The Cold Start: Getting a From-Nothing Environment Ready, Fast
This is the signature problem of serverless. A sync request arrives for a function with zero warm instances; everything between “request arrives” and “handler runs” is the cold start, and it is user-visible.
Approach A: Boot a fresh VM/container per cold start (the naive instinct)
On a cold miss, pick a host, launch a container or VM, pull the image, start the language runtime, run the function’s init, then invoke.
on cold_miss(F):
host = pick_host()
vm = boot_vm() # hundreds of ms to seconds
pull_image(F) # tens of MB over network
start_runtime() # JVM/Node/Python interpreter boot
run_init(F) # user's top-level code, SDK clients, etc.
run_handler(event)
Where it breaks: every step is slow and serial. Booting a full VM is seconds; even a container with a cold image pull is hundreds of ms to seconds; a JVM cold start alone can be a second-plus; the user’s init code (opening DB pools, loading models) adds more. A cold start measured in seconds on the synchronous path is a broken product - users abandon, SLAs are blown. And if you provision strong isolation (a real VM) naively, cold start is worse. Naive per-cold-start boot forces an ugly isolation-versus-latency trade and loses on both.
Approach B: Lightweight microVMs with pre-initialized runtime
Replace the full VM/container with a microVM (Firecracker-style): a stripped minimal guest with a tiny kernel, no device emulation beyond essentials, booting in ~100-150ms, and pre-bake the runtime into a base image so the interpreter/JVM is already installed.
- Progress: boot drops from seconds to ~100ms, and you keep VM-grade isolation (each function gets its own guest kernel - essential for packing untrusted tenants, see part 3). Pre-baked runtime images mean you are not installing Node/Python on the hot path.
- Where it still breaks: ~100ms boot plus artifact pull plus the user’s init code (which you do not control and can be hundreds of ms - loading a 200MB model, establishing connection pools) can still blow past the sub-second target, and 100ms on every cold start across billions of invocations is a lot of latency and wasted CPU. You have cut it, not solved it. The remaining cost is boot + runtime-ready + user-init, repeated from scratch every time.
Approach C: Snapshot and restore - resume a pre-initialized environment
Do the expensive work once, snapshot the ready-to-serve state, and restore from it per cold start:
Build phase (once per function version, off the hot path):
- boot a microVM, load the runtime, run the function's init code to the
point just before the first handler call
- SNAPSHOT the full VM state (memory + CPU registers + devices) to a file
Cold start (hot path):
- RESTORE the microVM from the snapshot -> it comes back already booted,
runtime loaded, user-init already done, paused right before handler()
- resume execution and call handler(event)
- restore is tens of ms (lazy-page the memory in), not hundreds-to-seconds
- This collapses the cold start. Restoring a snapshot skips boot and runtime start and user init - the environment resumes already warmed to the pre-handler state. With lazy/on-demand memory loading (page in guest memory as touched rather than copying it all up front) and copy-on-write shared pages across restores of the same snapshot, a restore is tens of ms and shares memory across many instances of the same function. This is the technique that makes sub-second (often ~100-200ms) cold starts real even with strong VM isolation.
- Handle snapshot hazards. A restored-from-snapshot instance shares any state captured at snapshot time - including things that must be unique per instance, like random seeds, cached timestamps, or (dangerously) a connection with a fixed identifier. The runtime must re-seed randomness and reset such state on resume (hooks the runtime fires after restore) so two restored instances do not share a PRNG stream or a stale connection. Snapshots are also per function version and invalidated on deploy. Pre-initialized snapshots cannot capture truly dynamic init (a value fetched per-instance), so the function contract is “init should be deterministic and idempotent.”
- Shrink user init too. Offer the runtime hooks to defer heavy init, and surface cold-start cost so developers who care can trim it; offer provisioned concurrency (keep N instances pre-warmed and never scaled to zero) for latency-critical functions that opt in and pay for always-warm capacity - trading the scale-to-zero saving for guaranteed no cold start.
The cold-start answer: microVMs for VM-grade isolation at ~100ms boot, snapshot-and-restore to skip boot + runtime + user-init and resume a pre-warmed environment in tens of ms, lazy-paged copy-on-write memory to make restore cheap and shareable, runtime hooks to re-seed per-instance state safely, and provisioned concurrency as an opt-in escape hatch for functions that cannot tolerate any cold start. Strong isolation stops costing latency.
2. The Warm Pool: Which Functions to Keep Warm, and Scale-to-Zero
A warm instance is an environment that finished an invocation and is kept idle, ready for the next call - it avoids a cold start but it holds RAM on a host while doing no work. With 10M functions you cannot keep them all warm. The warm pool is the central economic and latency trade-off.
Approach A: Keep every instance warm forever (naive) - or kill it immediately (the other naive)
Two failing extremes. Keep-forever: after a function runs, never reap it; its warm instance lives indefinitely. Kill-immediately: tear down the environment the instant the handler returns, so every invocation is a cold start.
Where they break: keep-forever means 10M functions each pin at least one environment’s RAM forever - millions of machines of idle memory, and you are bankrupt (this also defeats scale-to-zero, the thing that makes the business model work). Kill-immediately means a function called 100 times a second pays 100 cold starts a second - catastrophic latency and wasted CPU re-creating the same environment. Neither balances the two costs that matter: idle warm RAM versus cold-start frequency.
Approach B: Idle TTL with keep-alive reuse and scale-to-zero
Keep an instance warm for a bounded idle window after each use, reuse it for calls that arrive within the window, and reap it (scale that function toward zero) when the window expires:
after handler returns:
mark instance idle, start/refresh an IDLE_TTL timer (e.g. ~5-15 min)
on new invocation of F:
if an idle warm instance exists -> reuse it (reset TTL) # warm hit
else -> cold start (and the new instance then goes warm)
on IDLE_TTL expiry with no reuse:
reap the instance, free its RAM; if it was the last instance of F,
F is now at ZERO instances -> scale-to-zero achieved, F costs nothing
- This is the core mechanism and it works because of locality. Real traffic is bursty and clustered: a function called once is likely to be called again soon. A modest idle TTL captures that burst - a function getting steady traffic stays warm (every call refreshes the TTL), and a function that truly went quiet is reaped and scaled to zero. You pay idle RAM only for the TTL window after the last call, not forever.
- Tune the TTL by signal, not a constant. A fixed TTL is crude. Bias it with the function’s historical invocation pattern: a function called every few minutes keeps a longer effective warm window (or a provisioned floor); a function called once a day gets reaped fast. Keep more instances of a hot function warm (match warm-pool depth to recent concurrency) and let cold functions fall to zero. The controller predicts rather than only reacts for the functions where it has signal.
- Decouple the idle-RAM cost from the host. Because a warm idle microVM is paused (restored lazily, copy-on-write memory shared across instances of the same function), an idle warm instance of a popular function costs far less unique RAM than N full copies - which is what lets you keep a deep warm pool for hot functions cheaply. For the long tail, snapshots let a reaped function be re-created in ~100ms, so reaping it is cheap to undo.
- Async triggers relax the TTL. For functions whose latency is not user-facing (queue/stream/cron), cold starts are hidden, so the controller reaps them more aggressively and keeps the warm RAM for the sync functions where cold starts hurt. The warm-pool budget is spent where it buys latency.
Idle-TTL reuse captures traffic bursts so steady functions stay warm while quiet ones scale to zero; history-biased TTLs and warm-pool depth matched to recent concurrency put the limited warm-RAM budget where cold starts are both frequent and user-visible; paused copy-on-write microVMs make a deep warm pool for hot functions cheap; and snapshots make reaping-and-restoring the long tail cheap enough that scale-to-zero is nearly free to reverse.
3. Placement and Fair Multiplexing on a Packed, Shared Fleet
An invocation must land on a host. You are packing hundreds of distinct tenants’ functions on each machine and multiplexing a shared fleet across millions of functions - placement decides both latency (is there a warm instance here?) and fairness (does one tenant’s burst starve others?).
Approach A: Random host, one tenant per host (naive) - or pure bin-pack (the other naive)
Random/one-tenant-per-host: assign each function a dedicated host (or random hosts), ignoring who else is there. Pure bin-pack: greedily stuff every host to its memory limit regardless of tenant mix or warm locality.
Where they break: dedicating hosts per tenant wastes enormous capacity (most tenants are idle; you would rent idle machines by the hundred-thousand) and kills scale-to-zero economics. Pure bin-packing ignores two things that matter: warm locality (routing a call to a host that has no warm instance of F forces a cold start even though a warm one exists elsewhere) and fairness/blast radius (packing one tenant’s many functions on few hosts means their burst, or their crash, or their escape, takes down everyone co-located). Neither balances packing density against warm-hit rate against fair isolation.
Approach B: Warm-aware placement + per-tenant limits + spread for isolation
Make placement a scored decision and enforce fairness and blast-radius limits:
- Route to warm first, create with locality. The placement service keeps a placement map: for each function, which hosts hold warm (idle) instances. A route prefers a host with a warm idle instance of
F(a warm hit, no cold start). On a cold miss it picks a host to create on by a score that balances free capacity, not over-packing one tenant, and (for snapshot sharing) co-locating instances of the same function so copy-on-write memory is shared. This keeps warm-hit rate high and density high. - Oversubscribe CPU, account memory honestly. Idle warm functions use ~0 CPU, so a host can hold hundreds of warm microVMs while only tens run at once - oversubscribe vCPU aggressively against concurrent executions, but account memory strictly (an idle warm microVM still holds its working set). Admission on a host is “can I fit the memory and is concurrent CPU under the oversubscription ceiling,” not a naive VM count.
- Per-tenant and per-function concurrency limits are the fairness mechanism. Every tenant has a concurrency quota (max simultaneous executions) and each function a share of it. The controller enforces it at placement: a tenant bursting to 10,000 concurrent is throttled at their limit (429 on sync, retried/delayed on async) rather than being allowed to consume the whole fleet and evict everyone else’s warm pools. Reserve a slice of fleet capacity as shared burst headroom that no single tenant can fully claim. This is what stops one customer’s spike from being everyone’s outage.
- Spread a tenant across hosts to bound blast radius. Deliberately do not pack one tenant’s instances onto few hosts. Spread them so (a) their burst draws from many hosts’ headroom rather than saturating a few, and (b) a single host failure, or a single sandbox escape, touches a small slice of any one tenant and only co-locates tenants who were going to share a machine anyway behind a strong (microVM) boundary. Isolation is VM-grade per instance (own guest kernel, seccomp, dropped caps, per-function IAM role, network egress only per role, no cross-tenant visibility) precisely because co-location is the norm.
- Keep placement metadata fast and available. The placement map and function config are read on every invocation, so they are served from a fast, replicated, cached store (in-memory with a durable backing); a stale read at worst causes an extra cold start (route to a host whose warm instance just got reaped), never a correctness bug, so you can cache aggressively and reconcile asynchronously.
Warm-aware scoring keeps the warm-hit rate high while still packing densely; CPU oversubscription against concurrent (not total) executions is what makes packing millions of idle functions economical; per-tenant concurrency quotas plus reserved burst headroom stop any one tenant from monopolizing the fleet; deliberate spread bounds the blast radius of a burst, a host failure, or an escape; and VM-grade per-instance isolation makes dense multi-tenant co-location safe.
4. Autoscaling One Function From Zero to Thousands, Fast
A function goes viral: from 1 warm instance to needing 5,000 concurrent in thirty seconds. Scaling must be fast, bounded, and not trample the rest of the fleet.
Approach A: Scale reactively off a slow metrics loop (naive)
Watch average CPU/utilization on a ~1-minute metrics interval and add instances when it is high.
Where it breaks: serverless bursts happen in seconds, not minutes. A CPU-averaging, minute-granularity control loop is hopelessly behind - by the time it reacts, the burst has already throttled or timed out thousands of requests. CPU is also the wrong signal: a function can be concurrency-bound (waiting on I/O) with low CPU while every instance is busy. Slow, wrong-signal scaling means a viral function just returns errors.
Approach B: Demand-driven, per-invocation scaling with bounded ramp
Scale on the signal that is the demand - concurrent in-flight invocations - at invocation granularity, with guardrails:
- Scale on concurrency, synchronously with demand. The placement path already knows, per function, how many instances are busy versus warm-idle. When an invocation arrives and no warm instance is free, that is the scale signal: create one (cold start) immediately, up to the function’s concurrency limit. Scaling is driven by actual arriving requests, instance-by-instance, not by a lagging metrics average - so it tracks a spike in real time.
- Bound the ramp to protect the fleet. Allow fast scale-up but rate-limit the burst (e.g. add instances in large steps per interval, then linearly) so one function cannot cold-start thousands of microVMs on shared hosts in one instant and starve everyone. Beyond the per-function/tenant concurrency limit, additional requests throttle (sync 429 with retry-after; async stay buffered in the queue and drain as capacity frees). The queue for async triggers is itself the shock absorber - a spike buffers instead of toppling the fleet.
- Reuse aggressively during the burst. As instances finish, they go warm-idle and the next queued request reuses them rather than cold-starting another - so the number of cold starts during a ramp is roughly the peak concurrency, not the total request count. Snapshot-restore makes each of those cold starts ~100ms, so ramping to thousands of instances takes seconds, not minutes.
- Scale down gently. After the spike, let idle TTL reap instances back down toward zero (part 2). Scale-down is lazy on purpose - reaping too eagerly right after a burst risks a second cold-start storm if the traffic returns; a modest idle window smooths it.
- Isolate scaling blast radius. Because a scaling function is spread across many hosts (part 3) and capped by its tenant quota with a reserved shared-burst pool that no single tenant can exhaust, one function’s zero-to-5,000 ramp draws from fleet-wide headroom fairly rather than crushing a few hosts or evicting other tenants’ warm pools.
Concurrency-driven, per-invocation scaling tracks a real-time spike that a metrics-averaging loop would miss; a bounded ramp plus per-tenant limits and a reserved burst pool let a function scale to thousands in seconds without starving the fleet; aggressive warm reuse keeps the cold-start count during a ramp near peak-concurrency rather than total-requests; and snapshot-restore makes each of those cold starts cheap enough that zero-to-thousands is a seconds-scale operation.
API Design & Data Schema
Management API (control plane - deploy and configure)
POST /v1/functions # create/deploy a function
{ "name": "resize-image",
"runtime": "python3.13" | "nodejs22" | "container",
"handler": "app.handler",
"code_ref": "s3://artifacts/sha256:abc...", # content-addressed artifact
"memory_mb": 512,
"timeout_s": 30,
"env": { "BUCKET": "thumbs" },
"role": "arn:iam:role/resize", # function's identity/permissions
"concurrency_limit": 1000, # max simultaneous instances
"provisioned": 0 } # pre-warmed floor (opt-in)
-> 201 { "function_id": "fn_7Qd2", "version": 1, "state": "active" }
PUT /v1/functions/{id}/config # update memory/timeout/env/limits
POST /v1/functions/{id}/versions # publish a new immutable version
GET /v1/functions/{id} # config, current version, metrics
DELETE /v1/functions/{id}
POST /v1/functions/{id}/triggers # wire an event source
{ "type": "http" | "queue" | "storage" | "schedule" | "stream",
"source": "...", "batch_size": 10 }
Invoke API (data plane - trigger execution)
POST /v1/functions/{id}/invoke # synchronous: caller waits
Headers: X-Invoke-Type: RequestResponse
Body: <event payload JSON>
-> 200 { "result": {...}, "duration_ms": 42,
"billed_ms": 42, "memory_mb": 512, "cold_start": false }
-> 429 { "error": "TooManyRequests", "retry_after_ms": 50 } # over limit
-> 500 { "error": "FunctionError", "type": "Unhandled",
"stack": "...", "request_id": "req_9x" }
POST /v1/functions/{id}/invoke # asynchronous: returns immediately
Headers: X-Invoke-Type: Event
-> 202 { "request_id": "req_9x", "state": "queued" }
(buffered durably; at-least-once; DLQ after max retries)
GET /v1/functions/{id}/invocations/{req_id} # async status/result lookup
Internal worker/agent contract (not public)
CREATE(function_id, version, snapshot_ref, memory_mb, role)
-> microVM slot: restore snapshot, ready to serve # a cold start
ROUTE(function_id) -> (host, slot) of a warm idle instance, or MISS
RUN(slot, event) -> { result|error, duration_ms, memory_peak_mb, cold }
REAP(slot) -> tear down an idle instance (idle-TTL expiry)
HEARTBEAT(host) -> warm-pool state, free capacity, health -> placement map
Data model
functions (relational - control-plane config):
function_id STRING PRIMARY KEY
tenant_id STRING INDEX
name STRING
current_version INT
runtime STRING
memory_mb INT
timeout_s INT
concurrency_limit INT
provisioned INT # pre-warmed floor
role STRING
state ENUM # active | disabled | deleting
created_at, updated_at TIMESTAMP
INDEX idx_tenant (tenant_id) # a tenant's functions
function_versions (relational, immutable rows):
function_id STRING PARTITION KEY
version INT CLUSTERING KEY
code_ref STRING # content-addressed artifact pointer (object store)
snapshot_ref STRING # pre-initialized VM snapshot for fast cold start
config_hash STRING
env JSON
published_at TIMESTAMP
PRIMARY KEY (function_id, version) # immutable; deploy bumps version
placement (in-memory / fast KV, durable-backed - read on every invoke):
function_id STRING KEY
warm_instances LIST< { host_id, slot_id, state: idle|busy, last_used } >
running_count INT # concurrency for scaling decisions
desired_count INT # controller's target
# high-churn, read-heavy, eventually-consistent OK (stale -> extra cold start,
# never a correctness bug). Served from memory, reconciled from agent heartbeats.
invocations_meter (wide-column / append-only - billing, 20B rows/day):
tenant_id STRING PARTITION KEY (+ time bucket)
request_id STRING CLUSTERING KEY
function_id STRING
start_ts TIMESTAMP
billed_ms INT
memory_mb INT
cold_start BOOL
status ENUM # ok | error | throttled | timeout
PRIMARY KEY ((tenant_id, day), request_id) # append-mostly; aggregated nightly
SQL vs NoSQL - the choice and why. Function config and versions want transactional updates, uniqueness, and queries by tenant, at modest size (millions of functions, slow-changing) - a sharded relational store (Postgres/MySQL, sharded by tenant_id/function_id) fits cleanly, so SQL for the control plane. The placement map is high-churn, read on every one of billions of invocations, and tolerates staleness (a stale entry at worst causes an extra cold start) - so an in-memory, replicated KV (a fast distributed cache with durable backing), not a transactional DB, because the invoke path cannot afford a synchronous DB hit and does not need strong consistency. Metering is pure append at enormous volume, read in aggregate for billing - a wide-column store (Cassandra/Bigtable) partitioned by (tenant, time) and rolled up nightly. Artifacts and snapshots are large immutable blobs - object storage, content-addressed and worker-cached. The deciding factor is the access pattern: transactional-and-queryable -> SQL; hot-read-tolerant-of-staleness -> in-memory KV; append-at-scale -> wide-column; large immutable blobs -> object store.
Bottlenecks & Scaling
Where it breaks first, in order, and the fix for each:
1. Cold starts dominate user-visible latency. A from-nothing environment is the slowest thing in the system and it is on the sync path. Fix: microVMs (VM isolation at ~100ms boot) + snapshot-and-restore that resumes a pre-initialized environment (skip boot, runtime, user-init) in tens of ms + lazy copy-on-write memory so restore is cheap and shared across instances; provisioned concurrency as an opt-in for latency-critical functions. This is the single most important optimization.
2. Warm-pool RAM is the scarce resource. Keeping environments warm for 10M functions is economically impossible; keeping too few makes cold starts frequent. Fix: idle-TTL reuse with scale-to-zero - reuse within a bounded idle window, reap after; history-biased TTLs and warm-pool depth matched to recent concurrency to spend the warm budget where cold starts are frequent and user-visible; paused copy-on-write microVMs so a deep warm pool for hot functions is cheap.
3. A single tenant’s burst starves the fleet. One function going viral consumes capacity and evicts everyone else’s warm pools. Fix: per-tenant and per-function concurrency quotas enforced at placement, reserved shared-burst headroom no single tenant can exhaust, throttle (429 / buffered async) beyond the limit, and spread instances across many hosts so a burst draws fleet-wide rather than saturating a few.
4. Autoscaling lags the spike. Minute-granularity, CPU-averaged scaling cannot track a seconds-long burst. Fix: concurrency-driven, per-invocation scaling (a free-warm-instance miss is the scale signal), a bounded ramp, aggressive warm reuse so cold starts during a ramp approximate peak concurrency not total requests, and snapshot-restore to make each cheap.
5. Placement metadata is read on every invocation. The routing lookup is the hottest read in the system. Fix: serve the placement map and function config from a replicated in-memory KV with durable backing; tolerate staleness (worst case: one extra cold start) to cache aggressively; reconcile from agent heartbeats asynchronously. Never put a synchronous transactional DB read on the invoke path.
6. Multi-tenant isolation on packed hosts. Hundreds of distinct tenants share one machine; a weak boundary is a cross-tenant breach. Fix: VM-grade per-instance isolation (own guest kernel via microVM, seccomp allowlist, dropped caps, per-function IAM role, network egress only per role, no cross-tenant process/filesystem visibility), spread tenants to bound blast radius, and treat a host as expendable - an escape hits one spread-out slice, not the fleet.
7. Artifact/snapshot pull on cold start. Fetching a large image or snapshot over the network stretches the cold start. Fix: content-addressed artifacts and snapshots cached on worker local disk; pre-distribute hot functions’ snapshots to hosts that will run them; lazy-page snapshot memory so you pull working-set pages on demand, not the whole image, before the first handler call.
8. Async event flood / thundering herd. A cron storm or a queue backlog wakes thousands of idle functions at once. Fix: the durable per-function queue buffers async events; pollers pull at a sustainable, rate-limited batch rate; cold starts for async are hidden from users so they can ramp gradually; DLQ after max retries so a poison event does not cycle forever.
9. Metering accuracy and volume. 20B invocations/day of billing records, and the measurement must be tamper-proof. Fix: meter duration from the platform/kernel, not the function’s own clock (the tenant’s code cannot influence its bill); write metering append-only to a wide-column store, aggregate nightly, keep raw for an audit window then roll to cold storage. Metering is shipped async off the invoke response path.
10. Control plane / router as a single point of failure. If placement or the frontend dies, nothing is invoked. Fix: stateless, multi-AZ, horizontally-scaled frontend and placement tier behind load balancers; the placement map replicated; the scaling controller sharded by function (no global lock) with leader election per shard; durable queues for async so a control-plane blip delays but does not drop events.
11. Function-code or runtime poison kills hosts. A function that reliably crashes its microVM (or a bad snapshot) burns cold starts forever. Fix: circuit-break a function after repeated create/run failures (back off, mark unhealthy, alert the tenant) instead of retrying into the fleet; version-pin snapshots and roll back a bad snapshot to on-demand init; attempt caps -> error/DLQ so one broken function stops cycling.
12. Hot-function memory skew. One wildly popular function’s deep warm pool crowds a host. Fix: copy-on-write snapshot sharing makes many instances of the same function cheap on a host; spread its instances across hosts; cap per-host instances-of-one-function so a single function cannot own a machine; autoscaling shifts its capacity across the fleet.
Wrap-Up
The trade-offs that define this design:
- The platform multiplexes a shared warm pool, it does not dedicate capacity. Dedicating an environment per function is impossible at 10M functions; the entire product quality is the warm-hit rate - how often an event finds a warm environment versus paying a cold start - over an oversubscribed, densely packed fleet. Every component serves that.
- Cold start is the latency, so make it rare and cheap. Snapshot-and-restore on microVMs resumes a pre-initialized environment in tens of ms with full VM isolation - the technique that makes sub-second cold starts real; idle-TTL warm reuse makes them rare; provisioned concurrency is the opt-in for functions that can tolerate none.
- Scale-to-zero and zero-cold-start are in direct tension, and that tension is a dial. Reap idle functions and they cost nothing but pay a cold start when they wake; keep them warm and they are instant but cost idle RAM. History-biased TTLs, warm-pool depth, and provisioned concurrency are how you place each function on that dial - aggressively for cost on the long tail, warm for latency on the hot, user-facing path.
- Fairness is enforced at placement, not hoped for. Per-tenant concurrency quotas, reserved shared-burst headroom, and deliberate spread across hosts stop one tenant’s zero-to-thousands burst from evicting everyone else’s warm pools or saturating a few machines - multi-tenancy is safe only because isolation is VM-grade and blast radius is bounded by design.
- The invoke path is a placement decision, never a DB transaction. Route-to-warm, create-on-miss, or throttle - all from a fast, staleness-tolerant in-memory placement map fed by heartbeats, with metering and logs shipped async, so billions of invocations add only a few ms of platform overhead on the warm path.
One-line summary: a stateless event frontend and a sharded placement/scaling controller that, per invocation, route to a warm microVM or create one within per-tenant limits, over a densely packed oversubscribed fleet where each function lives on an idle-TTL warm pool that scales to zero - with sub-second cold starts from snapshot-restore, rareness from history-biased warm reuse, fairness from per-tenant quotas and spread, and safety from VM-grade per-instance isolation.
Comments