Everyone thinks a container orchestrator is a scheduler with a for-loop. “Keep a list of nodes, keep a list of containers, assign each container to the node with the most free memory, SSH in and docker run it - it is bin-packing plus a deploy script.” That mental model survives exactly until the interviewer asks the questions that turn it into a real distributed system: a node goes silent for thirty seconds - is it dead, or is it alive and healthy but partitioned from the control plane behind a flaky switch? If you reschedule its containers onto other nodes and the node was actually alive, you now have two copies of a workload that was supposed to have one, both writing to the same database. You push a new version of a service that has forty running replicas - how do you replace all forty without a single second where zero of them are serving traffic, and how do you stop the rollout automatically the moment the new version starts crash-looping? The scheduler places a container on a node and sends the “run this” command, but the network drops the acknowledgment - did it start or not, and if you retry, do you now have two? Ten thousand nodes all report their status every ten seconds - does the control plane fall over under a million status writes a minute?

A container orchestration system is deceptive because the single-node docker run demo works perfectly and falls apart the moment you need desired-state convergence (you declare “I want 40 replicas of service X,” not “start a container on node 7,” and the system must make reality match that declaration and keep it matching forever), failure recovery under uncertainty (a node that stops responding is indistinguishable from a node that is partitioned, and acting on the wrong guess either loses availability or duplicates a workload), zero-downtime rollouts (replace every replica of a running service while it keeps serving, and roll back automatically on failure), and scale (tens of thousands of nodes, hundreds of thousands of containers, a control plane that cannot be a single process holding everything in memory). The whole design comes down to a few decisions that are not obvious: making a declarative desired state the source of truth stored in a consistent datastore, modeling the control plane as a set of independent reconciliation loops that each continuously watch desired-vs-actual and drive the gap to zero idempotently, separating scheduling (deciding placement, done once) from execution (making it true, done continuously by a per-node agent), and being honest that under a partition you cannot know if a node is dead - so you pick an availability-vs-safety trade-off explicitly rather than pretend the uncertainty away.

This sits in the same family as a distributed job scheduler (which runs tasks to completion) and a workflow orchestration engine (which runs DAGs), but it is a different problem: those run work that finishes, while an orchestrator runs long-lived services that must stay up forever and must be continuously reconciled against a declared desired state. Let me do it properly.

Functional Requirements (FR)

In scope:

  • Declarative workload management. A user declares desired state: “run 40 replicas of service checkout, image checkout:v7, each needing 2 CPU and 4 GB, with these env vars and this health check.” They do not say where or how. The system makes it true and keeps it true. The unit of scheduling is a pod - one or more tightly-coupled containers co-scheduled on one node sharing a network namespace.
  • Scheduling (placement). For each pod that needs a home, pick a node that has enough free resources and satisfies constraints (node selectors/affinity, anti-affinity so replicas spread across failure domains, taints/tolerations). Placement is a decision made once per pod; keeping it running is a separate, continuous concern.
  • Self-healing / failure recovery. If a pod crashes, restart it. If a node dies (stops heartbeating), reschedule its pods onto healthy nodes - but only after deciding, under uncertainty, that the node is actually gone. Actual replica count must converge back to the desired count without human action.
  • Rolling deployments with zero downtime. Update a service to a new image by replacing old replicas with new ones incrementally (bounded surge and unavailability), gating each step on new replicas becoming healthy, so the service keeps serving throughout. Automatic rollback when the new version fails its health checks.
  • Service discovery and load balancing. Pods are ephemeral and get new IPs; a stable virtual endpoint (service) load-balances across the current healthy replicas, and the set of backends updates as pods come and go.
  • Horizontal autoscaling. Scale a service’s replica count up or down based on a metric (CPU, request rate, queue depth) by adjusting the desired count, which the normal reconciliation then converges to.
  • Resource isolation and limits. Each container gets requested CPU/memory guaranteed and is capped at its limit (cgroups), so one workload cannot starve its neighbors on a shared node.
  • Observability and control. Query the state of any workload, node, and pod; stream logs; drain a node for maintenance; roll back a deployment.

Explicitly out of scope (say it so you own the scope):

  • The container runtime itself. Pulling images, setting up namespaces/cgroups, and actually starting a container is the runtime’s job (containerd/CRI-O). We tell a node agent “run this pod spec”; the agent calls the runtime. We own orchestration, not the Linux primitives.
  • The overlay network / CNI implementation. We assume pods get routable IPs via a network plugin and that the service load-balancer can program data-plane rules. We define the service abstraction and its backend set; building the packet-forwarding fabric (VXLAN, eBPF, iptables/IPVS) is a networking subsystem.
  • Persistent storage provisioning. We schedule pods and can honor volume constraints, but provisioning and attaching network block storage is a storage controller’s job; we reference it, not rebuild it.
  • Multi-cluster federation and the full image registry. One cluster’s control plane. Cross-cluster scheduling, registry internals, and image build pipelines are separate systems.
  • Exactly-once execution of a pod. We keep at least the desired number of healthy replicas and converge toward exactly that number, but during a partition we may transiently run more than desired (the safety-vs-availability trade-off). We make the trade-off explicit, not vanish.

The one decision that drives everything: desired state is a declarative document in a strongly-consistent datastore, and the control plane is a set of stateless reconciliation loops that each watch the gap between desired state and observed actual state and drive it to zero with idempotent actions - they hold nothing authoritative in memory, so a crash is just a re-read, and every action is safe to repeat. That is what makes self-healing automatic (the gap reappears, the loop closes it again), makes the control plane crash-safe (restart, re-watch, reconcile), and turns “deploy,” “scale,” “recover from node death,” and “roll back” into the same operation: change desired state (or observe changed actual state) and let the loop converge. Everything below is a consequence of it.

Non-Functional Requirements (NFR)

  • Scale: on the order of tens of thousands of nodes per cluster (call it up to ~15,000), hundreds of thousands of running pods (~150,000+), thousands of distinct services/workloads, and a steady churn of pod creations/deletions during rollouts and autoscaling in the thousands per second at peak. Node status updates dominate write load: 15,000 nodes heartbeating/reporting every few seconds is the single largest traffic source into the control plane.
  • Latency (control-plane reaction, not request serving): when desired state changes or a failure is observed, the system should begin converging quickly - a pod that becomes schedulable should be placed within a p99 of a couple of seconds, and a failure signal should begin reconciliation within seconds. We optimize the gap between “something changed” and “a loop acted on it,” not the runtime of the container.
  • Availability: the control plane targets 99.95%+, but the crucial property is that the data plane survives control-plane downtime. If the whole control plane is offline, already-running pods keep running and keep serving traffic; you simply cannot schedule new ones or react to failures until it recovers. No workload goes down because the control plane blinked.
  • Consistency: desired state and the authoritative record of what is scheduled where is strongly consistent (linearizable) - two schedulers must not both place a pod, a rollout must advance deterministically, a node’s pod assignments must not race. Actual-state reporting (node/pod status) is allowed to be eventually consistent and lossy (it is just the latest observation; stale is tolerable, the loop re-reads).
  • Durability: desired state is the system of record and must never be lost - it is replicated and persisted before it is acted on. Losing it means forgetting what the cluster is supposed to be running.
  • Partition behavior (stated explicitly): under a network partition the control plane is a CP system for its own state (the datastore needs quorum; a minority partition cannot accept writes), and it makes a deliberate, tunable safety-vs-availability choice for workloads on unreachable nodes: wait a grace period before declaring a node dead and rescheduling, trading recovery speed for not duplicating workloads on a node that was merely partitioned.

Back-of-the-Envelope Estimation (BoE)

Let me ground the design in numbers.

Cluster and pod volume:

Nodes:                        15,000
Pods per node (avg):          ~30 (CPU/mem bound; mix of big and small)
Running pods:                 15,000 * 30 = ~450,000 pods
Distinct services/workloads:  ~5,000, avg ~90 replicas each
Pod churn (rollouts+autoscale+healing): peak ~3,000 pod create/delete per sec
   (a fleet-wide rollout of many services at once dominates this)

Control-plane write load (the real pressure is node status, not user actions):

Node status reports: 15,000 nodes * 1 report / 5 sec = 3,000 writes/sec, steady,
   24/7, independent of what users do. This is the floor.
Pod status reports:  450,000 pods, each reports state changes; at steady state
   low, but during a rollout thousands of pods flip Pending->Running per sec.
User/controller writes (desired-state edits, scheduling decisions):
   thousands/sec during rollouts.
Combined peak write rate into the datastore: ~10,000-20,000 writes/sec.
Watch/read load: every controller and every node agent WATCHES its slice of
   state. 15,000 node agents each watching "pods assigned to me" + thousands of
   controllers watching their objects = the datastore must fan out changes to
   tens of thousands of long-lived watchers efficiently, NOT be polled.

The first non-obvious number: the dominant load is not scheduling, it is status reporting and watch fan-out. 15,000 nodes writing status every few seconds is a constant 3,000 writes/sec floor that has nothing to do with user activity, and tens of thousands of watchers need to be notified of changes without re-reading the whole store. A naive “every agent polls the full state every second” design dies here long before scheduling does.

Storage:

Pod object (spec + status): ~4 KB packed.
450,000 pods * 4 KB         = ~1.8 GB of pod objects.
Node objects: 15,000 * ~2 KB = ~30 MB.
Services, deployments, configmaps, secrets, etc.: a few hundred MB.
Total desired+actual state:  on the order of a few GB - it FITS IN MEMORY.
This is the key shape: the authoritative state is small (single-digit GB),
   so the consistent datastore holds the whole cluster state and serves
   watches from an in-memory cache. The hard part is not storage size, it is
   the write/watch RATE and the correctness of state transitions.

Reschedule storm after a rack failure:

A rack/zone fails: say 500 nodes * 30 pods = 15,000 pods need rescheduling
   at once. The scheduler must place 15,000 pods against the remaining nodes'
   free capacity quickly, without thundering-herd double-placement, and the
   grace-period logic must avoid firing on a transient 2-second blip.

The headline shape: small authoritative state (a few GB, fits in memory) but a high and constant write/watch rate dominated by node status, with the correctness-critical work being state-machine transitions (place a pod, declare a node dead, advance a rollout) that must be linearizable and idempotent. Size the datastore for write throughput and watch fan-out, not for bytes; make the whole thing a set of reconcile loops so recovery and healing are free; and treat “is that node dead or partitioned” as the central hard decision, not an edge case.

High-Level Design (HLD)

The system splits into a control plane (the brain, a handful of replicated nodes) and a data plane (the fleet of worker nodes that actually run containers). The control plane is: an API server (the single front door and the only thing that talks to the datastore - every read, write, and watch goes through it); a consistent datastore (a Raft-backed key-value store like etcd holding all desired and actual state); a scheduler (watches for unscheduled pods, assigns each to a node); and a set of controllers (reconciliation loops - one per object kind: a Deployment controller, a ReplicaSet controller, a node-lifecycle controller, etc.). Each worker node runs a node agent (the kubelet analog) that watches the pods assigned to it and drives the local container runtime to make them real, plus a service proxy that programs load-balancing to the current backends.

  USERS / CI                      ┌──────────────── CONTROL PLANE (replicated) ─────────────────┐
 ┌──────────────┐                 │                                                              │
 │ kubectl apply│                 │   ┌──────────────┐     watch/write     ┌──────────────────┐ │
 │ (desired     │   1. POST /apply│   │  API SERVER  │◄───────────────────►│  DATASTORE (Raft)│ │
 │  state yaml) │────────────────►│   │ (front door, │   authz, validate   │  etcd: desired + │ │
 └──────────────┘                 │   │  only writer │                     │  actual state,   │ │
                                   │   │  to store)   │                     │  serves WATCH    │ │
                                   │   └──────┬───────┘                     └──────────────────┘ │
                                   │          │ 2. controllers + scheduler WATCH their objects   │
                                   │   ┌──────┴─────────────────────────────────────────────┐   │
                                   │   │  SCHEDULER: watch Pending pods -> pick node -> write │   │
                                   │   │              pod.nodeName (a scheduling DECISION)    │   │
                                   │   │  DEPLOYMENT CTRL: desired replicas/version -> manage │   │
                                   │   │              ReplicaSets, drive rolling update       │   │
                                   │   │  REPLICASET CTRL: ensure N pods exist for a RS       │   │
                                   │   │  NODE-LIFECYCLE CTRL: watch node heartbeats ->       │   │
                                   │   │              on grace-period expiry, evict pods      │   │
                                   │   └──────────────────────────────────────────────────────┘ │
                                   └───────────────────────────▲──────────────────────────────────┘
                                             3. node agents WATCH "pods where nodeName==me"
                                                & POST node/pod status (heartbeat) ──┐
            ┌───────────────────────────────────────────────────────────────────────┘
            ▼  DATA PLANE (worker fleet)
 ┌────────────────────────┐   ┌────────────────────────┐   ┌────────────────────────┐
 │  NODE n1               │   │  NODE n2               │   │  NODE n3 ...            │
 │  node agent (kubelet)  │   │  node agent            │   │  node agent             │
 │   - watch my pods      │   │   - run containers via │   │   - heartbeat status    │
 │   - run via runtime    │   │     container runtime  │   │   - restart crashed pods│
 │   - report status      │   │   - report status      │   │                         │
 │  service proxy ────────┼───┼──► program LB to backend pods (service -> pod IPs)   │
 │  container runtime     │   │  container runtime     │   │  container runtime      │
 └────────────────────────┘   └────────────────────────┘   └────────────────────────┘

Request flow - declaring desired state (the write path):

  1. A user kubectl applys a Deployment: “service checkout, 40 replicas, image v7, 2 CPU / 4 GB each, rolling-update strategy, health check.” The API server authenticates, validates the spec, assigns it a version, and writes the desired-state object durably to the datastore. The user’s job is done - they declared intent; nothing is scheduled yet. The write is the single source of truth from this instant.
  2. Controllers that watch Deployments see the new object. The Deployment controller creates a ReplicaSet (desired: 40 pods of v7); the ReplicaSet controller sees “I should have 40 pods but 0 exist” and creates 40 pod objects in the datastore, each in state Pending with no node assigned. Still nothing is running - these are desired-state records.

Request flow - scheduling and running (the convergence path):

  1. The scheduler watches for pods with no nodeName. For each, it runs placement: filter nodes that fit (enough free CPU/mem, satisfy affinity/taints), score the survivors (spread, bin-pack, least-loaded), pick the best, and write pod.nodeName = chosen back through the API server. This is a decision, recorded in state - the scheduler does not start anything.
  2. The node agent on the chosen node is watching “pods where nodeName == me.” It sees a new assigned pod, pulls the image, and tells the local container runtime to start the containers (set up cgroups, namespaces, network, volumes). It then continuously reports the pod’s status (Pending -> ContainerCreating -> Running or CrashLoopBackOff) back to the API server, and heartbeats the node’s own health every few seconds.
  3. The service proxy on every node watches the service’s backend set (the healthy pod IPs) and programs local load-balancing rules, so traffic to the service’s virtual IP spreads across the current healthy replicas. As pods come and go, the backend set updates and the proxies re-program.

Request flow - healing and failure (the reconcile-under-failure path):

  1. If a pod crashes, the node agent restarts it (with backoff) and reports the state; the desired count is still met locally. If a pod is deleted or a node is lost, the ReplicaSet controller observes “actual < desired” and creates replacement pod objects, which the scheduler places elsewhere - the gap reappears, the loop closes it. Self-healing is just reconciliation of the same desired state.
  2. The node-lifecycle controller watches node heartbeats. If a node stops heartbeating, it does not immediately reschedule - it waits a grace period (the partition-vs-death decision), then marks the node NotReady, evicts (tombstones) its pods, which makes the owning ReplicaSets see “actual < desired” and create replacements that the scheduler places on healthy nodes. This is the single most important and most dangerous flow, and part 2 of the deep dive is entirely about getting it right.

The key architectural insight: there is exactly one source of truth (desired + recorded actual state in the datastore), one writer path (the API server), and many independent stateless loops that each watch a slice of that state and drive their small piece of the gap to zero. No loop holds authoritative state; each is idempotent and crash-safe; “deploy,” “scale,” “heal,” and “recover” are all the same move - change the state, let the loops converge. That is what makes the control plane recover from its own crashes for free and lets each concern (scheduling, replicas, rollouts, node lifecycle) evolve as a separate, testable loop.

Component Deep Dive

Four hard parts: (1) the reconciliation-loop control plane itself - the engine that makes everything else a special case of “converge desired to actual”; (2) scheduling pods across the cluster efficiently and correctly without double-placing; (3) recovering from node failure under partial network failure - the dead-vs-partitioned decision, the hardest correctness problem here; and (4) rolling deployments with zero downtime and automatic rollback. I will walk each from naive to scalable.

1. The Reconciliation-Loop Control Plane

The temptation is to write an imperative orchestrator: an API call to “deploy” directly SSHes to nodes and starts containers; “scale up” directly starts more. This is the model that cannot survive failure.

Approach A: Imperative orchestration (the naive instinct)

A central orchestrator exposes commands. deploy(service) picks nodes and issues start container RPCs to each. scale(service, +5) issues 5 more starts. It keeps an in-memory map of what is running where.

on deploy(svc):
   for i in range(svc.replicas):
       node = pick_node()
       rpc(node, "start", svc.image)       # fire and forget
       running[svc].append(node)           # in memory
on node_crash(node):                        # if we even detect it
       for svc on node: rpc(pick_node(), "start", svc.image)

Where it breaks: three fatal flaws. First, it loses state on restart - the running map is in memory, so an orchestrator crash forgets what the cluster is supposed to be running; on restart it has no idea whether checkout has 40 replicas or 3, and cannot reconcile. Second, every command is a one-shot with no convergence - if the start RPC’s acknowledgment is dropped by the network, the orchestrator does not know if the container started; retrying risks a duplicate, not retrying risks a missing replica, and there is no loop that notices and fixes the discrepancy later. Third, failure handling is a pile of special cases - “node crashed,” “pod crashed,” “deploy half-failed,” “network blipped” each need bespoke imperative recovery code, and they interact combinatorially. Imperative orchestration treats the happy path as the design and every failure as an exception to hand-handle; in a system where failure is constant, that is backwards.

Approach B: Declarative desired state + independent reconciliation loops

Invert the whole thing. The user never issues commands; they declare desired state (“40 replicas of v7”) as a document in the datastore. The control plane is a set of controllers, each an infinite loop of the same shape:

loop forever (or on watch event):
   desired = read desired state for my objects      # from the datastore
   actual  = read observed actual state             # from the datastore
   diff    = desired - actual
   for each item in diff:
       take ONE idempotent action to shrink the gap # create/delete/update an object
   # then sleep until the next watch event or resync tick
  • Convergence replaces commands. “Deploy” is a write to desired state; the loop notices actual (0 pods) != desired (40 pods) and creates pods. “Scale up” is editing a number; the loop creates the difference. “A pod died” shows up as actual (39) != desired (40); the same loop creates one replacement. There is no separate “healing” code - healing is what the loop does by definition. This collapses deploy, scale, and self-heal into one mechanism.
  • Idempotent, level-triggered actions make retries and crashes safe. Each action is “make this one object exist / not exist / match,” conditioned on current state (a CAS / resource-version check through the API server). If the loop runs twice, the second pass sees the gap already closed and does nothing. The loop is level-triggered, not edge-triggered: it reacts to the current difference, not to an event stream it must not miss. A missed watch event is harmless because the next resync re-reads the full state and reconciles the level. This is why a controller can crash mid-reconcile and simply restart - it re-reads desired and actual and continues; it holds nothing it cannot rebuild.
  • Separate loops per concern, one source of truth. Instead of one giant orchestrator, there is a Deployment controller, a ReplicaSet controller, a node-lifecycle controller, an endpoints controller, each owning one object kind and one small transformation (Deployment -> ReplicaSets -> Pods -> scheduled Pods -> running containers). They compose: the Deployment controller’s output (a ReplicaSet with desired=40) is the ReplicaSet controller’s input. Each is independently testable, independently crash-safe, and does one thing. The datastore is the only shared state; the API server is the only writer; coordination between loops is “write an object the next loop watches,” never direct RPC.
  • Watch, do not poll. With tens of thousands of controllers and node agents, polling the full state is impossible. Each watcher registers a watch on its slice (a ReplicaSet controller watches its pods; a node agent watches pods assigned to it) and the API server streams incremental changes from an in-memory cache of the datastore, each change tagged with a monotonic resource version so a watcher that disconnects can resume from where it left off (or re-list and re-watch). This turns “notify 30,000 watchers of a change” from a fan-out storm into cheap incremental pushes, and a periodic full resync backstops any missed event (level-triggered, remember - correctness does not depend on never missing an event).

The move that makes the whole system work: desired state is a declarative document, the control plane is a set of level-triggered reconciliation loops that each drive one slice of actual toward desired with idempotent actions, and they coordinate only through a single consistent store behind a single API-server writer. Every other feature in this design is an instance of this pattern.

2. Scheduling Pods Across the Cluster

A pod needs a node with enough resources that satisfies its constraints. Doing this for hundreds of thousands of pods against tens of thousands of nodes, correctly and without double-placing, is its own problem.

Approach A: Greedy “most free node,” one global lock (naive)

A single scheduler, on each pending pod, scans all nodes, picks the one with the most free memory, and assigns it. To avoid races, it holds a global lock while scheduling.

Where it breaks: two problems. First, correctness under concurrency - if you run two schedulers for availability, both can read “node 7 has 8 GB free,” both place a 6 GB pod there, and now node 7 is oversubscribed; the single global lock that prevents this also serializes all scheduling into one thread and caps throughput. Second, greedy “most free” packs badly and ignores constraints - it ignores anti-affinity (all 40 replicas can land on one node, so one node’s death takes the whole service down), ignores taints/affinity, and “most free memory” causes resource stranding (a node with lots of free memory but no free CPU still gets picked for a memory-light pod, wasting it). And scanning all 15,000 nodes for every one of thousands of pods per second is O(pods x nodes) - too slow at a reschedule storm.

Approach B: Filter-score two-phase scheduling, optimistic concurrency, batched and capped scans

Model scheduling as a two-phase pipeline per pod, assign by writing the decision through the API server with optimistic concurrency, and bound the cost of scanning:

schedule(pod):
   feasible = [n for n in nodes if fits(n, pod.requests)        # Filter (predicates):
                  and tolerates(n.taints, pod.tolerations)       #   hard constraints -
                  and satisfies(pod.affinity, n, placed_pods)]   #   a node is in or out
   if feasible empty: pod stays Pending, retry later (emit "unschedulable" event)
   scored = [(n, score(n, pod)) for n in feasible]              # Score (priorities):
               # score = w1*spread(anti-affinity) + w2*least-loaded
               #       + w3*bin-pack + w4*image-locality ...      #   rank the survivors
   best = argmax(scored)
   write pod.nodeName = best  IF pod.resourceVersion unchanged  # optimistic CAS
      on conflict: someone else touched it; re-read and retry
  • Filter then score separates “can it run here” from “should it.” Filtering applies hard predicates (fits resources, tolerates taints, satisfies required affinity) and reduces 15,000 nodes to a feasible set; scoring ranks only the survivors by soft preferences (spread replicas across failure domains for availability, prefer least-loaded or bin-pack by policy, prefer nodes that already have the image). This is both faster (score a handful, not all nodes) and correct (anti-affinity is a filter, so replicas spread).
  • Optimistic concurrency, not a global lock. The scheduler does not lock the cluster. It computes a placement and writes pod.nodeName conditioned on the pod’s resource version (a compare-and-set through the API server). If two schedulers (or a scheduler and a retry) race, only one write wins; the loser sees a conflict, re-reads, and reschedules. More importantly, the node agent is the final arbiter of capacity: when the agent receives the assigned pod, it admits it only if the node genuinely has room right now; if not, it rejects it and the pod goes back to Pending for rescheduling. So the scheduler’s view can be slightly stale without corrupting the node - placement is a proposal, admission is the commit. This is what lets scheduling run concurrently and tolerate a stale cache.
  • Bound the scan: cache, percentage-of-nodes, and batching. The scheduler keeps an in-memory cache of node state fed by watches (never re-list all nodes per pod). To avoid O(nodes) per pod, it scores only a sample / a percentage of feasible nodes (stop once you have enough good candidates) rather than all of them - at 15,000 nodes, scoring the best ~100 feasible candidates is plenty and keeps per-pod latency flat as the cluster grows. During a reschedule storm, process pending pods in batches and spread placements (do not stack a storm of replacements onto the few emptiest nodes, which just creates new hotspots).
  • Preemption and priority. When a high-priority pod cannot fit anywhere, the scheduler may preempt (evict) lower-priority pods on a node to make room, recording the decision in state; the evicted pods go back to Pending and reschedule elsewhere. This needs priority classes on pods and a bounded, graceful eviction - otherwise a priority inversion starves critical workloads.

What makes scheduling scale and stay correct: a filter-then-score pipeline that treats anti-affinity as a hard predicate, optimistic-concurrency writes of the placement decision (with the node agent as the real admission gate) instead of a global lock, and bounded candidate scanning from a watch-fed node cache. Placement is a proposal; the node commits it; concurrency is safe because the final capacity check is local.

3. Node Failure Recovery Under Partial Network Failure

A node stops heartbeating. This is the hardest decision in the system, because a dead node and a partitioned-but-healthy node produce the identical signal - silence - and the two correct responses are opposite.

Approach A: Reschedule immediately on missed heartbeat (naive)

If a node misses, say, two heartbeats (10 seconds), declare it dead and reschedule all its pods onto healthy nodes at once.

Where it breaks: the node was not dead - it was partitioned from the control plane by a flaky top-of-rack switch, but perfectly alive and still running its pods and still serving traffic (the data plane does not need the control plane to keep running). The control plane, seeing silence, declares it dead and starts 30 replacement pods elsewhere. Now every workload on that node runs in two places at once: the original copies on the partitioned-but-alive node, and the replacements. For a stateless web service that is wasteful; for a singleton workload with a side effect - a leader, a job writing to a database, a payment processor - it is a correctness disaster (split-brain, double-writes, two “leaders”). Reacting fast to a signal you cannot interpret turns a transient network blip into a duplicated-workload incident. Worse, a flapping switch causes reschedule thrash: declare dead, reschedule, node comes back, now double, delete, flaps again.

Approach B: Grace period + eviction via desired-state, fencing for singletons, rate-limited

You cannot distinguish dead from partitioned, so you choose a deliberate policy and make the dangerous cases safe:

node-lifecycle controller:
   track last_heartbeat per node (updated by node agent, every ~5s)
   if now - last_heartbeat > node_ready_grace (e.g. 40s):
        mark node NotReady (stop scheduling NEW pods to it)
   if now - last_heartbeat > eviction_grace (e.g. 5 min, tunable):
        for each pod on node: set deletionTimestamp (tombstone it in desired state)
        # this makes the owning ReplicaSet see actual<desired -> create replacements
        # which the scheduler places on HEALTHY nodes
   when node heartbeats again within grace:
        clear NotReady; its pods (if the node still has them) are honored again
  • A grace period trades recovery speed for safety, explicitly. You wait (tunable, often minutes) before declaring a node dead and evicting its pods. During the wait, the workload is possibly down (if the node really died) - that is the cost - but you avoid duplicating a workload for a node that was merely partitioned for 30 seconds. The grace period is the single most important tuning knob, and it is a business decision, not a default: a stateless fleet can evict fast; a cluster running stateful singletons must wait longer or require stronger evidence. State the trade-off; do not pretend a right answer exists.
  • Recovery goes through desired state, not a side channel. The controller does not SSH anywhere. It tombstones the dead node’s pods (marks them for deletion in the datastore); the normal ReplicaSet reconciliation then observes actual < desired and creates replacements that the scheduler places on healthy nodes. Recovery is the same reconcile loop as everything else - which is why it is correct and crash-safe. When the control plane itself was down during the failure, it catches up on restart by comparing state, no lost signal.
  • Fencing makes the dangerous case safe. For workloads where running twice is a correctness bug, reschedule-after-grace is not enough - the partitioned node might still be running the old copy. You need fencing: before a replacement for a singleton takes over, ensure the old copy cannot act. Options the orchestrator provides: leases with fencing tokens (the workload acquires a lease; the new copy gets a higher token and the shared resource rejects the old token - this is how a stateful leader avoids split-brain), or node fencing / STONITH (power-off or network-isolate the suspect node via an out-of-band controller before rescheduling its singletons). The orchestrator cannot guarantee exactly-one for a non-cooperating workload; it provides leases/fencing and an at-most-one guarantee for workloads that use them, and at-least-one (possibly duplicated) for the rest. Be honest about which guarantee a given workload gets.
  • Rate-limit evictions to avoid cascading failure. When a whole rack fails, do not evict and reschedule 15,000 pods in one burst - that can overload the remaining nodes and the scheduler and cause a cascade. Cap the eviction rate (and use pod-disruption budgets so a service never drops below a minimum healthy count during voluntary disruptions), so recovery is paced. Also dampen flapping: a node that recovers within grace is never evicted; use the grace window itself as the debounce.
  • The node agent self-limits under partition. The other side of the trade-off: a well-behaved node agent that loses contact with the control plane for too long can stop admitting new work and optionally shut down its pods if configured for safety-critical workloads, so even a partitioned node does not diverge arbitrarily. This bounds how long a duplicate can exist without cooperation from the workload.

The crux: you cannot tell dead from partitioned, so you pick a grace period as an explicit availability-vs-safety trade-off, recover by tombstoning pods through normal desired-state reconciliation, and make the run-twice-is-wrong case safe with leases/fencing rather than hoping the old copy is gone. Fast reaction to an ambiguous signal is the bug; a deliberate, fenced, rate-limited response is the fix.

4. Rolling Deployments with Zero Downtime and Automatic Rollback

Replace all 40 replicas of a running service with a new version, without a single moment of zero healthy replicas, and roll back automatically if the new version is broken.

Approach A: Delete all old, start all new (naive recreate)

Stop all 40 old pods, then start 40 new ones.

Where it breaks: there is a window - however short - where zero replicas are serving, which is a full outage for the service. And if the new image is broken (crash-loops, fails its health check), you have just taken the service to zero and the replacement never comes up - a self-inflicted total outage with no automatic way back, because the old version is already gone. Recreate trades the entire availability guarantee for simplicity.

Approach B: Bounded-surge rolling update, health-gated, with automatic rollback

Model the deployment as two ReplicaSets (old and new) whose replica counts the Deployment controller shifts gradually, gated on health, bounded by surge and unavailability limits:

Deployment: desired 40 replicas, image v7, strategy { maxSurge: 25%, maxUnavailable: 0 }
old_rs (v6): desired starts at 40
new_rs (v7): desired starts at 0

reconcile step (the Deployment controller loop):
   # maxSurge=10 -> may run up to 50 total; maxUnavailable=0 -> never < 40 healthy
   scale new_rs up toward target, but only so total pods <= 40 + maxSurge
   WAIT until the newly added v7 pods pass their readiness probe (become Ready)
   only THEN scale old_rs down, keeping healthy(old)+healthy(new) >= 40 - maxUnavailable
   repeat until new_rs=40 and old_rs=0
  • Surge and unavailability bounds keep capacity above the floor the whole time. maxUnavailable is the number of replicas you may drop below desired during the rollout (0 means never fewer than 40 healthy); maxSurge is how many extra you may run temporarily (headroom to add new before removing old). With maxUnavailable: 0, maxSurge: 25%, the controller adds up to 10 new pods, waits for them to pass readiness, then removes 10 old - total healthy never dips below 40. This is how “replace everything” becomes “replace a few at a time while always keeping enough serving.”
  • Readiness probes gate every step and keep traffic off not-yet-ready pods. A new pod is not counted as available until it passes its readiness probe (it is actually serving), and - critically - the service load balancer only routes to Ready pods, so a new pod that is starting up or warming caches receives no traffic until it is ready, and the rollout does not advance until the step’s new pods are ready. This is what makes it zero-downtime, not just zero-replica-count-dip: never is traffic sent to a pod that cannot serve it. Separately, a liveness probe restarts a hung pod, and a startup probe gives slow-booting apps time before liveness kicks in.
  • Automatic rollback on failure. The controller watches the new pods’ health as it advances. If new pods fail readiness or crash-loop past a threshold within a progress deadline (e.g. no forward progress for N minutes), it stops the rollout and, if configured, rolls back: because both ReplicaSets still exist and desired state is versioned, rollback is just “scale new_rs back to 0, scale old_rs back to 40” - the same rolling mechanism in reverse, converging to the previous known-good version. The old version was never fully removed until the new one proved healthy, so there is always a safe target. This is the direct payoff of modeling a deployment as two coexisting ReplicaSets rather than an in-place mutation.
  • Graceful shutdown so in-flight requests are not dropped. Removing an old pod is not a kill. The pod is first removed from the service’s backend set (so new requests stop arriving), then sent a termination signal with a grace period during which it finishes in-flight requests and drains connections, and only force-killed if it overruns. Without this, “zero downtime” still drops the requests that were mid-flight on a terminating pod. Ordering matters: deregister from the load balancer before signaling shutdown.
  • It is all still reconciliation. The Deployment controller is just another loop: desired = “40 of v7 with these bounds,” actual = “current old/new ReplicaSet sizes and pod readiness,” and each tick it takes one bounded step toward the goal. A controller crash mid-rollout is a non-event - on restart it re-reads the two ReplicaSets’ current sizes and pod health and continues the rollout from the actual current level. Pause, resume, and canary (hold new_rs at a small count, watch metrics, then proceed) are variations on the same loop.

What makes zero-downtime rollout work: two coexisting ReplicaSets whose sizes a reconcile loop shifts in bounded steps, gated on readiness probes with the load balancer routing only to Ready pods, backstopped by graceful connection-draining on shutdown and an automatic rollback (scale back to the still-present old ReplicaSet) when the new version fails its progress deadline. You always keep enough healthy replicas, you never route to a pod that cannot serve, and the old version survives until the new one proves itself.

API Design & Data Schema

Declarative / control API (control plane, through the API server)

The API is declarative - you PUT the desired object, you do not call imperative verbs. Everything is a resource with a spec (desired) and a status (observed).

POST   /apis/apps/v1/namespaces/{ns}/deployments        # declare a workload
  { "metadata": { "name": "checkout" },
    "spec": {
      "replicas": 40,
      "selector": { "app": "checkout" },
      "strategy": { "type": "RollingUpdate",
                    "rollingUpdate": { "maxSurge": "25%", "maxUnavailable": 0 },
                    "progressDeadlineSeconds": 600 },
      "template": {                                      # the pod spec to replicate
        "metadata": { "labels": { "app": "checkout" } },
        "spec": {
          "containers": [ { "name": "checkout", "image": "checkout:v7",
              "resources": { "requests": {"cpu":"2","memory":"4Gi"},
                             "limits":   {"cpu":"2","memory":"4Gi"} },
              "readinessProbe": { "httpGet": {"path":"/ready","port":8080},
                                  "periodSeconds": 5 },
              "livenessProbe":  { "httpGet": {"path":"/healthz","port":8080} } } ],
          "affinity": { "podAntiAffinity": "spread across zones" },
          "tolerations": [ ... ] } } } }
  -> 201  (writes desired state; controllers converge asynchronously)

PATCH  /.../deployments/checkout   { "spec": { "replicas": 60 } }   # scale = edit desired
PATCH  /.../deployments/checkout   { "spec": { "template": { image: "checkout:v8" } } }  # rollout
POST   /.../deployments/checkout/rollback   { "toRevision": 11 }    # explicit rollback
GET    /.../deployments/checkout    -> spec + status { readyReplicas, updatedReplicas, conditions }

GET    /.../pods?fieldSelector=spec.nodeName=n1      # node agent watches ITS pods
GET    /.../nodes/n1    -> node status { capacity, allocatable, conditions:[Ready], lastHeartbeat }
POST   /.../nodes/n1/drain   -> cordon + evict pods honoring disruption budgets (maintenance)

# the WATCH primitive that the whole control plane runs on:
GET    /.../pods?watch=true&resourceVersion=12345
  -> long-lived stream of {ADDED|MODIFIED|DELETED, object} events since version 12345

Internal node-agent / status API (data plane)

PATCH  /.../nodes/{node}/status   { conditions, capacity, lastHeartbeat }   # every ~5s
PATCH  /.../pods/{pod}/status     { phase: Running|CrashLoopBackOff, containerStatuses,
                                    conditions: [Ready] }                    # on change
# node agent does NOT receive pushes; it WATCHES pods assigned to it and reconciles locally.

Data model: the consistent datastore (etcd-style key-value, Raft-backed)

Everything is a versioned object under a key path, with an in-memory watch cache in the API server. The store is strongly consistent (linearizable) with compare-and-swap on a per-object resourceVersion - this is what makes every controller write safely conditional.

Pod object (the hot, numerous object - ~450K of them):

key: /pods/{namespace}/{name}
spec:   { containers:[{image, resources{requests,limits}, probes}],
          nodeName: STRING|null,          # null = unscheduled (scheduler watches these)
          affinity, tolerations, priorityClass, terminationGracePeriodSeconds }
status: { phase: Pending|Running|Succeeded|Failed,
          conditions: [Ready, ContainersReady],
          podIP, hostIP, startTime, containerStatuses:[{ready, restartCount, state}] }
metadata: { labels{}, ownerReferences:[ReplicaSet uid],   # who owns me (for GC)
            resourceVersion: BIGINT,       # CAS token - every write checks this
            deletionTimestamp: TIMESTAMP|null,             # tombstone for graceful delete
            uid }

Node object:

key: /nodes/{name}
spec:   { podCIDR, taints:[{key,effect}], unschedulable: BOOL (cordon) }
status: { capacity{cpu,memory,pods}, allocatable{...},
          conditions: [{type:Ready, status:True|False|Unknown, lastHeartbeatTime}],
          nodeInfo, addresses }
# Ready=Unknown after missed heartbeats -> node-lifecycle controller's grace timer starts

Deployment / ReplicaSet (desired-state workload objects):

/deployments/{ns}/{name}: spec{replicas, selector, strategy, template, progressDeadline}
                          status{readyReplicas, updatedReplicas, availableReplicas, conditions}
/replicasets/{ns}/{name}: spec{replicas, selector, template}  # a Deployment owns 1..2 of these
                          status{replicas, readyReplicas}     # (old + new during a rollout)
# ownerReferences link Pod -> ReplicaSet -> Deployment; cascading delete + GC follow them.

Service / Endpoints (service discovery):

/services/{ns}/{name}:  spec{ selector{app:checkout}, clusterIP(virtual), ports }
/endpoints/{ns}/{name}: subsets:[{ addresses:[podIP...] }]  # ONLY Ready pods, maintained by
                        # the endpoints controller watching pods matching the selector.
                        # service proxies watch THIS and program local load balancing.

Lease objects (for leader election among control-plane replicas, and workload fencing):

/leases/{name}: { holderIdentity, leaseDurationSeconds, renewTime, leaseTransitions }
# scheduler + each controller run active-passive: one leader holds the lease and acts,
# standbys wait. Also the primitive workloads use for fenced singleton election.

SQL vs NoSQL - the choice and why. This is a case where the answer is neither a general SQL database nor a general NoSQL store, but a consistent, watchable key-value store (etcd: Raft-replicated, linearizable, with native watch and CAS), and the reasoning is specific. The requirements are: (1) strong consistency / linearizable CAS - a pod must not be placed twice, a rollout step must be deterministic, and every controller write is update if resourceVersion unchanged, which is a compare-and-swap the store must support natively; an eventually-consistent store would let two controllers both act on a stale read and corrupt state. (2) Native watch with resumable cursors - the entire control plane is tens of thousands of long-lived watchers that need incremental change streams keyed by a monotonic version, not polling; this is etcd’s defining feature and the reason a plain relational DB (which has no efficient “stream me changes since version X to 30,000 watchers” primitive) is a poor fit. (3) Small total state that fits in memory - a few GB, so the store keeps everything in RAM and the API server keeps a watch cache, making watch fan-out cheap. (4) No need for rich queries or joins - access is by key (/pods/{ns}/{name}) or by label/field selector over a kind, served from the cache; you do not need SQL’s query engine. The deciding factor is the combination of linearizable CAS and watch - that pair is exactly what a reconciliation-loop control plane runs on, and it is why real orchestrators (Kubernetes) use etcd rather than Postgres or a document store. The one caveat: etcd’s write throughput is the scaling ceiling (part of the next section), so you keep high-churn, high-cardinality data (events, logs, metrics) out of it and in separate stores.

Bottlenecks & Scaling

Where it breaks first, in order, and the fix for each:

1. Imperative orchestrator loses state on crash and cannot converge. One-shot commands with in-memory state cannot self-heal or recover. Fix: declarative desired state in a consistent store + level-triggered reconciliation loops. The control plane holds nothing authoritative in memory; a crash is a re-read; deploy, scale, and heal are the same converge operation. This is the single most important decision.

2. The datastore (etcd) write throughput is the hard ceiling. 15,000 nodes heartbeating every few seconds is a constant multi-thousand-writes/sec floor, and etcd does not scale writes horizontally (every write goes through Raft consensus). Fix: slow and shrink status writes - heartbeat via a tiny separate Lease object instead of rewriting the whole fat Node object each time, lengthen heartbeat periods, and coalesce status updates; keep high-churn data (events, metrics, logs) out of etcd entirely in separate stores; and shard by splitting workloads across multiple clusters (federation) rather than growing one etcd past its write ceiling. The authoritative state is small, so the limit is write rate, not size.

3. Watch fan-out to tens of thousands of watchers. Every change must reach every interested controller and node agent. Fix: the API server fronts etcd with an in-memory watch cache and serves all watches from it (etcd itself fans out to the few API servers, not to 30,000 clients); changes are streamed incrementally tagged with resource versions, and a watcher that falls behind re-lists and re-watches. Never poll; never watch the whole store when you can watch one node’s slice via a field selector.

4. API server is a throughput bottleneck / SPOF. It is the only writer and the hub of all traffic. Fix: run multiple stateless API server replicas behind a load balancer (they are stateless - all state is in etcd), so reads and watches scale horizontally; only the etcd write path is shared. Rate-limit and prioritize requests (a fairness scheme) so a buggy controller spamming writes cannot starve node heartbeats.

5. Scheduler is a single decision-maker and can double-place. Fix: run scheduler replicas active-passive via leader election (a Lease) so exactly one schedules at a time (scheduling is cheap enough for one leader at this scale); correctness against stale cache comes from optimistic-concurrency placement writes + the node agent as the final admission gate, not from the leader assumption. Bound per-pod cost with a watch-fed node cache and percentage-of-nodes scoring.

6. Node death vs network partition - the central correctness risk. Reacting fast duplicates workloads; reacting slow loses availability. Fix: a tunable grace period before declaring a node dead, recovery via tombstoning pods through normal reconciliation, leases/fencing (STONITH) to make run-twice-is-wrong workloads safe, rate-limited eviction + pod-disruption budgets to avoid cascades, and node agents that self-limit under long partition. The trade-off is explicit, not defaulted.

7. Reschedule storm after a rack/zone failure. 15,000 pods need placing at once and can overload the survivors and the scheduler. Fix: rate-limit evictions and rescheduling, process pending pods in batches, spread placements (do not stack replacements on the emptiest few nodes), and honor disruption budgets so services degrade gracefully rather than all-at-once.

8. Rollout takes the service down or cannot roll back. Recreate drops to zero; in-place mutation has no safe target. Fix: bounded-surge rolling update (two coexisting ReplicaSets, maxUnavailable/maxSurge), readiness-gated steps with the LB routing only to Ready pods, graceful connection draining on shutdown, and automatic rollback to the still-present old ReplicaSet on progress-deadline failure.

9. Thundering-herd image pulls during a rollout. Hundreds of nodes pull a new image from the registry simultaneously. Fix: image pre-pull / caching on nodes, a registry mirror / pull-through cache per zone, and peer-to-peer image distribution for large fleets; stagger the rollout so pulls spread over time (the surge bound already paces this).

10. Hot node / resource stranding and skew. Greedy placement overloads some nodes and strands resources on others. Fix: bin-pack or spread by policy in the scoring phase, anti-affinity as a hard filter so replicas spread across failure domains, descheduler sweeps that rebalance long-running pods off hot nodes, and enforce requests/limits (cgroups) so a noisy neighbor cannot starve co-tenants.

11. Control-plane total outage. etcd quorum lost or all API servers down. Fix: the data plane is designed to survive it - running pods keep running, node agents keep restarting crashed containers and serving traffic from cached pod specs; you lose the ability to schedule/heal/deploy until recovery, but no workload goes down. Run etcd as a 3 or 5-node Raft quorum across failure domains with regular backups, and size so a minority partition correctly refuses writes rather than splitting brain.

12. Secret and config sprawl / blast radius. A bad config or secret rollout hits everything at once. Fix: version ConfigMaps/Secrets as objects, roll config changes through the same gradual, health-gated deployment mechanism (a config change triggers a rollout), encrypt secrets at rest in etcd, and scope RBAC so a controller only writes its own object kinds.

Wrap-Up

The trade-offs that define this design:

  • Declarative desired state + reconciliation loops, not imperative commands. Desired state is the source of truth in a consistent store; the control plane is independent, level-triggered, idempotent loops that drive actual toward desired. We trade the obviousness of “run this command” for a system where deploy, scale, self-heal, and recover are the same operation and a control-plane crash is just a re-read. This separation is the whole design.
  • Scheduling is a proposal; the node commits it. A filter-then-score scheduler writes placement with optimistic concurrency, but the node agent’s local admission check is the real capacity gate - so scheduling runs on a possibly-stale cache without ever corrupting a node, and anti-affinity lives in the filter so replicas spread.
  • You cannot tell dead from partitioned, so you choose. A tunable grace period is an explicit availability-vs-safety trade-off; recovery flows through normal pod-tombstoning reconciliation; and fencing/leases make the run-twice-is-wrong case safe. Fast reaction to an ambiguous silence is the bug; a deliberate, fenced, rate-limited response is the fix. The orchestrator gives at-least-one by default and at-most-one only to workloads that use fencing - and says so.
  • Zero-downtime rollout is bounded convergence with a safety net. Two coexisting ReplicaSets whose sizes a loop shifts in readiness-gated steps, the load balancer routing only to Ready pods, graceful draining on shutdown, and automatic rollback to the still-present old version when the new one misses its progress deadline. You always keep enough healthy replicas and never route to a pod that cannot serve.
  • The data plane outlives the control plane. Running workloads survive a total control-plane outage; you lose scheduling and healing, not uptime. The authoritative state is small enough to fit in memory, so the scaling limit is the datastore’s write rate and watch fan-out, which you fight with leases, caching, and multi-cluster sharding - not bigger disks.

One-line summary: a declarative desired state in a linearizable, watchable key-value store is driven toward reality by a set of stateless, level-triggered reconciliation loops - a filter-then-score scheduler that proposes placements the node commits, a node-lifecycle loop that recovers from failure through a grace-period-gated, fenced, rate-limited tombstone-and-reschedule, and a deployment loop that rolls out new versions in bounded, readiness-gated steps with automatic rollback - where self-healing, scaling, failure recovery, and zero-downtime deploys are all the same move: change the state and let the loops converge.