A device management platform sounds like a CRUD app over a fleet table until you say the numbers out loud. One billion devices. Each one holds a live connection and pushes a telemetry reading every 30 seconds. Each one must be reachable, right now, to receive a command or a firmware update that you decide to send from the cloud. The interviewer lets “it’s just MQTT” sit for a beat and then asks the questions that break it: where does a billion live TCP connections terminate, how do you find the one server holding the socket for device d-8831xy when you need to push it a command, what happens to the 33 million telemetry messages per second, how do you roll firmware to 50 million devices without melting your own download tier or bricking the fleet, and what happens when a device that was online three seconds ago vanishes mid-update.
The core problem is not storage and it is not the dashboard. It is a persistent, bidirectional connection layer that scales horizontally while letting you address any single device out of a billion by name. Downstream telemetry is a firehose; upstream commands are needle-in-haystack routing to a socket whose location changes every time a device reconnects. Get the connection layer and its routing right and the rest is pipelines and object storage. Get it wrong and nothing else matters. Let me build it properly.
Functional Requirements (FR)
In scope:
- Device connectivity. Devices open and hold a persistent, authenticated, bidirectional connection (MQTT over TLS). The platform tracks which devices are online and where their socket lives.
- Telemetry ingestion. Each device publishes a telemetry payload every ~30s. The platform ingests, buffers, and lands it durably for downstream consumers (time-series store, stream processors, alerting).
- Command dispatch (cloud to device). An operator or service sends a command (
reboot,set-config,read-sensor) to a specific device or a group. The platform routes it to the exact edge node holding that device’s connection and delivers it, with ack/retry semantics. - Device state and shadow. A “device shadow” (reported state + desired state) so the cloud can express intent (“desired firmware = v4.2”) and reconcile it when the device next connects, even if the device was offline when the intent was set.
- OTA firmware updates. Push a firmware image to a target cohort, staged and rate-limited, with progress tracking, health checks, and automatic rollback on a bad batch.
- Fleet registry and lifecycle. Provision, authenticate, name, group, and decommission devices. Query “which devices are on firmware < v4.0” or “which are offline > 24h.”
- Connection state and last-will. Detect disconnects (keepalive), fire a last-will message, and expose online/offline presence.
Explicitly out of scope (say it so you own the scope):
- The device firmware itself and the physical hardware. I design the cloud platform and the on-device agent’s protocol contract, not the embedded code.
- The analytics and ML on telemetry. Anomaly detection, predictive maintenance, dashboards - they consume the telemetry stream I produce. I land the data cleanly and stop there.
- Billing and the customer-facing web console. Separate systems that call my APIs.
- The CDN build-out. I use an object store + CDN for firmware images as a dependency; I do not design the CDN.
The one decision that drives everything: the connection plane and the data plane are separate systems with separate scaling laws. The connection plane is stateful (it holds sockets and must be addressable per device); the data plane is a stateless, horizontally sharded stream pipeline. Conflate them and you cannot scale either. Keep them apart, joined only by a routing directory, and both become tractable.
Non-Functional Requirements (NFR)
- Scale: 1B registered devices, ~700M concurrently connected at peak (not every device is always on). Telemetry every 30s. OTA campaigns to cohorts of tens of millions.
- Connection density: a single edge node (well-tuned Linux, epoll, tuned file descriptors) holds ~250k-500k concurrent MQTT connections. 700M / 350k ≈ 2,000 edge nodes just to terminate connections. This number sizes the whole connection tier.
- Ingestion latency: telemetry should be durably buffered within ~1s of arrival at the edge. It is not a user-facing read path; throughput matters more than per-message latency.
- Command latency: a command to an online device should be delivered p99 under ~2s end to end. This is the interactive path and the hard one, because it requires finding the socket.
- Availability: 99.9%+ for ingestion and command dispatch. A device must always be able to reconnect somewhere; no single edge node failure may strand a region.
- Consistency: telemetry is eventually consistent and append-only (lost duplicates are fine, strict ordering is per-device only). The device shadow is read-your-writes per device - desired state set by an operator must be the version the device reconciles against. The connection routing directory is eventually consistent but fast-converging (a few seconds of staleness means a retry, not a lost command).
- Durability: telemetry must be durably persisted (at-least-once) before the edge acks. Firmware images and the device registry are durable and backed up. Live connection state is ephemeral - it is rebuilt on reconnect.
- Safety: OTA must never brick the fleet. Staged rollout, health gates, and automatic halt/rollback are first-class requirements, not features.
Back-of-the-Envelope Estimation (BoE)
Real numbers with the arithmetic, because they size every tier.
Connections:
1B registered, ~70% concurrently online at peak = 700M live connections.
Per edge node (epoll, ~8GB for connection state, tuned fds) ≈ 350k conns.
Edge nodes needed = 700,000,000 / 350,000 = 2,000 nodes.
Per-connection memory ≈ 20KB (buffers + session + TLS state).
700M * 20KB = 14 TB of connection-state RAM across the tier.
Two thousand connection terminators. That alone tells you the connection tier is its own fleet with its own autoscaling, and that per-connection overhead (keepalive, memory) is a primary cost driver.
Telemetry ingest (the firehose):
700M online devices, 1 message / 30s = 700,000,000 / 30 ≈ 23M messages/sec.
With retries, reconnect bursts, and the long tail, size for ~33M msg/sec peak.
Payload ≈ 200 bytes (device_id, ts, a handful of sensor floats).
Ingest bandwidth = 33M * 200 bytes ≈ 6.6 GB/sec = ~53 Gbps sustained.
Telemetry storage/year:
23M msg/sec * 86,400 s/day ≈ 2 trillion messages/day.
2e12 * 200 bytes ≈ 400 TB/day raw.
With compression (time-series columnar, ~8x) ≈ 50 TB/day stored.
Per year ≈ 18 PB compressed. Hot tier keeps ~7-30 days; the rest tiers to
cheap object storage, and most raw telemetry is downsampled/rolled up and
expired via TTL, not kept raw forever.
That storage number forces the point: you cannot keep raw per-device telemetry indefinitely. The design must downsample and tier aggressively.
Command / dispatch path (upstream):
Commands are rare vs telemetry: maybe 1M commands/sec at a busy peak
(bulk config pushes, OTA notifications). Each requires ONE routing lookup
(device_id -> edge node) plus a delivery. The lookup is the cost, not volume.
Routing directory size:
Online device -> (edge_node_id, session). 700M entries.
Each ~40 bytes (device_id + node_id + session_epoch + ts).
700M * 40 bytes ≈ 28 GB. Sharded across a cluster, this is a modest
in-memory/KV footprint - the challenge is write churn, not size.
The directory churns on every connect/disconnect. At 700M devices with, say, a 2-hour average session, that is 700M / 7200s ≈ ~100k directory writes/sec just from reconnect churn, before any mass-reconnect event.
OTA download tier:
Firmware image ≈ 8 MB. A campaign to 50M devices = 50M * 8MB = 400 TB
of egress. At a naive "notify everyone at once," 50M devices pulling 8MB
in a 5-minute window = 400TB / 300s ≈ 1.3 TB/sec = 10+ Tbps. Impossible
without staging. This single number is why OTA MUST be rate-limited and
CDN-fronted, not a broadcast.
The takeaways: a 2,000-node stateful connection tier (14TB of socket RAM), a 33M msg/sec telemetry firehose that must be sharded and tiered (18PB/year), a routing directory that is small in bytes but hot in writes (~100k+ writes/sec of pure churn), rare but latency-sensitive command routing that hinges on a single lookup, and an OTA path that is an egress bomb unless staged. The connection tier, the routing directory, and OTA staging are the design drivers - not raw storage.
High-Level Design (HLD)
Two planes joined by a directory. The connection plane (edge gateway fleet) terminates device connections and is stateful. The data plane (ingest bus + processors + stores) is stateless and sharded. A routing directory maps device_id -> edge_node so the command path can find any device’s live socket. A control plane (registry, shadow, OTA orchestrator) holds durable truth about devices and drives intent.
┌──────────────────────────────────────────┐
1B devices │ CONTROL PLANE │
(MQTT/TLS) │ ┌────────────┐ ┌──────────┐ ┌────────┐ │
│ │ │ Registry │ │ Device │ │ OTA │ │
│ │ │ (provision, │ │ Shadow │ │ Orch. │ │
│ │ │ authz) │ │ (desired/ │ │(stage, │ │
│ │ └────────────┘ │ reported) │ │ gate) │ │
│ │ └──────────┘ └────────┘ │
│ └───────▲───────────────▲──────────────▲────┘
│ │ shadow/cmd │ │
┌────▼─────────────────┐ │ │ │
│ LOAD BALANCER (L4) │ │ │ enqueue cmd │
│ + connection-aware │ │ │ │
│ routing │ ┌─────┴──────┐ ┌────┴─────────┐ │
└────┬─────────────────┘ │ Command │ │ Routing │ │
│ sticky-ish │ Dispatcher │───▶│ Directory │ │
┌────▼──────────────────┐ │ (cmd topic │ │ device_id -> │ │
│ EDGE GATEWAY FLEET │ │ per node) │ │ edge_node │ │
│ ~2000 nodes │◀─┤ │ │ (sharded KV) │ │
│ ┌──────┐ ┌──────┐ │ └─────────────┘ └──────────────┘ │
│ │node 1│..│node N│ │ ▲ ▲ │
│ │350k │ │350k │ │ │ route cmd │ register/ │
│ │conns │ │conns │ │─────────┘ │ heartbeat │
│ └──┬───┘ └──┬───┘ │────────────────────────────┘ │
└─────┼─────────┼────────┘ │
│ publish telemetry │
┌─────▼─────────▼─────────────────────────────────────────┐ │
│ INGEST BUS (Kafka/Pulsar, topic partitioned by │ │
│ hash(device_id)) 33M msg/sec │ │
└─────┬───────────────────────────┬────────────────────────┘ │
│ │ │
┌─────▼──────────┐ ┌───────────▼───────────┐ ┌───────────────▼┐
│ Stream proc. │ │ Time-series store │ │ Shadow updater │
│ (rollups, │ │ (hot: Cassandra/ │ │ (reported state │
│ alerting, │ │ ClickHouse; cold: │ │ -> shadow) │
│ anomaly) │ │ S3 + TTL) │ └─────────────────┘
└─────────────────┘ └────────────────────────┘
┌──────────────────┐
firmware pull (HTTPS) ─────────────────────────▶│ Object store + │
│ CDN (firmware) │
└──────────────────┘
Connect path (device comes online):
- Device resolves the platform endpoint, the L4 load balancer hands it to an edge gateway node, and the device opens an MQTT-over-TLS connection.
- The edge node authenticates the device (mutual TLS / per-device cert or token against the Registry), establishes the MQTT session, and registers the device in the Routing Directory:
device_id -> {edge_node_id, session_epoch, connected_at}. - The edge node checks the Device Shadow for pending desired state (e.g. “desired firmware v4.2”, “config v9”) and reconciles - pushing any pending command the device missed while offline.
Telemetry path (downstream firehose), continuous:
- Device publishes telemetry to its MQTT topic every 30s. The edge node receives it, does minimal validation, and publishes to the Ingest Bus (Kafka/Pulsar) on a topic partitioned by
hash(device_id). The edge acks the device only after the bus accepts the write (at-least-once durability). - Stream processors consume the bus: one path lands raw/rolled-up telemetry into the time-series store, another runs alerting/anomaly detection, another updates the device shadow’s “reported state.”
Command path (upstream, the hard one), per command:
- A service/operator calls the Command Dispatcher: “send
reboottod-8831xy.” - The dispatcher looks up
d-8831xyin the Routing Directory, getsedge_node_42, and forwards the command to that node (via a per-node command topic / direct RPC). edge_node_42finds the live socket ford-8831xyand pushes the command down the MQTT connection. The device acks; the ack flows back and the command is marked delivered. If the device is offline, the command is written to the shadow/queue for delivery on next connect.
OTA path:
- The OTA Orchestrator defines a campaign (target cohort, image, rollout policy), uploads the image to the object store/CDN, and sets
desired firmware = vXon the shadows of the cohort in staged batches. - Each batch’s devices receive a command/shadow-delta pointing to a CDN URL (not the image itself over MQTT). They download over HTTPS, verify a signature, flash, reboot, reconnect, and report the new version. The orchestrator watches health and only releases the next batch if the current one is healthy.
The key insight: the connection plane is the only stateful, per-device-addressable tier, and the Routing Directory is the sole bridge that lets the stateless command path find a stateful socket. Telemetry never needs addressing (it is a sharded stream); commands always do (they are per-device), and that asymmetry defines the architecture.
Component Deep Dive
The hard parts, naive-first then evolved: (1) the horizontally scalable persistent connection layer, (2) routing a command to one device out of a billion, (3) the telemetry ingestion firehose, (4) OTA at fleet scale without bricking anything.
1. The persistent bidirectional connection layer
Naive approach: one big MQTT broker (or a small cluster) that every device connects to. Devices connect, publish telemetry, the broker fans it out and lets you publish commands back by topic.
Where it breaks:
- A single broker cannot hold 700M connections. Even the best brokers top out in the low millions of connections per node due to file descriptors, memory, and epoll scaling. You need thousands of nodes, which a monolithic broker is not.
- Vertical scaling dead-ends. You cannot buy a machine with 14TB of connection-state RAM and 700M file descriptors. The connection count forces horizontal scaling from the first day.
- Broker-internal topic routing does not scale to per-device addressing. Classic MQTT fan-out assumes topic-based pub/sub; routing a command to one specific device out of a billion by walking broker subscription tables is not what brokers are built to do efficiently at this scale.
- Reconnect storms. If one broker dies holding millions of connections, they all reconnect at once and hammer whatever is left. A monolith has no blast-radius control.
First evolution: a horizontal fleet of stateless-ish edge gateway nodes behind an L4 load balancer. Each node is a connection terminator holding ~350k devices. Add nodes to add capacity. This is right, but it exposes the real problem immediately: once device X’s socket lives on node 42, how does anything in the cloud know to send node 42 a command for X? The connections are spread across 2,000 nodes with no inherent map. That is the question the whole design turns on.
The answer: edge nodes own connections and continuously publish their ownership to a Routing Directory; the directory is the index over the connection fleet.
- Edge node responsibilities. Terminate TLS, authenticate the device against the Registry, maintain the MQTT session and keepalive, forward inbound telemetry to the Ingest Bus, receive commands addressed to its devices and push them down the sockets. The node holds the socket; it does not hold durable truth - if it dies, its devices reconnect elsewhere and re-register.
- Registration on connect. The moment a device’s session is established, the node writes
device_id -> {node_id, session_epoch}to the Routing Directory with a TTL/lease. Thesession_epochis a monotonic token so a stale (older) registration can never overwrite a newer one after a fast reconnect. - Heartbeat / lease renewal. The node renews the lease for its devices periodically. If the node dies, leases expire and the directory entries clear; the devices reconnect (to a new node) and re-register. This makes the directory self-healing without a central failure detector.
- Keepalive tuning is load-bearing. 700M devices with an MQTT keepalive means the edge tier is processing keepalive traffic constantly. Push keepalive intervals long (devices sending telemetry every 30s already prove liveness - telemetry itself can serve as the keepalive, so you set MQTT keepalive to, say, 90s and let the 30s telemetry cadence keep it alive without extra PINGREQ).
- Blast radius and reconnect control. Spread nodes across many availability zones; cap how fast a single node accepts new connections (connection admission rate limiting) so a reconnect storm after a node loss is absorbed gradually, not as a thundering herd. Use jittered client-side reconnect backoff in the device agent contract so a region’s devices do not all reconnect on the same tick.
device connect ──▶ L4 LB ──▶ edge_node_42
edge_node_42: auth(device) ; open MQTT session
Directory.PUT device_id -> {node:42, epoch:17} (lease 120s)
check Shadow(device_id) for pending desired state -> push
every 60s: Directory.RENEW lease for all my devices
on node death: leases expire ; devices reconnect to node K ; re-register
The mental model: the edge fleet is a distributed hash of sockets, and the Routing Directory is the lookup table that turns device_id into “which node holds the socket right now.” The fleet scales by adding nodes; addressing scales by keeping the directory fast and correct through reconnect churn.
2. Routing a command to one device out of a billion
Naive approach: broadcast the command to all edge nodes and let the one holding the device deliver it. Dispatcher sends reboot d-8831xy to all 2,000 nodes; node 42 recognizes it owns the device and delivers, the rest ignore it.
Where it breaks:
- Fan-out amplification. Every command hits all 2,000 nodes. At 1M commands/sec that is 2 billion node-messages/sec of wasted work. The 1,999 nodes that do not own the device burn CPU filtering.
- No targeting for groups. Sending to a cohort of 10M devices by broadcasting 10M commands to all nodes is catastrophically wasteful.
- It does not actually solve “where is it.” You have replaced a lookup with a broadcast, which scales inversely with fleet size - the bigger the fleet, the worse broadcast gets.
First evolution: look the device up in the Routing Directory, send only to the owning node. Dispatcher does GET d-8831xy -> node_42, forwards only to node 42. One targeted hop. This is correct and it is the design - but it surfaces the real subtleties: the directory must be fast under read and write churn, sharded (28GB and 100k+ writes/sec), and correct across reconnects (the device may have moved between lookup and delivery).
The refined design: a sharded, leased Routing Directory plus a race-safe delivery protocol.
- Shard the directory by
hash(device_id). A distributed KV store (Redis Cluster, or a purpose-built in-memory sharded service) partitions the 700M entries across many shards. Reads and writes for a device always hit the same shard; no cross-shard coordination. 100k writes/sec of churn spread over, say, 100 shards is 1k writes/sec/shard - trivial. - Leases, not permanent entries. Each entry carries a TTL renewed by the owning node’s heartbeat. A dead node’s entries expire automatically. The
session_epochguards against a reconnect race: if the device reconnected to node 50 while the dispatcher was mid-flight to node 42, node 42 sees the device is gone (socket closed) and NACKs; the dispatcher re-reads the directory (now pointing at node 50 with a higher epoch) and retries. The epoch prevents an old node from “winning” a stale write. - Command delivery is at-least-once with device acks. The command carries a
command_id. The device acks on receipt; the edge node relays the ack to the dispatcher, which marks it delivered. No ack within a timeout -> retry (re-resolve + resend). Devices dedupe oncommand_idso a retried command is not executed twice (idempotency on the device side is part of the agent contract). - Offline devices go through the shadow, not the directory. If the lookup misses (device offline) or delivery fails terminally, the command is written as desired state in the Device Shadow. When the device reconnects, the edge node reconciles the shadow and delivers the pending command. This is how you get “reliable eventually” delivery to a fleet that is constantly going offline - the shadow is the durable queue, the directory is the fast path for online devices.
- Group / cohort commands fan out through the data plane, not per-device RPC from one dispatcher. For “reboot all v3.x devices,” the control plane resolves the cohort from the Registry and writes desired state to shadows in batches; online devices get a nudge via their owning nodes (resolved per shard in bulk), offline ones reconcile on reconnect. The fan-out is parallelized across directory shards and edge nodes, never funneled through a single sender.
POST /commands { device:"d-8831xy", cmd:"reboot" }
dispatcher: node,epoch = Directory.GET("d-8831xy") # -> node_42, epoch_17
send(node_42, {device, cmd, command_id, epoch_17})
node_42: if session(device).epoch != 17 or socket closed: NACK(stale)
else push over MQTT ; await device ACK ; relay ACK
dispatcher: on ACK -> delivered ; on NACK/timeout -> re-GET + retry
on device offline -> Shadow.setDesired(device, cmd) # deliver on reconnect
The principle: online delivery is a single sharded lookup plus a race-safe, epoch-guarded, idempotent push; offline delivery degrades to durable desired-state in the shadow. Broadcast is never used; the directory turns a billion-device haystack into an O(1) shard read.
3. The telemetry ingestion firehose
Naive approach: edge nodes write each telemetry message straight into the database. Device publishes, edge node does an INSERT into the time-series DB.
Where it breaks:
- 33M writes/sec will not land directly on a database. No OLTP database, and few time-series stores, take 33M individual writes/sec without buffering and batching in front. The edge tier would block on DB latency and back-pressure onto the connections.
- Coupling connection health to DB health. If the database slows, edge nodes slow, keepalives get missed, connections drop, devices reconnect en masse, and now a DB hiccup becomes a connection storm. The firehose must be decoupled from storage.
- No replay, no multiple consumers. Writing straight to the DB means alerting, anomaly detection, and shadow updates each have to re-query the DB or you have to multi-write. There is no single durable log to fan out from.
First evolution: put a durable log (Kafka/Pulsar) between the edge and everything else. Edge nodes publish telemetry to the Ingest Bus; consumers read from it. This decouples ingestion from storage, gives replay, and lets many consumers share one stream. Right move - but now partitioning and ordering matter.
The design: a log partitioned by hash(device_id), with per-device ordering, tiered storage, and aggressive downsampling.
- Partition by
hash(device_id). All of a device’s telemetry lands on one partition in order, which preserves per-device ordering (the only ordering that matters - you never need a global order across devices). Thousands of partitions spread the 33M msg/sec across the cluster. Partition count is sized for throughput and consumer parallelism, and the key choice avoids hot partitions as long as device IDs are well-distributed. - Batch at the edge. Edge nodes batch many devices’ messages into larger produce requests to the bus (e.g. flush every 200ms or 1MB) so the bus sees high-throughput batched writes, not 33M tiny ones. The edge acks the device after the batch is durably accepted (at-least-once).
- Consumers, each its own concern. (a) A storage sink writes to the time-series store in batches. (b) A stream processor (Flink) does windowed rollups (per-device 1-min/1-hour aggregates), threshold alerting, and anomaly detection. (c) A shadow updater reflects the latest reported values into the device shadow. Each consumer group reads the same log independently and can replay.
- Time-series store, hot + cold tiering. Hot store (Cassandra or ClickHouse) keeps recent raw/rolled data (7-30 days) for queries and alerting, partitioned by
device_idand clustered by time. Cold store is object storage (Parquet on S3) for historical, queried rarely via a lakehouse engine. A TTL expires raw hot data; only rollups and cold archives survive long term. This is what keeps 18PB/year from becoming 18PB of expensive hot storage. - Downsample on the write path. Most telemetry is uninteresting (a temperature that did not change). The stream processor rolls raw 30s samples into 1-minute and 1-hour aggregates; raw data is retained short-term, aggregates long-term. Optionally the device agent sends deltas / only-on-change for slow-moving signals, cutting ingest volume at the source.
- Back-pressure that protects the connection tier. If the bus or a consumer lags, the edge buffers briefly and, if needed, sheds or samples telemetry rather than dropping connections. Losing a telemetry sample is acceptable (eventual consistency, at-least-once best effort); dropping a connection and triggering a reconnect storm is not. Telemetry loss degrades gracefully; connection loss cascades.
edge nodes ──batch(200ms)──▶ Ingest Bus [partitions = hash(device_id)]
│ │ │
┌───────────┘ ┌────────┘ ┌──────┘
┌──────▼──────┐ ┌──────▼───────┐ ┌───▼─────────┐
│ storage sink │ │ Flink: rollup │ │ shadow │
│ -> TS store │ │ + alert + │ │ updater │
│ (hot, TTL) │ │ anomaly │ │ (reported) │
└──────┬───────┘ └───────────────┘ └─────────────┘
rollups + raw expiry ──▶ S3 (cold, Parquet) for history
The load-bearing decisions: a durable partitioned log decouples the firehose from storage and the connection tier; partitioning by device_id gives free per-device ordering and even spread; and tiering + downsampling keep the storage bill finite. Ingestion is a stateless sharded stream problem - the opposite shape from the stateful connection tier - and that is exactly why they are separate planes.
4. OTA firmware updates at fleet scale
Naive approach: when a new firmware drops, push it to every target device at once over the connection. Set desired firmware on all 50M devices, they download and flash.
Where it breaks:
- The egress bomb. 50M devices * 8MB in a tight window is terabytes-per-second of egress (see BoE: ~10+ Tbps). Your own infrastructure cannot serve it; pushing an 8MB image over MQTT connections is even worse - the connection tier is not a file server.
- Simultaneous reboot = mass reconnect storm. 50M devices flashing and rebooting at once all disconnect and reconnect together, slamming the connection tier and the Routing Directory with tens of millions of re-registrations in seconds.
- No safety net. If the firmware is bad, you have just bricked 50M devices with no way to stop or roll back. One bad image, whole fleet down. This is the nightmare scenario the design exists to prevent.
- Offline devices missed. Devices offline during the push never get the update, and there is no reconciliation.
The design: signed images on a CDN, desired-state via the shadow, staged canary rollout with health gates and automatic halt/rollback.
- Images live on object storage + CDN, delivered over HTTPS, not over the connection. The OTA command/shadow-delta carries a CDN URL + version + signature/hash, a few hundred bytes. Devices pull the 8MB image from the CDN, which is built to serve massive parallel egress and caches at the edge. The connection tier only ever carries the tiny pointer, never the payload.
- Desired state through the shadow makes it reliable for a flaky fleet. The orchestrator sets
desired.firmware = vXon each target device’s shadow. Online devices are nudged via their owning edge node; offline devices reconcile on reconnect. Either way the intent is durable - a device that was unplugged for a week gets the update when it comes back. The shadow is the reconciliation ledger. - Staged / canary rollout. Never all at once. Roll in cohorts: 0.1% canary, then 1%, 5%, 25%, 100%, each gated. Within a stage, rate-limit the devices told to download (a token-bucket releasing, say, N devices/sec) so CDN egress and the reconnect rate stay bounded - this directly defuses both the egress bomb and the reconnect storm.
- Health gates between stages. After each stage, the orchestrator watches the cohort’s reported health via telemetry/shadow: did devices reconnect, report the new version, and resume normal telemetry within a window? Compute a success rate. If success < threshold (devices bricking, crash-looping, not reporting back), halt the rollout automatically and do not release the next stage. This is the mechanism that bounds a bad image’s blast radius to the canary.
- Automatic rollback. Devices keep an A/B partition scheme (two firmware slots): flash the inactive slot, boot it, and if the new image fails a self-test or does not get a “commit” from the cloud within a timeout, roll back to the known-good slot on the next boot. The platform can also push a “revert to vX-1” desired state to a cohort that got a bad image. A/B + commit-on-health is what makes “never brick the fleet” actually true - a failed flash falls back, it does not brick.
- Signature verification on device. The device verifies the image signature/hash before flashing (part of the agent contract), so a corrupted or tampered download is rejected, not flashed. The platform never ships an unsigned image.
- Progress and resumability. Devices report OTA progress (
downloading,downloaded,flashing,rebooting,committed,failed) into their shadow. The orchestrator tracks per-device and per-cohort progress; interrupted downloads resume via HTTP range requests rather than restarting.
OTA campaign(target cohort, image vX):
upload image -> object store -> CDN ; sign(image)
for stage in [0.1%, 1%, 5%, 25%, 100%]:
pick stage devices from cohort
rate-limited release (N/sec):
Shadow.setDesired(device, {firmware:vX, url:cdn/..., sig:...})
online? nudge via edge node : deliver on reconnect
device: pull from CDN (HTTPS, resumable) -> verify sig
-> flash INACTIVE slot -> reboot into it -> self-test
-> report health ; cloud "commit" or auto-rollback to good slot
wait + measure cohort success rate
if success < THRESHOLD: HALT (page on-call) ; else next stage
The synthesis: OTA separates the heavy payload (CDN, HTTPS, rate-limited) from the lightweight intent (signed pointer via the durable shadow), and wraps the whole thing in a staged, health-gated, A/B-rollback controller. The connection tier carries only pointers; the CDN carries bytes; the shadow carries intent; and the canary + health gate + A/B rollback make a bad image a non-event instead of a fleet-wide brick.
API Design & Data Schema
The device-facing protocol is MQTT; the operator/service-facing control is REST; the data plane is streams.
Device-facing (MQTT topics, per the agent contract)
# device publishes telemetry
PUB devices/{device_id}/telemetry { ts, metrics:{temp:41.2, rpm:1180, ...} }
# device publishes reported state / OTA progress
PUB devices/{device_id}/state { firmware:"v4.1", config_ver:9, ota:"committed" }
# device subscribes for commands + desired state deltas
SUB devices/{device_id}/commands { command_id, type:"reboot"|"set-config"|..., args }
SUB devices/{device_id}/desired { firmware:"v4.2", url, sig, config:{...} }
# device acks a command
PUB devices/{device_id}/ack { command_id, status:"ok"|"failed", detail }
Operator / service-facing (REST, control plane)
POST /api/v1/devices # provision
body: { device_id, cert_fingerprint, group, meta }
-> 201 { device_id, status:"registered" }
POST /api/v1/devices/{id}/commands # send a command to one device
body: { type:"reboot", args:{} }
-> 202 { command_id, delivery:"online"|"queued_offline" }
GET /api/v1/devices/{id}/shadow # read desired + reported state
-> 200 { reported:{firmware:"v4.1",...}, desired:{firmware:"v4.2",...},
connected:true, edge_node:"node_42", last_seen:... }
PUT /api/v1/devices/{id}/shadow/desired # set intent (reconciled on connect)
body: { config:{sample_rate:15} }
-> 200 { version: 10 }
POST /api/v1/ota/campaigns # launch a staged OTA rollout
body: { target:{group:"v3.*"}, image_url, version:"v4.2",
stages:[0.1,1,5,25,100], health_threshold:0.98, rate_per_sec:2000 }
-> 202 { campaign_id, status:"canary" }
GET /api/v1/ota/campaigns/{cid} # rollout progress + health
-> 200 { stage:"5%", success_rate:0.991, devices_done:N, status:"running" }
GET /api/v1/devices?firmware_lt=4.0&offline_gt=24h # fleet queries
-> 200 { devices:[...], cursor }
Data stores - the right tool per plane
1. Routing Directory - sharded in-memory KV (Redis Cluster / purpose-built), NOT a relational DB. device_id -> {edge_node, session_epoch, lease_expiry}. 700M keys, ~28GB, sharded by hash(device_id). Leased with TTL, renewed by edge heartbeats. In-memory because it is read on every command and written on every connect/disconnect (100k+ writes/sec); durability is pointless since it is rebuilt on reconnect.
Redis Cluster (sharded by hash(device_id)):
SET route:{device_id} {node, epoch} EX 120 # on connect, renewed
GET route:{device_id} # command routing lookup
# entry self-expires if the owning node dies (lease not renewed)
2. Device Registry + Shadow - a durable, sharded NoSQL store (DynamoDB / Cassandra). Device identity, auth material, group, lifecycle, and the shadow (desired + reported state). Sharded by device_id (the natural partition key). NoSQL because access is almost always a single-key lookup by device_id at 1B rows - no joins, no cross-device transactions - and it must scale horizontally far past what a single relational node holds.
Table: devices (partition key = device_id)
device_id (PK) | cert_fp | group | firmware | config_ver
| reported_state (json) | desired_state (json)
| last_seen | status | created_at
GSI: group, GSI: firmware # for cohort / fleet queries
A SQL store is kept only for low-cardinality relational control data (users, OTA campaign definitions, audit) where transactions and joins matter and volume is small. The 1B-row device data is NoSQL - the access pattern is a pure key-value lookup at a scale no single SQL instance survives.
3. Telemetry store - time-series (Cassandra / ClickHouse) hot + object storage cold. Partitioned by device_id, clustered by time; TTL on raw hot data; rollups and cold Parquet on S3 for history. Chosen for high write throughput and time-range scans per device, not point consistency.
TS table (partition = device_id, clustering = ts DESC):
device_id | ts | metric | value
TTL 30d on raw ; rollups (1m/1h) kept 1-2y ; cold -> S3 Parquet
4. Ingest Bus - Kafka/Pulsar, partitioned by hash(device_id). The durable log between edge and consumers. Append-only, replayable, the single fan-out point for storage/alerting/shadow. Retention ~days (replay buffer), not the system of record.
5. Firmware images - object store + CDN. Immutable, signed binaries addressed by content hash. Durable, versioned, globally cached for massive parallel pull.
Why this split: the directory is ephemeral and write-hot, so in-memory KV; device identity/shadow is durable, huge, single-key, so sharded NoSQL; telemetry is append-only time-series at petabyte scale, so a TS store with tiering; the bus is a replayable log, so Kafka; images are big immutable blobs, so object store + CDN. There is no single “IoT database” - each plane’s lifetime and access pattern gets its own store.
Bottlenecks & Scaling
Where it breaks first, in order, and the fix for each:
1. Connection tier capacity and reconnect storms (breaks first). No node holds 700M connections; and when a node (or AZ) dies, millions reconnect at once. Fix: horizontal edge fleet (~2,000 nodes, ~350k conns each) behind an L4 LB, spread across AZs/regions. Connection admission rate limiting per node and jittered client reconnect backoff in the agent contract absorb storms. Let 30s telemetry double as keepalive to cut keepalive load. Capacity scales linearly by adding nodes.
2. The Routing Directory under write churn. 100k+ writes/sec of pure connect/disconnect churn, spiking during mass reconnects, plus a read on every command.
Fix: shard by hash(device_id) so churn and reads spread evenly (1k writes/sec/shard over 100 shards). Leases with TTL make it self-healing (dead node’s entries expire). session_epoch guards reconnect races so stale writes never win. During a mass-reconnect event, the per-shard write rate rises but stays bounded and parallel.
3. Telemetry firehose overwhelming storage. 33M msg/sec cannot hit a database directly, and coupling DB latency to the edge causes connection drops.
Fix: a durable partitioned log (Kafka) decouples edge from storage; batch at the edge; partition by device_id for ordering and spread; tier hot/cold with TTL and downsample to rollups so 18PB/year does not sit in hot storage. Back-pressure sheds telemetry (acceptable) before it ever drops connections (cascading).
4. Command fan-out to large cohorts. A 10M-device command funneled through one dispatcher or broadcast to all nodes melts something. Fix: resolve cohorts in the control plane, write desired state to shadows in batches parallelized across directory shards and edge nodes, nudge online devices per shard, reconcile offline ones on reconnect. Never broadcast; never single-sender fan-out.
5. OTA egress bomb and simultaneous reboot. 50M devices pulling 8MB at once is 10+ Tbps and a fleet-wide reconnect storm. Fix: CDN-hosted signed images (connection tier carries only a tiny pointer), staged canary rollout (0.1% -> 100%), and rate-limited release (token bucket, N devices/sec) bounding both egress and reconnect rate.
6. Bricking the fleet on a bad image. One bad firmware with no safety net is a catastrophic, unrecoverable failure. Fix: health gates between stages (auto-halt if canary success < threshold), A/B partitions with commit-on-health (failed flash auto-rolls-back to the good slot), on-device signature verification, and a push-able revert desired state. A bad image stays contained to the canary.
7. Hot devices / hot shards. A misbehaving device spamming telemetry, or a skewed device_id space, overloads one partition/shard.
Fix: per-device rate limiting at the edge (drop/throttle a device exceeding its quota), and a well-distributed hash on device_id for partitioning. A single device’s load is bounded by its quota; the hash prevents systematic skew.
8. Shadow update contention. Every device’s reported state updates the shadow constantly; hot keys and write amplification can overload the registry store. Fix: coalesce reported-state updates in the shadow updater (write the latest value per device per window, not every message), and keep the write-hot ephemeral routing separate from the durable shadow so churn does not touch the registry.
9. Single points of failure. The directory, the bus, the dispatcher, the orchestrator. Fix: directory and bus are replicated and sharded with failover; edge nodes are stateless (device re-registers on reconnect); the dispatcher and OTA orchestrator are stateless services behind LBs driving durable state in the shadow/registry. Loss of any one node degrades a slice, not the fleet.
10. Multi-region. A billion devices are global; routing a command across regions on every command adds latency and a cross-region dependency.
Fix: deploy the connection + directory + ingest planes per region, home each device to its nearest region, and keep the command loop (lookup + delivery) entirely in-region. The registry/shadow and OTA orchestration replicate globally (a device’s home region owns its shadow). Cross-region is only for the rare case of addressing a device homed elsewhere, resolved via a global directory of device_id -> home_region.
Wrap-Up
The trade-offs that define this design:
- Two planes, not one system. The connection plane is stateful and per-device-addressable; the data plane is a stateless sharded stream. They scale by different laws and are kept separate, joined only by the Routing Directory. Conflating them is the mistake that makes a billion devices impossible.
- A directory over the socket fleet. Addressing one device out of a billion is a single sharded, leased, epoch-guarded lookup - never a broadcast. The directory is ephemeral and write-hot by design (rebuilt on reconnect), which is exactly why it is in-memory and sharded, not a durable database.
- Durable intent via the shadow for a flaky fleet. Devices are constantly offline, so reliable delivery cannot depend on the device being online. Desired state in the shadow is the durable queue; the directory is the fast path for online devices; reconciliation on reconnect catches the rest.
- Telemetry as a decoupled, tiered firehose. A partitioned durable log separates ingestion from storage and from the connection tier, per-device partitioning gives free ordering, and tiering + downsampling keep 18PB/year finite. Telemetry loss degrades gracefully; connection loss cascades, so back-pressure protects connections first.
- OTA as a staged controller, not a push. Big bytes on the CDN, tiny signed pointer via the shadow, canary stages with health gates, rate-limited release, and A/B rollback. The whole machine exists so a bad image hits the canary and halts, never the fleet.
One-line summary: an IoT platform where a horizontally scaled stateful edge fleet terminates a billion persistent MQTT connections and publishes socket ownership into a sharded, leased Routing Directory, telemetry flows as a device-id-partitioned durable firehose into a tiered time-series store, commands reach any single device through one epoch-guarded directory lookup with offline delivery falling back to durable desired-state in the device shadow, and OTA rolls firmware from a CDN in rate-limited, health-gated canary stages with A/B rollback - two planes joined by one directory, so addressing a billion devices is an O(1) read instead of a broadcast.
Comments