Almost every large system needs one boring-sounding thing: a way to mint unique identifiers. Tweet IDs, order IDs, message IDs, row keys. It sounds trivial until you write down the real constraints: the IDs must be unique across thousands of machines, they must be generated at hundreds of thousands per second, no single box is allowed to be a bottleneck or a point of failure, and (the part people miss) they should be roughly sortable by creation time so you can use them as primary keys and range-scan recent data.
That last requirement is what turns this from “call uuid4() and move on” into a real design. The whole problem lives at the intersection of three tensions: uniqueness without coordination, time-ordering without a global clock, and availability without a single authority. Get the ID layout right and everything downstream (indexes, sharding, pagination) gets easier. Get it wrong and you inherit fragmented B-tree indexes, hot shards, and an ID service that takes the whole company down when it hiccups.
Let me do this the way I would in an interview: naive first, break it, evolve.
Functional Requirements (FR)
In scope:
- Generate globally unique IDs. Across all machines, all data centers, forever. No two calls ever return the same ID.
- IDs are 64-bit integers. Fits in a
BIGINT, along, a database primary key. This is a hard constraint I will justify in the estimation. - Roughly time-sortable. IDs generated later should generally sort after IDs generated earlier (k-sorted / roughly monotonic). This lets us use the ID as a clustered primary key and paginate “newest first” without a separate timestamp column.
- High throughput, low latency. Callers get an ID in well under a millisecond, locally, without a network round trip per ID if we can help it.
- Numeric / compact. IDs should be as small as possible so indexes stay dense.
Explicitly out of scope (say this out loud so you control scope):
- Cryptographic unpredictability. Sequential-ish IDs are enumerable. If you need unguessable public identifiers (e.g. to stop scraping), you layer opacity on top; the generator itself optimizes for uniqueness and ordering, not secrecy.
- Human readability. These are machine keys, not coupon codes.
- Strict total ordering. We do not promise that ID A generated 2ms before ID B on another machine is numerically smaller. That would require global coordination on every call, which defeats the point. We promise rough ordering.
- Perpetual lifetime beyond ~70 years. A 41-bit millisecond timestamp gives us decades, not eternity. We design for a multi-decade epoch and move on.
The single decision that drives everything: do we coordinate on every ID, or do we partition the ID space so machines can generate independently? Coordination gives perfect ordering but a bottleneck. Partitioning gives scale but only rough ordering. Snowflake picks partitioning, and so will we.
Non-Functional Requirements (NFR)
- Scale: target ~1M IDs/sec system-wide at peak, growing. Design should not fall over at 10x.
- Latency: p99 under 1ms per ID from the caller’s perspective. Ideally the ID is generated in the caller’s process or a same-host sidecar, so it is memory-speed with no network hop.
- Availability: 99.999% for ID generation. This is the dependency of the whole platform - if IDs cannot be minted, writes stop everywhere. It must be more available than anything that depends on it, which means no single point of failure, full stop.
- Uniqueness: absolute. A duplicate ID is a data-corruption incident (two orders collapse into one row). This is non-negotiable and takes priority over ordering.
- Ordering: k-sortable. IDs from the same machine are strictly monotonic; IDs across machines are ordered to within clock-skew tolerance (single-digit milliseconds).
- Durability / statelessness: the generator should hold as little durable state as possible. Ideally a node can crash and restart and still never produce a duplicate, without reading a database on the hot path.
Back-of-the-Envelope Estimation (BoE)
Let me ground the bit budget with real numbers, because the whole design is a bit-packing exercise.
Throughput target:
Peak target ~1,000,000 IDs/sec system-wide
Design headroom 10x -> ~10,000,000 IDs/sec
Why 64 bits: a 64-bit signed integer has 63 usable bits (we keep the sign bit 0 so IDs stay positive and sort correctly as signed longs). We need to divide those 63 bits among: a timestamp, a machine identifier, and a per-machine sequence counter. Let me size each.
Timestamp bits. We want millisecond resolution so IDs are ordered finely, and we want the epoch to last decades.
Milliseconds in a year = 1000 * 60 * 60 * 24 * 365 ≈ 3.15 * 10^10
41 bits = 2^41 ≈ 2.2 * 10^12 milliseconds
2.2 * 10^12 / 3.15 * 10^10 ≈ 69.7 years
So 41 bits of milliseconds buys ~69 years from a custom epoch. Use a recent epoch (say Jan 1 2025) so all 69 years are in the future, not wasted on the 1970-2025 gap.
Machine bits. How many ID-generating nodes must we address at once?
10 bits = 2^10 = 1024 unique machine IDs.
1024 concurrent generators is plenty; you can split it further (e.g. 5 bits data-center, 5 bits worker) if you run multiple regions. If you ever need more, you recycle a machine ID after a node is decommissioned - it is a slot, not a serial number.
Sequence bits. Per machine, per millisecond, how many IDs must we mint?
Leftover bits = 63 - 41 (time) - 10 (machine) = 12 bits
12 bits = 2^12 = 4096 IDs per machine per millisecond
= 4,096,000 IDs/sec per single machine
One machine alone does ~4M IDs/sec. With 1024 machines the theoretical ceiling is ~4 billion IDs/sec, absurdly beyond our 10M/sec design target. The bit budget is comfortable.
The final 64-bit layout:
1 bit 41 bits 10 bits 12 bits
+-----+---------------------+-----------+--------------+
| 0 | timestamp (ms) | machine | sequence |
+-----+---------------------+-----------+--------------+
sign since custom epoch 0..1023 0..4095
Storage / bandwidth: an ID is 8 bytes. Even at 1M/sec, that is 8 MB/sec of raw ID bytes - negligible. There is no storage problem here; the IDs live inside other systems’ rows. The cost is entirely CPU and coordination, not bytes.
Coordination cost: the only thing a Snowflake node needs from the outside world is a unique machine ID once at startup, not per ID. So the coordination service is hit ~once per node boot (thousands of times a day across the fleet), never on the hot path. That is the whole trick.
High-Level Design (HLD)
The clean architecture: ID generation is a library or sidecar co-located with the caller, plus a tiny coordination service that hands out machine IDs at boot. No network hop on the generate path.
┌──────────────────────────────────────┐
│ Coordination Service │
│ (ZooKeeper / etcd - leases machine │
│ IDs 0..1023, NTP-synced clocks) │
└───────────────┬──────────────────────┘
│ machine_id lease
│ (once, at boot)
┌───────────────┬──────────┴──────────┬───────────────┐
│ │ │ │
┌──────▼──────┐ ┌──────▼──────┐ ┌──────▼──────┐ ┌──────▼──────┐
│ App Node 1 │ │ App Node 2 │ ... │ App Node N │ │ ID Sidecar │
│ ┌─────────┐ │ │ ┌─────────┐ │ │ ┌─────────┐ │ │ (for non- │
│ │Snowflake│ │ │ │Snowflake│ │ │ │Snowflake│ │ │ JVM apps) │
│ │ library │ │ │ │ library │ │ │ │ library │ │ └─────────────┘
│ │machine=7│ │ │ │machine=42 │ │ │machine=N│ │
│ └─────────┘ │ │ └─────────┘ │ │ └─────────┘ │
└─────────────┘ └─────────────┘ └─────────────┘
│ │ │
└───── generate ID in-process, memory speed, no RPC ─────┘
Each node also runs NTP (chrony) to keep its wall clock within a few ms.
Request flow (generate one ID):
- Caller invokes
nextId()in-process (or over a localhost socket to a sidecar). - The library reads the current wall-clock time in milliseconds.
- It compares to the last timestamp it saw:
- If the clock is on the same millisecond, increment the 12-bit sequence.
- If the clock advanced, reset the sequence to 0.
- If the sequence overflows (more than 4096 in one ms), busy-wait until the next millisecond.
- If the clock went backwards (skew), refuse to generate until time catches up (the clock-skew handling below).
- It composes the 64-bit integer:
(timestamp - epoch) << 22 | machine_id << 12 | sequence. - Returns the ID. No lock contention across machines, no network call, no database read.
Boot flow (get a machine ID):
- On startup, the node contacts the coordination service (ZooKeeper/etcd).
- It acquires an exclusive lease on a machine ID from the pool
0..1023. - It records the lease (with a TTL/heartbeat) so no other node can take that ID while it is alive.
- It caches the machine ID for its lifetime. If it loses the lease (network partition), it stops generating rather than risk a duplicate.
The genius of Snowflake is that steps 2-4 happen once per process lifetime. The hot path (steps 1-5 of generate) touches nothing but local memory and the local clock.
Component Deep Dive
Three hard parts: (1) picking the generation strategy at all - UUID vs DB ticket vs Snowflake; (2) handing out machine IDs safely without a single point of failure; (3) surviving clock skew, which is the one thing that can silently break Snowflake. I will walk each from naive to correct.
1. Generation Strategy: UUID vs DB Ticket vs Snowflake
This is the core decision. Three families of answers; I will break each.
Approach A: UUIDv4 (the naive instinct)
Just call uuid4(). 122 random bits, generated locally, zero coordination, no bottleneck, no SPOF. Uniqueness is probabilistic but the collision odds are astronomically low.
import uuid
def next_id():
return uuid.uuid4() # e.g. "f47ac10b-58cc-4372-a567-0e02b2c3d479"
Where it breaks:
- 128 bits, not 64. It does not fit in a
BIGINT. It doubles the size of every primary key and every foreign key and every index that references it. On a billion-row table that is real memory and disk. - Not sortable. UUIDv4 is random, so consecutive IDs land at random positions in a B-tree index. This causes index fragmentation and page splits - inserts scatter across the whole index instead of appending to the end. On write-heavy tables this tanks insert throughput and balloons the index. This is the classic reason random UUIDs are a bad primary key.
- No time information. You cannot derive “when was this created” or “give me the newest 100” from the ID itself.
UUIDv4 is the trap: trivially correct on uniqueness, quietly terrible as a database key. (UUIDv7, which prefixes a millisecond timestamp, fixes the sortability and is a legitimate modern option - but it is still 128 bits. If a 128-bit ID is acceptable, UUIDv7 is a fine, coordination-free answer. Our requirement is 64 bits, so we keep going.)
Approach B: Database ticket server / auto-increment (the “just use the DB” idea)
Keep a single table with an auto-increment column. Every time you need an ID, insert a row and take the generated value. One source of truth, perfectly sequential, perfectly sortable, 64-bit.
CREATE TABLE tickets (id BIGINT AUTO_INCREMENT PRIMARY KEY, stub CHAR(1));
REPLACE INTO tickets (stub) VALUES ('a'); -- Flickr's classic trick
SELECT LAST_INSERT_ID();
Where it breaks:
- Single point of failure. One database mints all IDs. If it goes down, every write in the company that needs an ID stops. This violates our most important NFR.
- Throughput ceiling. Every ID is a network round trip plus a write to one row. You are not doing 1M/sec through a single auto-increment. Even with the round trip you are latency-bound, not memory-bound.
- The two-server “fix” and its cost. The classic Flickr answer: run two ticket servers, one handing out even IDs (
auto_increment_increment=2, offset=1) and one odd (offset=2). Now you have HA and double throughput. But it only takes you from 1 to 2 (or N) servers, ordering across them is no longer strict, and you are still doing a DB round trip per ID. It is a real production pattern, just not a 1M/sec-with-local-latency one.
The DB ticket server is honest and simple and used in production, but it fights our latency and availability targets. Its one genuine advantage - strict global ordering - is something we explicitly gave up in the NFRs.
Approach C: Range/segment allocation (batch the DB, fix the round trip)
Compromise: keep a DB counter but hand out ranges, not single IDs. A node asks the DB for a block of 1,000 (or 100,000) IDs, gets [5,000,000 .. 5,000,999], and serves them from local memory. It only returns to the DB when the block runs low (pre-fetching the next block before exhaustion so it never stalls).
DB row: next_range_start = 5,000,000
Node A -> gets [5,000,000 .. 5,000,999] (serves from RAM)
Node B -> gets [5,001,000 .. 5,001,999]
The DB is hit once per 1,000 IDs, so a single modest database can back a very high aggregate rate, and the hot path is memory-speed. This is exactly how systems like Twitter’s earlier scheme and many ORMs (Hibernate’s hilo) work.
Where it breaks / its limit: the IDs are only roughly sorted - node A is handing out 5,000,xxx while node B hands out 5,001,xxx, so ordering across nodes is chunky, not time-based. And you still depend on the DB being reachable to refill (mitigated by pre-fetching several blocks). It is a great answer when you want dense, compact, mostly-monotonic 64-bit IDs and can tolerate the DB dependency. But the ordering is by allocation, not by wall-clock time, which matters if different nodes allocate at very different rates.
Approach D: Snowflake (the answer)
Compose the 64-bit ID from timestamp | machine_id | sequence as laid out in the BoE. Each node generates locally at memory speed, uniqueness comes from the structure (no two nodes share a machine ID; one node never repeats a timestamp+sequence), and time-ordering comes for free because the high bits are the timestamp.
class Snowflake:
EPOCH = 1735689600000 # 2025-01-01 in ms
def __init__(self, machine_id):
self.machine_id = machine_id & 0x3FF # 10 bits
self.seq = 0
self.last_ts = -1
def next_id(self):
ts = self._now()
if ts < self.last_ts:
raise ClockMovedBackwards(self.last_ts - ts) # handled below
if ts == self.last_ts:
self.seq = (self.seq + 1) & 0xFFF # 12 bits
if self.seq == 0: # overflowed this ms
ts = self._wait_next_ms(self.last_ts)
else:
self.seq = 0
self.last_ts = ts
return ((ts - self.EPOCH) << 22) | (self.machine_id << 12) | self.seq
Why this wins for our requirements:
- 64-bit, sortable, no hot-path coordination. All three requirements at once.
- No SPOF on the generate path. Every node is independent. The coordination service is only touched at boot.
- Uniqueness by construction, not by probability. Unlike UUIDv4, there is no collision math to worry about, as long as machine IDs are unique and clocks do not go backwards.
The two things that can break it are the two remaining deep dives: duplicate machine IDs and clocks moving backwards.
| Strategy | Bits | Sortable | Hot-path coordination | SPOF | Verdict |
|---|---|---|---|---|---|
| UUIDv4 | 128 | No | None | None | Simple but bad DB key |
| UUIDv7 | 128 | Yes | None | None | Great if 128 bits ok |
| DB auto-increment | 64 | Strict | RPC per ID | Yes | Bottleneck + SPOF |
| Range allocation | 64 | Roughly | RPC per block | Soft (DB refill) | Good, DB-dependent |
| Snowflake | 64 | Roughly | None (boot only) | None | The answer |
2. Machine ID Assignment Without a Single Point of Failure
Snowflake’s uniqueness rests entirely on: no two live nodes ever hold the same machine ID. How do you guarantee that across 1024 slots and a fleet that autoscales, restarts, and occasionally splits from the network?
Naive: hard-code the machine ID in config
Set machine_id=7 in each node’s config file. Works until two nodes accidentally get the same config (copy-paste, bad templating, a cloned VM image), and now both mint IDs with machine 7 - silent duplicate IDs. In an autoscaling world where nodes come and go, static config is a landmine. It also wastes slots on nodes that are down.
Better: lease machine IDs from ZooKeeper/etcd
On boot, a node asks a consensus store (ZooKeeper, etcd) for a machine ID. The store hands out an unused slot from 0..1023 and records it as an ephemeral, TTL-backed lease tied to the node’s session.
Node boots -> connect to ZooKeeper
-> create ephemeral znode /snowflake/workers/<next-free-id>
-> that id is now leased to this node's session
Node runs -> heartbeats keep the session (and lease) alive
Node dies -> session expires -> ephemeral znode vanishes -> id returns to pool
Why this removes the SPOF and stays safe:
- Consensus store is itself a quorum (3 or 5 nodes). It has no single point of failure and it is the standard place to keep this kind of small, strongly-consistent coordination state.
- Ephemeral leases self-heal. If a node crashes, its session times out and the slot is automatically freed for reuse. No manual cleanup, no leaked slots.
- Boot-only dependency. The consensus store is on the boot path, not the generate path. If ZooKeeper is briefly unavailable, already-running nodes keep minting IDs from memory; only new node boots are delayed. That is exactly the right failure mode - existing capacity is unaffected.
The partition trap and how to handle it
The dangerous case: a running node gets partitioned from ZooKeeper. Its session eventually expires, ZooKeeper frees its slot, and a new node might grab that same machine ID - while the partitioned node, still running, thinks it owns it. Now two live nodes share a slot. Duplicate IDs.
The fix is a rule on the node side: if a node loses its lease/session, it must stop generating IDs immediately (fail closed), and only resume after re-acquiring a lease. We trade a brief local outage on that one node for the guarantee that we never emit a duplicate. Uniqueness beats availability for a single node; the fleet as a whole stays up because other nodes are unaffected. This is the correct priority given our NFRs.
For extra safety at very large scale, split the 10 machine bits into datacenter_id (5) + worker_id (5), and lease within each datacenter’s own ZooKeeper ensemble. This localizes coordination and avoids one global store being on every node’s boot path across regions.
3. Clock Skew and Clocks Moving Backwards
This is the subtle killer. Snowflake assumes the wall clock is monotonic - each next_id() sees a timestamp greater than or equal to the last. Real clocks are not monotonic: NTP corrections, leap seconds, and VM live-migration can push the wall clock backwards by milliseconds or seconds. If the clock jumps back, a node can re-enter a timestamp range it already used and, with the same sequence values, produce IDs it has already emitted. Duplicates again.
Naive: ignore it
Assume clocks only go forward. In a lab, fine. In production with NTP stepping the clock and hypervisors pausing VMs, you will see backward jumps, and the failure is silent until two rows collide on a unique key days later. Never ignore it.
Handling strategy, in order of jump size
- Small backward jump (a few ms): wait it out. If
now < last_tsby a small amount, the node refuses to generate and busy-waits until the clock catches back up tolast_ts. A few milliseconds of added latency on a rare event is acceptable, and it guarantees we never reuse a timestamp.
def _now_monotonic(self):
ts = self._now()
if ts < self.last_ts:
drift = self.last_ts - ts
if drift <= MAX_WAIT_MS: # e.g. <= 5ms: wait
while ts < self.last_ts:
ts = self._now()
else:
raise ClockMovedBackwards(drift) # too big to wait: fail closed
return ts
Large backward jump (seconds): fail closed and alarm. If the clock jumped back by more than a small threshold, waiting would stall the node for too long. Instead, stop generating, mark the node unhealthy, page an operator, and let the load balancer route around it. Better a dead node than a duplicate ID. The node rejoins once its clock is sane.
Prefer a monotonic clock source where possible. Read time from a source that cannot go backwards (e.g.
CLOCK_MONOTONICanchored to a wall-clock reference captured at boot, or a hybrid). Some Snowflake variants track the last-used timestamp in memory and simply never let the emitted timestamp regress, decoupling the ID timestamp from raw NTP steps. The emitted “time” becomes max(last_ts, wall_clock), so it is monotonic by construction even if the underlying clock stutters.
Keep clocks tight in the first place
- Run NTP/chrony on every node, synced to a reliable set of servers, so drift stays within single-digit milliseconds. Good sync is what makes cross-machine ordering meaningful (rough sort to within skew).
- Configure NTP to slew, not step, under normal drift - it speeds up or slows down the clock gradually rather than jumping it, which avoids backward jumps entirely for small corrections.
- Handle leap seconds with clock smearing (spread the extra second over hours) so there is never a hard backward step. Major cloud providers already smear; match their approach.
The mental model: NTP + slewing keeps normal operation skew-free; the wait-it-out logic covers small residual jumps; fail-closed covers the catastrophic jump. Uniqueness is preserved in all three cases.
4. Sequence Overflow Within a Millisecond
Minor but worth stating. If a single node is asked for more than 4096 IDs in one millisecond, the 12-bit sequence overflows. The handling: spin-wait to the next millisecond and reset the sequence to 0.
def _wait_next_ms(self, last_ts):
ts = self._now()
while ts <= last_ts:
ts = self._now()
return ts
At 4096 IDs/ms that node is already doing ~4M IDs/sec; if you consistently hit the ceiling, you either add nodes (each has its own sequence space) or steal a bit from the machine field to widen the sequence. Almost no real workload needs 4M IDs/sec from one process, so this stays a theoretical guard in practice.
API Design & Data Schema
API
The generator is primarily a library embedded in each service (memory-speed, no RPC). For non-JVM or polyglot fleets that cannot embed it, expose a thin sidecar / service over the local host.
# Library (in-process) - the preferred path
long id = snowflake.nextId();
long[] ids = snowflake.nextIds(count); // batch, still local
# Sidecar HTTP (for polyglot apps, localhost only)
GET /id
Response 200:
{ "id": 195783472891234304 }
GET /id/batch?count=100
Response 200:
{ "ids": [ 195783472891234304, 195783472891234305, ... ] }
Errors:
503 node unhealthy (lost lease, or clock moved backwards) - do NOT generate
# Introspection - decode an ID back into its parts (handy for debugging)
GET /decode/195783472891234304
Response 200:
{
"timestamp_ms": 1782453900000,
"datetime": "2026-07-10T04:05:00Z",
"machine_id": 42,
"sequence": 0
}
The 503 while unhealthy behavior is the API-level expression of “fail closed on lease loss or clock skew.” A caller getting a 503 retries against another node/sidecar; it must never get a possibly-duplicate ID.
Data / coordination schema
There is almost no persistent data - that is the point. The only durable state is the coordination store’s machine-ID leases.
Machine ID lease (in ZooKeeper/etcd - ephemeral, strongly consistent):
/snowflake/workers/
<machine_id> -> {
session_id: "0x1a2b3c...", -- owning session; ephemeral
host: "app-node-42",
acquired_at: 1782453900000,
ttl_ms: 10000 -- heartbeat renews; expiry frees the slot
}
- Store choice: consensus KV (ZooKeeper/etcd), not SQL. We need strong consistency, ephemeral ownership tied to sessions, and quorum availability for a tiny amount of data. That is exactly what ZooKeeper/etcd are for, and exactly what a relational DB is bad at (no ephemeral ownership, single-writer). SQL would reintroduce the SPOF we are trying to kill.
- No table for the IDs themselves. IDs are computed, never stored by the generator. They live inside whatever rows the callers write. There is no “IDs” table, no read-before-write, no lookup.
Optional per-node persisted watermark (local disk, defensive):
last_emitted_ts.dat -> 1782453900000 -- highest timestamp this node has used
On restart, a node reads this watermark and refuses to emit any timestamp less than or equal to it, protecting against a clock that reset while the node was down. Cheap insurance against boot-time clock regressions.
Bottlenecks & Scaling
Where it breaks first, in order, and the fix for each.
1. Clock skew (breaks first, and silently). A backward clock jump is the single most likely cause of a real duplicate.
Fix: NTP/chrony everywhere with slewing (not stepping), leap-second smearing, wait-it-out for small backward jumps, fail-closed + alarm for large ones, and a monotonic emitted-timestamp (max(last_ts, now)) so the ID clock never regresses even when the OS clock does. This is the one you must get right.
2. Duplicate machine IDs. Two live nodes sharing a slot silently corrupts. Fix: lease machine IDs from a ZooKeeper/etcd quorum with ephemeral, TTL-backed leases; nodes fail closed on lease loss; split bits into datacenter + worker at multi-region scale. Never hard-code machine IDs.
3. Single point of failure on the generate path. There must not be one. Fix: generation is fully local (library/sidecar). No node depends on any other node or on the DB to mint an ID. The coordination store is only on the boot path, so its blips slow new boots, not live generation. The system as a whole degrades node-by-node, never all at once.
4. Sequence exhaustion on a single node (>4096 IDs/ms). Fix: spin-wait to the next millisecond; if sustained, add nodes (independent sequence spaces) or rebalance bits toward the sequence field. In practice no single process needs 4M IDs/sec.
5. Running out of machine IDs (>1024 live nodes). Fix: the 10-bit field is a slot pool, not a serial-number space - decommissioned nodes free their slot via lease expiry, so you only need 1024 concurrent nodes. If you genuinely need more, widen the machine field by stealing bits from the sequence (you rarely use all 12), or run independent Snowflake pools per region with a region prefix.
6. Epoch exhaustion (~69 years out). Fix: it is decades away; pick a modern epoch (2025) so you get the full 69 years. When it approaches, you migrate to a wider timestamp or a new epoch with a version bit - a problem for the 2090s, flagged now so nobody is surprised.
7. Cross-machine ordering is only approximate. Two IDs from different machines within a few ms are not guaranteed to sort by true creation time. Fix: accept it - we declared k-sortability, not total order, in the NFRs. Within a single machine, IDs are strictly monotonic. If a specific feature needs strict global order, that feature carries its own sequence; do not push that cost onto every ID.
8. Enumerable / predictable IDs. Sequential-ish IDs leak volume and are scrapeable. Fix: if the ID is exposed publicly, do not expose the raw Snowflake - map it through a bijective transform (Feistel/opaque encoding) at the API boundary. The internal ID stays sortable and unique; the external one is opaque. This is a presentation concern, kept off the generator.
Wrap-Up
The trade-offs that define this design:
- Snowflake over UUID and over a DB ticket server. We take a 64-bit, time-sortable, coordination-free ID. Versus UUIDv4 we win on size and sortability (no index fragmentation); versus a DB ticket server we win on latency and availability (no per-ID round trip, no SPOF). The price is that ordering is only roughly global, which we explicitly accepted.
- Partition the ID space instead of coordinating per ID. Uniqueness comes from structure - unique machine IDs plus per-machine monotonic timestamps and sequences - not from a central authority on the hot path. That is what removes the single point of failure.
- Uniqueness beats availability for a single node. On lease loss or a large clock jump, a node fails closed. We would rather a node stop than emit a duplicate, because a duplicate ID is silent data corruption discovered days later.
- Push coordination to boot time. The consensus store hands out machine IDs once per process, never per ID, so its availability requirement is decoupled from the generate path.
- Treat clock skew as a first-class failure mode. NTP slewing, leap-second smearing, wait-it-out, fail-closed, and a monotonic emitted clock together make the one silent failure loud and safe.
One-line summary: a 64-bit Snowflake ID - timestamp | machine_id | sequence - generated locally at memory speed with machine IDs leased once at boot from a ZooKeeper/etcd quorum, giving unique, time-sortable IDs at millions per second with no single point of failure, where the only real enemy is a backward-moving clock, handled by NTP discipline plus fail-closed logic.
Comments