A time-series database looks like the easiest database to build and is one of the hardest to build well. The data is trivial: a name, some tags, a timestamp, a number. No joins, no foreign keys, no complex transactions. So the interviewer says “store metrics and let me query them” and you reach for a table (series, ts, value) with an index, and it works beautifully for a week of one host. Then the real shape shows up: millions of writes per second that never stop, timestamps that march forward almost perfectly evenly, values that barely change between samples, queries that almost never fetch a single point and almost always aggregate a range (“average CPU over the last 6 hours, per host”), data that must live for months without bankrupting you, and a requirement that a range-aggregate over that history return in milliseconds. The naive table dies on every one of those axes at once.

This post is specifically about the storage engine - the database, not the monitoring product on top of it. I am not designing dashboards or an alerting pager here (that is a different system that uses a TSDB). I am designing the thing that has to swallow a write firehose, store it cheaply for a long time, and answer time-range aggregations fast. The whole design comes down to four decisions: how you absorb millions of ordered writes per second without a B-tree melting (a write-optimized, append-first engine), how you make the bytes tiny (compression that exploits time-ordering), how you keep old data from crushing storage and queries (continuous downsampling plus tiered retention), and how you answer a range-aggregate in milliseconds (time-partitioned, columnar blocks with pushed-down aggregation). Everything else is detail.

Let me build it properly.

Functional Requirements (FR)

In scope:

  • Write time-series points. Accept a stream of points, each (measurement, {tags}, timestamp, field_value) - for example cpu_usage{host="web-01", region="blr", core="3"} @ 1763251200000 = 74.2. A series is a unique (measurement + tag-set); a series is an ordered sequence of (timestamp, value) points. Writes are overwhelmingly append at the head (timestamps increasing), with a tolerance for slightly late and out-of-order arrivals.
  • Query raw points over a time range. “Give me the raw values of this series between t1 and t2.” The base case.
  • Range-aggregate queries. The dominant query: aggregate a time range into buckets - SELECT mean(value) ... WHERE host='web-01' AND time > now()-6h GROUP BY time(1m). Support sum, count, mean, min, max, last, percentiles, and rate/derivative, grouped by time bucket and optionally by tag.
  • Tag-based filtering. Resolve {region="blr", env="prod"} to the matching series quickly, so a query can target a subset of millions of series without scanning all of them.
  • Continuous downsampling. Automatically roll raw high-resolution data (say 10s) into coarser resolutions (1m, 1h) so long-range queries read few points.
  • Retention and tiering. Expire data by policy (raw for days, rollups for months), and move cold data from hot SSD to cheap object storage, transparently to queries.

Explicitly out of scope (own the boundary):

  • Dashboards, alerting, and the UI. Those are products built on top of the query API. We expose the API; we do not render charts or page anyone. The monitoring system is a client of this database.
  • Full-text logs and traces. This stores numeric points, not searchable text or span trees. Different access pattern (aggregation, not search). We note where exemplars would attach and move on.
  • Arbitrary OLTP / updates / deletes of individual points. A TSDB is append-mostly and immutable-after-flush. We support range deletes (retention) and overwrite-last-write-wins for a duplicate (series, ts), not row-level UPDATE.
  • Cross-series JOINs and relational modeling. You filter and aggregate series; you do not join them like relational tables.
  • The collection agent. How a host exposes cpu_usage (an agent, a /metrics endpoint, a client SDK) is assumed. We design from “points arriving at the write API” onward.

The single decision that drives everything: a TSDB is a write-heavy, append-at-the-head, aggregate-over-ranges engine where the bytes are time-ordered and nearly-constant, so it must be built as a write-optimized log-structured store with time-ordering-aware compression, continuous downsampling, and tiered retention - not as a row store with an index. Every other choice follows from that.

Non-Functional Requirements (NFR)

  • Scale: on the order of 5M active series, each written every ~10s, giving ~500K points/sec sustained and designing for ~2M points/sec peak (bursts when fleets autoscale or agents flush buffered backlog). Reads are far fewer in count but each one is a fan-out aggregation over many series and many points.
  • Latency: writes are fire-and-forget and must ack in single-digit milliseconds without blocking the writer. Range-aggregate query p99 under ~200ms for common shapes (served mostly from rollups and cache), and under ~1s for a heavy cold query touching object storage. A point-read of one series’s recent data is sub-10ms.
  • Availability: the write path targets 99.9%+ - a TSDB that drops writes is blind. The read path can degrade to slightly-stale rather than error. Writes must survive a node crash without loss of acknowledged data.
  • Consistency: eventually consistent, read-your-recent-writes within seconds. A point acknowledged is durable, but it may be a second or two before it is visible to queries (it sits in the in-memory head first). Monitoring-style data tolerates a dropped or late point among billions - we explicitly do not need exactly-once or strict linearizability, and we spend that relaxation on throughput and latency.
  • Durability: acknowledged recent data must survive single-node failure (WAL + replication). Old downsampled data is important in aggregate but losing one point among trillions is a non-event; it lives on durable object storage with its own replication.
  • Cost is a first-class NFR. Storing months of millions-of-points-per-second is a storage-cost problem before it is anything else. Compression ratio and the hot/cold tiering split are not optimizations - they decide whether the system is affordable at all.

Back-of-the-Envelope Estimation (BoE)

Grounding the design in numbers before drawing boxes.

Write traffic:

Active series:                 5,000,000          (5M)
Sample interval:               10 seconds
Points/sec = 5M / 10s       =  500,000 points/sec   (sustained)
Peak (autoscale + backlog)  ≈  2,000,000 points/sec  (size for this)

Writes are the firehose. Critically they are append-at-the-head: almost every incoming point has a timestamp slightly larger than the last one for its series. That ordering is the single property the whole storage engine exploits.

Read traffic:

Range-aggregate queries:   say ~5,000 queries/sec at peak (many clients,
                           dashboards polling, ad-hoc queries).
Each query: resolves tag matchers -> a set of series (1 to ~10,000),
            then aggregates a time window -> output buckets.

So it is not a pure “writes crush reads” shape. Writes dominate raw volume (500K points/sec vs 5K queries/sec), but each read fans out across many series and many points, so read work is real and the query path needs engineering too. The shared enemy of both paths is the number of points touched, which is why compression and downsampling pay off on both sides.

Storage - the whole ballgame:

Uncompressed point:   timestamp 8B + value 8B = 16 bytes.
Compressed point:     ~1.3 bytes  (delta-of-delta timestamps +
                      XOR float values - the Gorilla encoding).
                      ~12x compression. This decides affordability.

Raw ingest, compressed:
   500K pts/sec * 1.3 B  ≈ 0.65 MB/sec ≈ 56 GB/day ≈ 1.7 TB/month raw.
   Uncompressed that is ~20 TB/month - the difference between a
   cheap cluster and an impossible one.

Downsampling payoff:

Resolution   Interval  Retention   Points/series/day   Rel. size
raw          10s       7 days      8,640               1x
1m rollup    1m        30 days     1,440               1/6
1h rollup    1h        13 months   24                  1/360

A "last 90 days, per hour" query over 1,000 series:
   from 1h rollup:  1,000 * (90*24)      = 2,160,000 points read
   from raw 10s:    1,000 * (90*8,640)   = 777,600,000 points read
   -> 360x less work for the identical answer.

Storage total with tiering:

Raw (7d, hot SSD):      ~0.65 MB/s * 86400 * 7   ≈ 393 GB   (replicated 3x ≈ 1.2 TB SSD)
1m rollup (30d):        raw/6 at 30d             ≈ 280 GB
1h rollup (13mo):       raw/360 over 13 months   ≈ 20 GB
-> hot tier is small; months of history live cheaply as rollups,
   with cold raw/rollup blocks tiered to object storage at ~1/5 the $/GB of SSD.

Bandwidth:

Write:  500K pts/sec batched+compressed on the wire ≈ ~15-25 MB/sec inbound
        spread across the ingest fleet. Modest.
Read:   responses are downsampled series, KBs each -> a few MB/sec. A non-issue.
Internal: compaction/downsampling reads and rewrites blocks - the biggest
          internal I/O consumer, but it is background and batched.

Two conclusions drive the design. First, compression is not optional - at 500K points/sec, 16 bytes vs 1.3 bytes per point is a 12x cost difference, so the engine’s on-disk format is time-ordering-aware compression, period. Second, the hot working set is tiny - a week of raw is a few hundred GB, and everything older is coarse rollups on cheap storage - so we size RAM/SSD for the head and recent blocks, and push the long tail to object storage.

High-Level Design (HLD)

The database is a write-optimized, log-structured storage engine fronted by a stateless ingest tier and a query engine, with a background compactor that compresses, downsamples, and tiers data. Writers are sharded by series so each point lands deterministically and stays ordered.

          WRITE PATH (points, ~0.5-2M/sec, append-at-head, fire-and-forget)
  ┌──────────────────────────────────────────────────────────────────────────┐
  │  Collection agents / clients  --batched, compressed-->  Write API (LB)     │
  └───────────────────────────────────────┬────────────────────────────────────┘
                                           │
                                 ┌─────────▼──────────┐
                                 │   Ingest Service   │ (stateless fleet)
                                 │  - validate/parse  │
                                 │  - compute series_id = hash(name+tags) │
                                 │  - route by shard  │
                                 └─────────┬──────────┘
                                           │ partition by hash(series_id)
                            ┌──────────────▼───────────────┐
                            │   Durable log (Kafka/WAL)      │ absorbs bursts,
                            │   partitioned by series shard  │ decouples storage
                            └───┬───────────────────────┬────┘
                                │                       │
                    ┌───────────▼──────────┐   ┌────────▼───────────────┐
                    │  Storage Node (shard) │   │  Storage Node (shard)  │ ...N
                    │  ┌─────────────────┐  │   │   each OWNS a slice of │
                    │  │ WAL (seq append)│  │   │   the series keyspace  │
                    │  ├─────────────────┤  │   └────────┬───────────────┘
                    │  │ Head block (RAM)│  │            │
                    │  │  active ~2h,    │  │            │
                    │  │  per-series     │  │            │
                    │  └────────┬────────┘  │            │
                    │   flush   │ compress  │            │
                    │  ┌────────▼────────┐  │            │
                    │  │ Immutable blocks│  │            │
                    │  │ (SSD, time-part)│  │            │
                    │  └────────┬────────┘  │            │
                    └───────────┼───────────┘            │
                                │                        │
                    ┌───────────▼────────────────────────▼───────────┐
                    │  Compactor (background, per shard)              │
                    │  - merge small blocks -> big blocks             │
                    │  - DOWNSAMPLE raw -> 1m -> 1h rollup blocks      │
                    │  - apply RETENTION + TIER cold blocks to object │
                    └───────────┬─────────────────────────────┬──────┘
                    ┌───────────▼──────────┐       ┌───────────▼─────────┐
                    │  SSD (hot: head +    │       │  Object store (cold:│
                    │  recent raw blocks)  │       │  old raw + rollups) │
                    └──────────────────────┘       └─────────────────────┘
                                   READ PATH
  ┌──────────────────────────────────────────────────────────────────────────┐
  │  Query API  ->  Query Engine                                               │
  │   1. parse + plan (range, GROUP BY time, tag matchers)                     │
  │   2. Series Index: tag matchers -> series_ids (inverted index)            │
  │   3. pick coarsest resolution that satisfies step (raw|1m|1h)             │
  │   4. fan out to storage shards owning those series                        │
  │   5. shards push down per-block aggregation, return partial buckets       │
  │   6. merge partials, apply final aggregation, result cache                │
  └──────────────────────────────────────────────────────────────────────────┘

Write hot path - a point being stored:

  1. An agent batches points for many series and POSTs them compressed to the Write API behind a load balancer. Batching is essential: one HTTP request carries thousands of points, amortizing the request cost.
  2. The stateless Ingest Service parses and validates each point, computes its series_id = hash(measurement + sorted(tags)), and routes it by hash(series_id) mod N to the storage shard that owns that series. Routing by series guarantees every point of a series lands on the same node, in order.
  3. The point is appended to a durable log (Kafka or a replicated WAL) partitioned by shard, then acked. Ack after the durable append, not after storage - the write is now safe even if the storage node is momentarily behind. This decouples ingest latency from storage work and absorbs bursts.
  4. The owning Storage Node consumes its partition, writes each point to its WAL (sequential append - durability without random I/O), and appends it to the in-memory head block for that series. The head covers the current time window (say ~2 hours). The point is now queryable.
  5. When the head window closes, the node compresses the per-series columns (delta-of-delta timestamps + XOR values) and flushes them to an immutable on-disk block on SSD, carrying a per-block index of which series it holds. The WAL segment for that window can then be truncated.

Background path - the compactor:

  1. A per-shard Compactor continuously (a) merges many small flushed blocks into fewer larger ones (fewer files to open per query), (b) downsamples raw blocks into 1m and then 1h rollup blocks storing sufficient statistics per bucket, and (c) enforces retention by dropping whole expired blocks and tiering cold blocks (old raw, old rollups) to object storage. Because blocks are immutable, every one of these is a safe, idempotent, whole-file operation.

Read path - a range-aggregate query:

  1. The Query Engine parses mean(value) WHERE host='web-01' AND time>now()-6h GROUP BY time(1m), resolves the tag matchers to series_ids via the inverted series index, picks the coarsest resolution whose interval <= the requested step (6h / 1m step -> raw is fine; 90d / 1h step -> the 1h rollup), fans out to the shards owning those series, and asks each to aggregate.
  2. Each shard, for each relevant block, pushes the aggregation down - it scans only the blocks overlapping the time range, decompresses the needed series columns, and computes partial per-bucket aggregates (count/sum/min/max) without shipping raw points across the network. The engine merges the partials into final buckets, applies the final function (mean = sum/count), and returns. Repeated dashboard-style queries hit a result cache.

The architectural core: writes are cheap because they are sequential appends to a log and an in-memory head, never random updates to a B-tree; storage is cheap because immutable time-partitioned blocks compress time-ordered data ~12x and old data is downsampled and tiered; and reads are fast because time-partitioning lets a query ignore irrelevant time, the rollup hierarchy lets it read 360x fewer points, and aggregation is pushed down to the blocks. The TSDB is a specialized log-structured engine, and every design choice is downstream of “append-at-head, aggregate-over-ranges.”

Component Deep Dive

The four hard parts, each walked from the naive approach to where it breaks to the real design: (1) the write-optimized storage engine, (2) compression of time-ordered data, (3) continuous downsampling and retention tiering, and (4) the range-aggregate query engine with tag indexing.

1. The Write Path: B-Tree Row Store -> Log-Structured Columnar Engine

We take ~500K-2M ordered writes/sec and later read them as range aggregations. The storage engine is the whole game.

Approach A: Rows in a B-tree-indexed relational table (the naive instinct)

metrics(series_id, ts, value) with a B-tree index on (series_id, ts), one row per point.

INSERT INTO metrics (series_id, ts, value) VALUES (...);  -- per point
SELECT avg(value) FROM metrics
 WHERE series_id IN (...) AND ts BETWEEN ? AND ?;

Where it breaks:

  • Write amplification on the index. A B-tree keyed on (series_id, ts) must find and update the right leaf page for every insert. With 5M interleaved series arriving together, consecutive inserts hit scattered leaf pages - random I/O, page splits, and constant rebalancing. At 500K inserts/sec the index thrashes; the disk is doing random writes when the workload is logically sequential. B-trees are read-optimized (update-in-place); this workload is write-dominated.
  • Per-row overhead destroys the storage budget. Each point becomes a full tuple: row header, series_id repeated (8B), ts (8B), value (8B), plus index entries - call it 40-60 bytes/point on disk versus the ~1.3 bytes a TSDB achieves. That is a 30-40x storage blowup, turning our 1.7 TB/month into 50+ TB/month. You go bankrupt on storage before you hit a query problem.
  • The read shape is wrong too. “mean over a 5-minute window for 1,000 series” scatter-gathers across the index and reads full rows to pull one float column out of each. The engine reads 8x the bytes it needs and chases pointers across the tree.

A row store is built for random point reads and updates. This workload is sequential appends and columnar range scans. Wrong tool.

Approach B: A log-structured, time-partitioned, columnar engine (the TSDB)

Build the engine for the exact access pattern. Four ideas make it work, and they are the LSM idea specialized for time:

  • Append to an in-memory head, not an on-disk tree. Incoming points go to a per-series head block in RAM - just appending to the end of each series’s column buffers. No tree, no random I/O, O(1) amortized per point. This is the memtable of an LSM, but because points arrive time-ordered, the head is naturally sorted by timestamp per series - no sort needed on flush.
  • WAL for durability of the head. The head is volatile, so every point is first appended to a write-ahead log on disk - a pure sequential write. A crash replays the WAL to rebuild the head. We get durability without paying random-write cost: both the WAL and the head are append-only.
  • Flush to immutable, time-partitioned blocks. When the head’s time window (~2h) closes, its data is compressed and written once to an immutable block named by its [start, end] time range. Immutability is the gift that keeps giving: compaction, downsampling, deletion, replication, and caching all become trivial because a block never changes. A query for a range just picks the blocks whose range overlaps and ignores everything else - time-partitioning replaces a time index.
  • Columnar per-series layout inside a block. Within a block, a series’s timestamps and values are stored as two separate compressed columns, not interleaved rows. A range-mean reads only the value column (and timestamps to find bucket boundaries), decompresses it in a tight loop, and never touches other series. Columnar + per-series locality is what makes the scan fast and the compression possible.
Storage node, for one shard (a slice of the series keyspace):

   incoming point --> [WAL append] --> [Head block, RAM]
                                         series A: ts[...] val[...]
                                         series B: ts[...] val[...]
         (window closes every ~2h)         |
                                           v  compress + flush
   Immutable blocks on SSD (time-partitioned, columnar, compressed):
      block[08:00-10:00]  block[10:00-12:00]  block[12:00-14:00] ...
      each: per-series compressed columns + a mini series index

Storage nodes are sharded by hash(series_id): each node owns a slice of the series keyspace, holds the heads for its series, and writes its own blocks. Nodes are stateful but fully recoverable - WAL replay rebuilds the head, and the durable ingest log lets a replacement node replay from the last committed offset. This is the shape of every serious TSDB (Prometheus TSDB, InfluxDB TSM/IOx, VictoriaMetrics, M3DB): compress per series, partition by time, append-never-update, flush to immutable blocks, compact in the background.

2. Compression: Making Time-Ordered Data Tiny

At 500K points/sec, the on-disk byte count per point decides whether the system is affordable. Raw is 16 bytes; we need ~1.3. The trick is that time-series data is the most compressible data there is, because of two properties: timestamps are nearly evenly spaced, and consecutive values barely change.

Approach A: Store raw 8-byte timestamp + 8-byte float, maybe gzip the block (naive)

Write each (ts, value) as 16 bytes and run a general-purpose compressor (gzip/zstd) over the block.

Where it breaks: general-purpose compression treats the bytes as an opaque stream and finds some redundancy, but it has no idea that the timestamp column is an arithmetic sequence or that floats differ in only a few low bits. You get maybe 2-3x, not 12x, and you pay CPU to decompress the entire block even when a query wants a 5-minute slice of a 2-hour block. It also does not exploit per-column structure because rows interleave timestamps and values, defeating the pattern. You leave 4-5x compression on the floor, which at this scale is millions of dollars of storage.

Approach B: Column-specific, time-aware encoding (the Gorilla scheme)

Encode the two columns differently, each exploiting its structure. This is Facebook’s Gorilla encoding, the de facto standard.

  • Timestamps: delta-of-delta. Samples arrive every ~10s, so the delta between consecutive timestamps is ~10,000ms almost every time. Store the delta of the delta: 0 when the interval is steady, which packs into a single bit. Only a jitter or a gap costs more bits (a short variable-length code). A steadily-sampled series stores its entire timestamp column as a run of single bits.
timestamps:     1763251200000, +10000, +10000, +10000, +10003, +10000
deltas:                        10000,  10000,  10000,  10003,  10000
delta-of-delta:                   -,      0,      0,     +3,     -3
encoding:                        hdr,   1 bit,  1 bit, few bits, few bits
  • Values: XOR against the previous value. Consecutive gauge values (CPU 74.2 -> 74.3) or slowly-growing counters have almost identical IEEE-754 bit patterns. value XOR prev_value is therefore mostly zero; you store only the meaningful middle run of bits plus small control bits saying how many leading/trailing zeros there were. A flat or slowly-drifting series costs a bit or two per value.
values:   74.2        74.3        74.3        74.1
XOR prev: -           0x..small   0x0 (equal) 0x..small
store:    full first  few bits    1 bit       few bits
  • Result: ~1.3 bytes per point steady-state, ~12x over raw, and decompression is a tight bit-unpacking loop - far cheaper CPU than a general compressor over the whole block. Because columns are separate, a query reading only values skips the timestamp column’s decode when it can derive bucket boundaries from the block’s regular cadence.
  • Block framing for partial reads. A block stores series columns in independently-decodable chunks with an offset index, so a query for a sub-range decompresses only the chunks it overlaps, not the whole 2-hour block. This matters for the millisecond query target: you decode kilobytes, not megabytes.

The mindset: do not compress bytes, compress the pattern. Time-series data has a known shape - evenly-spaced increasing timestamps and slowly-changing values - and column-specific delta-of-delta and XOR encodings exploit exactly that shape to hit a ratio general compressors cannot touch. Compression here is not a nice-to-have; it is the feature that makes storing months of data possible.

3. Downsampling and Retention Tiering: Keeping Old Data Cheap

Raw 10s data for 13 months is both unstorable and unqueryable, and nobody looks at last quarter at 10-second detail. We trade resolution for size on old data.

Approach A: Keep everything at full resolution forever, delete by a TTL cron (naive)

Store every raw point and run a nightly job that deletes rows older than the retention.

Where it breaks on two axes:

  • Query cost scales with the full-resolution range. “Last 90 days, mean per hour” over 1,000 series must scan 1,000 * 90 * 8,640 ≈ 778M raw points to produce ~2,160 output values - throwing away 99.9997% of the work it did. You cannot serve that in milliseconds; you cannot serve it in seconds.
  • Row-by-row deletion is a nightmare in a log-structured store. Blocks are immutable; you cannot DELETE WHERE ts < x cheaply. A per-row TTL forces rewriting blocks constantly, and a B-tree version suffers delete-tombstone bloat and vacuum storms. Deletion must be whole-block, aligned to time partitions.

Full resolution forever is a storage disaster, a query disaster, and a deletion disaster.

Approach B: A rollup hierarchy with sufficient statistics, plus whole-block retention and tiering

The compactor continuously downsamples and the query engine reads the coarsest resolution that answers the question.

Resolution   Interval   Retention    Storage tier
raw          10s        7 days       hot SSD (recent) -> object (older)
1m rollup    1m         30 days      SSD + object
1h rollup    1h         13 months    object store (tiny)
  • Store sufficient statistics per bucket, not just the mean. A downsampled 1m bucket stores count, sum, min, max (and last), and for distributions, the histogram bucket counts. This is non-negotiable: if you stored only the mean, you could never later compute a correct sum, max, or a wider-window mean. With count and sum you can roll 1m into 1h correctly (sum the sums, sum the counts, min of mins, max of maxes) and derive mean at any level. Downsampling is thus composable and associative - 1h is built from 1m, which is built from raw, each exactly once.
  • Percentiles need histograms, not stored percentiles. You cannot average or re-aggregate a p99 - percentiles are not additive. So for latency-type fields you store per-bucket histogram bucket counts (le=10ms:900, le=50ms:990, ...) and downsample by summing bucket counts; the quantile is computed at query time from the summed histogram. This is why a histogram is a first-class field type.
  • Downsampling is block-scoped and idempotent. Because raw lives in immutable time blocks, the compactor rolls a closed block into a rollup block exactly once and never revisits it. A late point arriving after its block closed is handled by a short grace window before finalization, or lands in the next block and is reflected there - the eventual-consistency tolerance from the NFRs makes the rare misplaced point a non-event, unlike an exactly-once billing system.
  • Retention is a whole-block drop. Expiring “raw older than 7 days” means deleting the raw blocks whose time range is entirely past the cutoff - a metadata delete of whole immutable files, no row scanning, no tombstones, no vacuum. Time-partitioning makes retention O(number of blocks), not O(number of points).
  • Tiering moves cold blocks to object storage. Recent raw and rollups live on fast SSD; once a block ages past the hot window, the compactor copies it to object storage (S3/GCS) at ~1/5 the cost per GB and drops the SSD copy, updating the block catalog. Queries that need cold data read from object storage (slower, but rare and allowed up to ~1s). The block catalog - a small metadata service listing every block, its series/time range, resolution, and location - is what makes the tier transparent to queries.

The trade is explicit and permanent: you give up the ability to zoom into 10s detail on old data in exchange for storing and querying it 360x cheaper, and you make retention and tiering cheap whole-file operations. For time-series that is the right trade every time - the recent data is fat and fine, the old data is thin and coarse.

4. The Query Engine: Range-Aggregate in Milliseconds

The dominant query is aggregate(value) WHERE tags... AND time in [t1,t2] GROUP BY time(step). Three sub-problems: find the right series fast, read the right resolution, and aggregate without shipping raw points around.

Approach A: Scan all series’ blocks in the range and aggregate in the coordinator (naive)

For a query, open every block overlapping [t1,t2], stream all points to the query coordinator, filter by tag there, and aggregate.

Where it breaks:

  • No way to target series by tag. Without an index, WHERE host='web-01' means scanning every series to check its tags - millions of series read to find a handful. The time filter helps (only overlapping blocks), but within those blocks you still touch everything.
  • Shipping raw points to the coordinator saturates the network and the coordinator. A 6h/1m query over 1,000 series at raw 10s is 1,000 * 2,160 = 2.16M points streamed to one node to be bucketed there. The coordinator becomes the bottleneck and you blow the latency budget on data movement.
  • Reading raw for a long range reads 360x too much (the Approach-A-of-part-3 problem again).

Approach B: Inverted tag index + resolution selection + pushed-down aggregation

  • Inverted index for tags -> series. Maintain an inverted index: for each tag=value pair, a postings list of the series_ids that carry it. {region="blr", env="prod"} is answered by intersecting the two postings lists, yielding the matching series_ids directly - no scan. This is a search-index structure (Prometheus’s labels index, InfluxDB’s TSI). Its size is proportional to cardinality (distinct series), so the index must be bounded by the same discipline as the storage; a runaway high-cardinality tag (a request_id) explodes the index and is the classic way a TSDB falls over, so dangerous tags are dropped or bucketed at ingest. (The full cardinality-control story is a system in itself; here the key point is the index exists and is cardinality-bounded.)
  • Resolution selection from the step. The engine looks at the requested time range and GROUP BY time(step) and picks the coarsest stored resolution whose interval is <= the step: a 1m step reads raw or 1m; a 1h step reads the 1h rollup. A 90d/1h query reads 2.16M rollup points instead of 778M raw - the 360x win, and the reason the query fits in the latency budget.
  • Push aggregation down to the storage shards. The coordinator sends the time range, bucket step, and aggregation function to each shard owning relevant series. Each shard, per overlapping block, decompresses only the needed series’ value column and computes partial per-bucket aggregates (count, sum, min, max per output bucket) locally, returning a handful of numbers per bucket instead of the raw points. The coordinator merges partials (sum the sums, etc.) and applies the final function. Network carries kilobytes, not megabytes, and the decompress-and-aggregate runs in parallel across shards.
  • Result cache and query cache for repeated reads. Dashboard-style queries repeat every few seconds over mostly-overlapping ranges. Cache completed result buckets keyed by (series set, resolution, bucket range); a re-poll only computes the newest bucket and reuses the rest. This turns the common case into a near-free cache hit and is what holds p99 under 200ms under thousands of QPS.
  • Streaming / chunked execution for big scans. A heavy cold query (months from object storage) streams block-by-block and aggregates incrementally so memory stays bounded and the first buckets return while later ones compute.

The core idea: never touch a point you do not have to, and never move a point you can aggregate in place. The tag index narrows millions of series to the few you want, time-partitioning narrows blocks to the range, resolution selection narrows points by 360x, and pushed-down aggregation keeps the raw points on the node that already has them decompressed. Those four filters, stacked, turn a trillion-point store into a millisecond range-aggregate.

API Design & Data Schema

Write API (the write plane)

POST /v1/write
  Content-Type: application/x-protobuf        # batched, snappy-compressed
  body: [
    { "measurement":"cpu_usage",
      "tags":{ "host":"web-01", "region":"blr", "core":"3" },
      "points":[ {"ts":1763251200000,"value":74.2},
                 {"ts":1763251210000,"value":74.3} ] },
    ...
  ]
  204 No Content         -> points durably buffered (acked after WAL/log append).
  429 Too Many Requests  -> tenant over rate/cardinality cap (Retry-After header).
  400 Bad Request        -> malformed point / bad timestamp.

Notes:
  - Fire-and-forget: 204 means durably logged, not yet flushed to a block.
  - Duplicate (series, ts) is last-write-wins; no per-point UPDATE exists.
  - Points far outside the acceptable time window (too old / future) are
    rejected and counted in a dropped_points self-metric.

Query API (the read plane)

# Range-aggregate query (the dominant shape)
POST /v1/query
  {
    "measurement":"cpu_usage",
    "match":{ "region":"blr", "env":"prod" },     # tag matchers -> series
    "range":{ "start":1760659200000, "end":1763251200000 },
    "step":"1h",                                   # server picks the resolution
    "agg":"mean",                                  # sum|count|mean|min|max|last|p99
    "group_by":["host"]                            # optional tag grouping
  }
  200 OK
  {
    "resolution":"1h",          # which stored resolution answered (raw|1m|1h)
    "series":[
      { "tags":{"host":"web-01"},
        "points":[ {"ts":1760659200000,"value":71.4}, ... ] },
      ...
    ]
  }

# Raw point read (recent, one series)
GET /v1/series?measurement=cpu_usage&match={host:"web-01"}&start=..&end=..

# Metadata / introspection
GET /v1/tags                              # all tag keys
GET /v1/tags/region/values                # values for a tag
GET /v1/cardinality?by=measurement&top=20 # top measurements by series count

Admin / policy API (the control plane)

POST /v1/retention
  { "measurement":"cpu_usage",
    "tiers":[ {"resolution":"raw","keep":"7d"},
              {"resolution":"1m","keep":"30d"},
              {"resolution":"1h","keep":"395d"} ] }

POST /v1/downsample
  { "measurement":"cpu_usage", "from":"raw", "to":"1m",
    "aggs":["count","sum","min","max"] }

Data model: not one table - an index, compressed blocks, a WAL, and a catalog

The engine is not a relational table; it is a set of purpose-built structures.

Series + inverted tag index (RAM-resident, cardinality-bounded):

series_id   = hash(measurement + sorted(tags))        # stable 64/128-bit id
tag index   : for each (tag=value) -> postings list of series_ids
   region="blr" -> [sid_1, sid_7, sid_31, ...]
   env="prod"   -> [sid_7, sid_31, sid_55, ...]
   query {region="blr", env="prod"} = intersect postings lists
Size: proportional to CARDINALITY (5M series ≈ 8-15 GB). Bounded by admission control.

Time-series blocks (immutable, columnar, compressed, time-partitioned):

block { id, resolution(raw|1m|1h), [start_ts, end_ts], location(ssd|object) }
  per series_id:
     timestamps : delta-of-delta encoded     (~1 bit/point steady-state)
     values     : XOR-compressed floats        (Gorilla)
     (rollup blocks store per-bucket count,sum,min,max + histogram buckets)
  per-block mini index: which series_ids are present + chunk offsets
Why NoSQL/custom, not SQL: append-mostly, immutable, columnar, range-scan,
  12x compression - a B-tree row store blows up storage 30-40x and cannot
  serve range aggregations at 500K writes/sec.

Write-ahead log (durability of the in-memory head):

per shard: append-only WAL segments on disk.
  replayed on crash to rebuild the head block; truncated after a block flush.
  sequential writes only - no random I/O on the hot path.

Block catalog (small, strongly consistent metadata store):

catalog(block_id, shard, measurement_range, series_range, resolution,
        start_ts, end_ts, tier[ssd|object], object_path, size_bytes)
  - the query engine consults it to find which blocks (and where) a query
    needs, and the compactor updates it on flush/compact/downsample/tier.
  - this is the one place that needs transactional consistency; it is tiny
    relative to the data, so a replicated SQL/Raft store fits.

The split is deliberate: an inverted index for tag lookups, compressed immutable blocks for the numbers, a WAL for head durability, and a small strongly-consistent catalog for block metadata and policy. Each structure does the one thing it is best at. It is a purpose-built TSDB, not a relational database, because the workload - append-at-head, immutable-after-flush, columnar range-aggregate, 12x-compressible - is nothing like OLTP. The only strongly-consistent piece is the small catalog; the bulk data is eventually consistent and immutable.

Bottlenecks & Scaling

Where it breaks first, in order, and the fix for each.

1. Write throughput (~0.5-2M points/sec). No single node sustains it, and a B-tree row store dies on random index writes at this rate. Fix: stateless ingest tier appending to a durable log, storage nodes sharded by hash(series_id), each appending to an in-memory head + WAL (sequential I/O only) and flushing immutable compressed blocks. Add shards to scale writes linearly; a series always lands on one node, so points stay ordered and local.

2. Storage cost over months. Raw at 500K points/sec is terabytes/month; keeping it all on SSD is unaffordable. Fix: Gorilla compression (~12x), continuous downsampling into 1m/1h rollups (6x and 360x smaller), and tiered retention moving cold raw and rollups to object storage at ~1/5 the cost. The hot SSD tier holds only the head plus a week of raw; months of history live cheap and coarse.

3. Long-range query cost. Aggregating a quarter of raw 10s data scans ~778M points per 1,000 series. Fix: resolution selection (read the coarsest rollup matching the step - 360x fewer points), pushed-down per-block aggregation (ship kilobytes of partial buckets, not raw points), and a result cache for repeated dashboard polls. This is what holds query p99 under 200ms.

4. Cardinality explosion (the index killer). A high-cardinality tag (request_id, raw URL with an ID) multiplies distinct series into the billions, blowing up the RAM-bound inverted index and toppling nodes - independent of sample rate. Fix: admission control at ingest - drop/bucket dangerous tags via relabeling, enforce per-tenant series caps that reject (429) new series beyond a budget and shed one tenant’s runaway rather than the cluster, and self-monitor top measurements by series count. Cardinality is a front-door problem, not a storage problem.

5. Hot series / hot shard. A few extremely high-rate series (a global aggregate counter) concentrate writes on one shard. Fix: hash by full series identity so series spread evenly; for a genuinely hot single series, split it at the source with a low-cardinality shard tag and sum at query time - the controlled inverse of the cardinality problem.

6. Late and out-of-order points. A buffered agent reports 09:59 data at 10:03, after that head window is finalizing. Fix: a short grace window before a block is finalized/downsampled; genuinely late points land in a later block or are dropped and counted. Monitoring’s eventual-consistency tolerance (NFR) makes the rare misplaced point a non-event, versus exactly-once systems that would need reprocessing.

7. Compaction / downsampling I/O. The background compactor rewriting and rolling up blocks competes with foreground reads/writes for disk. Fix: rate-limit and schedule compaction, run it on separate I/O budget, prefer merging many small blocks (reduces per-query file opens) off-peak, and keep operations whole-block and idempotent so they can be paused/resumed. Immutability means compaction never blocks readers - old blocks stay readable until the new one atomically replaces them in the catalog.

8. Node loss with an un-flushed head. A storage node crashes holding hours of in-memory head data. Fix: WAL replay rebuilds the head on restart; the durable ingest log lets a replacement node replay from the last committed offset; replicate each shard (N=3) so a reader fails over to a replica with no data loss. Acknowledged data survives because it is in the log and the WAL before it is acked.

9. Query coordinator / fan-out hotspots. A query touching 10,000 series fans out to many shards and a single coordinator merges everything. Fix: partial aggregation on the shards shrinks what the coordinator merges; parallelize the merge; for very wide queries, a two-level merge (shard -> region aggregator -> coordinator). Push work down and out, keep the coordinator doing a cheap final combine.

10. Object-storage read latency for cold queries. A months-back query hits object storage, which is slower and has per-request overhead. Fix: large coalesced block reads (one ranged GET per block, not per chunk), a local SSD cache of recently-read cold blocks, and the latency budget that already allows ~1s for cold queries. Cold reads are rare; the rollups make them small; the catalog points straight at the object.

Wrap-Up

The trade-offs that define this design:

  • Log-structured, append-only engine over a B-tree row store. We write to an in-memory head + WAL and flush immutable, time-partitioned, columnar blocks, trading in-place updates (which we do not need) for sequential-only I/O that sustains millions of ordered writes/sec and compresses 12x - a row store would blow up storage 30-40x and thrash its index at this rate.
  • Compress the pattern, not the bytes. Delta-of-delta timestamps and XOR-encoded values exploit the known shape of time-series data (evenly-spaced, slowly-changing) to hit ~1.3 bytes/point, a ratio general compressors cannot reach - and compression is what makes storing months affordable at all.
  • Downsampling and tiering: resolution and heat for cost. Raw lives days on SSD; 1m and 1h rollups (storing sufficient statistics, and histograms for percentiles) live months on object storage. We give up 10s detail on old data for 360x cheaper long-range storage and queries, and retention becomes a cheap whole-block drop.
  • Push every filter down. A cardinality-bounded inverted index narrows millions of series to the few matched, time-partitioning narrows blocks to the range, resolution selection cuts points 360x, and pushed-down aggregation keeps raw points on the node that already decompressed them - stacking those is what turns a trillion-point store into a millisecond range-aggregate.
  • Eventual consistency over exactly-once. A point acknowledged is durable, but visibility lags by a second and a rare late point may be misplaced; we spend that relaxation on throughput and latency, because time-series tolerates it and billing-grade strictness would cost both.

One-line summary: a stateless ingest tier sharding points by series into log-structured storage nodes that append to a WAL and in-memory head, flush Gorilla-compressed immutable time-partitioned blocks, and run a background compactor that downsamples into a rollup hierarchy and tiers cold blocks to object storage - fronted by a query engine that uses an inverted tag index, resolution selection, and pushed-down aggregation to answer range-aggregates in milliseconds, trading exactly-once and full-resolution history for a database that ingests millions of points per second and stores them cheaply for months.