Everyone thinks a monitoring system is a database with a graph on top. “Every server writes its CPU and memory somewhere, you draw a line chart, and you send an email when the line goes red.” Then the interviewer adds the constraints that turn it into a real system: it is not a hundred servers, it is millions of time series reporting every few seconds; the same metric name with a slightly different label becomes a brand-new series, so a careless user_id label can explode your storage a thousandfold overnight; a query for “p99 latency across the fleet for the last 30 days” must not scan a trillion raw points; and an alert that fires two minutes late during an outage is worse than useless because the pager is how you find out production is down.

A metrics system is deceptive because the naive “write points to a table, SELECT them back” demo works fine for one dashboard and then dies against the actual shape of the workload: a relentless write firehose of numbers, a query surface that is almost always aggregating over ranges rather than fetching single rows, a cardinality problem that quietly bankrupts you, and an alerting engine that has to re-evaluate thousands of rules over fresh data continuously without ever falling behind real time. The whole design comes down to four decisions: how you ingest millions of samples per second without losing them (a purpose-built time-series path, not row inserts), how you keep queries and storage cheap over long ranges (downsampling into rollups so old data is coarse and small), how you stop cardinality from exploding (label discipline, limits, and a series index that does not fall over), and how you evaluate alerts continuously and fire in seconds (a rule engine reading recent data, not the whole history). All of it while ingest never blocks, dashboards stay under a second, and the pager is trustworthy.

Let me do it properly.

Functional Requirements (FR)

In scope:

  • Ingest metrics. Accept a stream of samples, each being (metric_name, {labels}, timestamp, value) - for example http_requests_total{service="checkout", region="ap-south-1", status="500"} @ 1751... = 42. Support counters, gauges, and histograms. This is the write firehose.
  • Store time series efficiently. A series is a unique (metric_name, label-set) combination; each series is a sequence of (timestamp, value) points. Store them so that writing is cheap and range queries are fast.
  • Query and aggregate over time ranges. A dashboard or an ad-hoc query asks “average CPU per host over the last hour” or “sum of request rate grouped by status code over 7 days.” Support aggregation (sum, avg, min, max, count, quantiles) and grouping by labels, over arbitrary ranges.
  • Downsample old data. Raw high-resolution points (say every 10s) are kept for a short window; older data is rolled up into coarser resolutions (1m, 5m, 1h) so a month-long query reads thousands of points, not billions.
  • Alerting. Let users define rules (“p99 latency > 500ms for 5 minutes”, “error rate > 1%”) that are evaluated continuously against fresh data, and fire notifications through channels (page, Slack, email) with grouping and deduplication so one incident is not a thousand pages.
  • Dashboards. Serve the aggregated query results to a visualization layer with sub-second latency for common panels.

Explicitly out of scope (say it so you own the scope):

  • Logs and traces. This is a metrics system - numeric time series. Full-text logs and distributed traces are separate stores with different access patterns (search, not aggregation). A real observability platform has all three, but conflating them in one design is how you lose the interview. We note where traces would attach (exemplars on a metric) and move on.
  • Long-term data warehousing / BI. Feeding metrics into an analytics warehouse for business reporting is a downstream concern. We serve operational monitoring: recent-to-medium-term, aggregation-heavy, latency-sensitive.
  • Anomaly detection / ML forecasting. Auto-baselining and predictive alerting is a large separate system on top of the query layer. We build threshold and rate-based alerting and note where an ML scorer would plug in.
  • The agent / instrumentation libraries. How each service exposes its metrics (a /metrics endpoint, a client SDK) is assumed. We design the pipeline from “samples arriving” onward.
  • Incident management. On-call schedules, escalation policies, and incident tracking (the PagerDuty layer) consume our alerts; we fire the alert and hand off.

The one decision that drives everything: a metrics system is a write-heavy, aggregate-over-ranges problem where the number of distinct series (cardinality) is the thing that kills you, so it must ingest a firehose cheaply, make old data small via downsampling, keep cardinality bounded, and evaluate alerts on fresh data continuously. So the design is dominated by a time-series-optimized ingest and storage path, a rollup hierarchy that trades resolution for size, a series index and cardinality controls that stop label explosions, and an alerting engine decoupled from the storage read path.

Non-Functional Requirements (NFR)

  • Scale: on the order of 10M active time series being written, each sampled every ~10s. That is roughly 1M samples/sec sustained (10M / 10s), spiking to ~2-3M/sec during incidents when services emit more, autoscaling adds hosts, and retries pile up. Reads are far lighter - dashboards and alert evaluations - but each read aggregates over many series and many points.
  • Latency: ingest is fire-and-forget and must accept a sample in a few milliseconds without blocking the reporting agent. Dashboard query p99 under ~1s for common panels (served largely from rollups and cache). Alert evaluation latency - data arriving to alert firing - on the order of seconds to ~1 minute, because a late page is a missed outage.
  • Availability: the ingest path and the alerting path are the critical ones and target 99.9%+. If ingestion drops, you are blind; if alerting is down, you do not find out you are blind. The query/dashboard path can degrade to slightly stale rather than error.
  • Consistency: monitoring data is eventually consistent and tolerant of small gaps. A missing sample or a series that is a few seconds behind is acceptable - you are looking at trends and thresholds, not billing to the cent. This is a crucial relaxation versus the ad-aggregator case: we do not need exactly-once. Approximate-but-timely beats exact-but-late for operations.
  • Durability: reasonable, not extreme. Recent data (hours to days) at full resolution must survive node failures - that is what you look at during an incident. Old data lives as downsampled rollups; losing a single raw point among billions is a non-event. Retention is tiered: raw for days, rollups for months to years.
  • Cardinality is a first-class NFR. The system must bound the number of distinct series and reject or drop pathological label sets before they blow up the index and storage. This is the single most important operational constraint and most interviewees forget it.

Back-of-the-Envelope Estimation (BoE)

Let me ground the design in numbers.

Traffic:

Active time series:            10,000,000        (10M)
Scrape / report interval:      10 seconds
Samples/sec = 10M / 10s     =  1,000,000 samples/sec   (sustained)
Incident spike (~2-3x)      ≈  2,500,000 samples/sec    (size for this)

The sustained 1M samples/sec is the honest baseline, but monitoring load is anti-correlated with health: exactly when production is on fire, services emit more, autoscaling spins up hosts (more series), agents retry buffered samples, and every engineer opens a dashboard. So the write path and the alert path both get their worst load at the worst time. We size ingest for ~2.5M samples/sec and treat spare capacity as an incident-survival feature, not waste.

Read vs write:

Writes (samples):     ~1M/sec sustained, ~2.5M/sec peak. The firehose.
Reads:
  - Alert evaluation: say 5,000 alert rules, each re-evaluated every 15s.
      = 5,000 / 15s ≈ 333 rule-evaluations/sec, but EACH scans many
        series over a few-minute window -> the dominant internal read load.
  - Dashboards:       thousands of panels, polled every 10-30s when open.
      Order of low thousands of queries/sec, each aggregating many series.

Writes dominate raw sample volume by orders of magnitude, but reads are not cheap: every read aggregates over hundreds or thousands of series and a window of points. So this is not a 1000:1 “writes crush reads” shape like a click counter - it is “writes are a firehose and reads are fan-out aggregations.” Both paths need engineering, and the shared enemy of both is cardinality.

Storage:

Raw point on disk (compressed):  ~1-2 bytes/point (delta-of-delta timestamps
                                 + XOR-compressed float values - the Gorilla
                                 encoding). Uncompressed it is 16 bytes;
                                 compression is the whole game here.

Raw ingest volume:   1M samples/sec * 1.5 bytes ≈ 1.5 MB/sec ≈ 130 GB/day
                     compressed. (Uncompressed it would be ~1.4 TB/day -
                     compression buys ~10x.)

Series index (the expensive part):
   10M series * (name + labels + postings entries) ≈ ~1-2 KB/series
   ≈ 10-20 GB of INDEX just to map label queries -> series IDs.
   This grows with CARDINALITY, not with sample volume - and it is what
   falls over first when someone adds a high-cardinality label.

Two things fall out. First, compression is not optional: at 1M points/sec, the difference between 16 bytes and 1.5 bytes per point is the difference between a sane cluster and a bankrupt one, so the storage engine uses delta-of-delta timestamp encoding and XOR float compression (the Gorilla scheme). Second, the index scales with cardinality, not sample rate - 10M series is a manageable ~15 GB index, but if a bad label pushes it to 1B series, the index is now ~1.5 TB of RAM-hungry postings and the whole system tips over. That is why cardinality is a first-class requirement.

Downsampling payoff:

Raw (10s resolution) kept:  ~7 days.
   1M samples/sec * 86,400 * 7 ≈ ~600B points, ~900 GB compressed.
Rolled up to 1m:  6x fewer points. Kept ~30 days.
Rolled up to 1h:  360x fewer than raw. Kept ~1-2 years.

A "last 30 days, per hour" query over 1,000 series reads:
   1,000 series * 720 hourly points = 720,000 points  (rollup)
vs the same query over raw 10s data:
   1,000 series * 259,200 raw points = 259,200,000 points  (360x more)

Downsampling is the query-cost lever: a month-long dashboard panel over rollups reads ~720K points; over raw data it reads ~260M. Same answer, 360x less work. Old data is coarse and small; recent data is fine and fat. That trade - resolution for size - is the storage shape of every monitoring system.

Bandwidth:

Ingest: ~1M samples/sec * ~25 bytes on the wire (batched, compressed)
        ≈ 25 MB/sec ≈ 200 Mbps inbound, spread across an ingest fleet.
Query:  dashboard responses are downsampled series, KBs each,
        low thousands/sec -> a few MB/sec. A non-issue.

Ingest bandwidth is modest when agents batch and compress; the scarce resources are sustained write throughput into the TSDB, RAM for the series index, and read fan-out for alert evaluation. Size for those, not for the network.

High-Level Design (HLD)

The system is an ingest pipeline feeding a time-series database, with a rollup engine making old data cheap, a query layer serving dashboards, and an alerting engine reading fresh data on a loop. Agents push (or are scraped for) samples; a stateless ingest tier validates, applies cardinality limits, and appends to a durable buffer; the TSDB writers consume the buffer, compress, and write to a time-partitioned store; a compactor downsamples; the query engine and alerter read.

              WRITE PATH (samples, ~1-2.5M/sec, fire-and-forget)
   ┌───────────────────────────────────────────────────────────────────────┐
   │  Service agents / exporters  --push OR scrape-->  Ingest Gateway (LB)   │
   └───────────────────────────────────────┬─────────────────────────────────┘
                                            │  batched, compressed samples
                                  ┌─────────▼──────────┐
                                  │   Ingest Service   │  (stateless, fleet)
                                  │  - validate schema │
                                  │  - CARDINALITY gate│  (limit series/tenant,
                                  │    + relabel/drop  │   drop bad labels)
                                  │  - route by series │
                                  └─────────┬──────────┘
                                            │ append (partition by series hash)
                             ┌──────────────▼───────────────┐
                             │  KAFKA / WAL  (durable buffer) │  absorbs spikes,
                             │  partitioned by series hash    │  decouples ingest
                             └───┬───────────────────────┬────┘  from storage
                                 │                       │
                     ┌───────────▼──────────┐   ┌────────▼───────────────┐
                     │  TSDB Writer (shard)  │   │  TSDB Writer (shard)   │  ...N
                     │  - in-mem head block  │   │  (each owns a set of   │
                     │  - compress (Gorilla) │   │   series by hash)      │
                     │  - flush to blocks    │   └────────┬───────────────┘
                     └───────────┬──────────┘            │
                                 │  time-partitioned blocks (2h) + series index
                     ┌───────────▼────────────────────────▼───────────────┐
                     │        Storage: recent (SSD)  +  archival (object)  │
                     │  raw 10s (7d) | 1m rollup (30d) | 1h rollup (1-2y)  │
                     └───────────┬──────────────────────────────┬─────────┘
                     ┌───────────▼──────────┐        ┌───────────▼─────────┐
                     │  Compactor / Rollup  │        │   Query Engine      │
                     │  - merge 2h blocks   │        │  - PromQL-style     │
                     │  - downsample ->1m,1h│        │  - fan out to shards│
                     │  - apply retention   │        │  - aggregate + cache│
                     └──────────────────────┘        └──────┬──────────────┘
                                             READ PATH        │
                        ┌─────────────────────────┬───────────┴───────┐
                        │   Alerting Engine        │   Dashboard API   │
                        │  - eval rules on a loop  │   (panels, cache) │
                        │  - for-duration state    │        │
                        │  - Alert Manager: group, │   ┌────▼────┐
                        │    dedupe, route, silence│   │   UI    │
                        └──────────┬───────────────┘   └─────────┘
                                   │ page / Slack / email
                             ┌─────▼──────┐
                             │  Notifiers │
                             └────────────┘

Request flow - a sample being written (the write hot path):

  1. Each service exposes metrics; an agent either pushes batches to the ingest gateway or the gateway scrapes the service’s /metrics endpoint on an interval. Pull (scrape) is the Prometheus model and gives the system control over timing and a natural liveness signal (“target down” = scrape failed); push (the OTel / StatsD model) suits short-lived jobs and serverless. We support both; internally everything becomes samples.
  2. The stateless Ingest Service validates the sample schema, then hits the cardinality gate: it applies relabeling rules (drop or aggregate away dangerous labels like user_id, request_id), and enforces per-tenant series limits. A sample belonging to a new series that would exceed the tenant’s cap is rejected and counted, not silently stored. This gate is what stops a label explosion from taking down the cluster.
  3. Accepted samples are appended to a durable buffer (Kafka or a replicated write-ahead log), partitioned by a hash of the series identity so all points of one series go to the same writer shard, in order. The ingest node acks fast; the sample is durable and will be stored even if a writer is momentarily behind.
  4. A TSDB Writer shard consumes its partitions, appends each point to an in-memory head block for its series, and periodically compresses and flushes completed time windows (say every 2 hours) to immutable on-disk blocks, each carrying its own series index. Recent data lives on fast SSD; a write-ahead log protects the not-yet-flushed head block against a crash.
  5. The Compactor runs in the background: it merges small blocks into larger ones, downsamples raw points into 1m and 1h rollups, and enforces retention (delete raw past 7 days, 1m past 30 days, 1h past 2 years), tiering old blocks to cheap object storage.

Request flow - the read paths:

  1. Dashboard query: the Query Engine parses a query (a PromQL-style expression: histogram_quantile(0.99, sum by (le) (rate(http_latency_bucket[5m])))), uses the series index to resolve the label matchers to a set of series IDs, picks the coarsest resolution that satisfies the range (raw for last hour, 1h rollup for last month), fans out to the shards owning those series, aggregates the partial results, and returns. Common panels hit a result cache.
  2. Alert evaluation: the Alerting Engine runs each rule on its own interval. A rule is just a query over a short recent window; the engine evaluates it, applies the for-duration (the condition must hold for N minutes before firing to avoid flapping), and emits alert state changes to the Alert Manager, which groups, deduplicates, silences, and routes them to notifiers.

The key architectural insight: the number of distinct series is the master variable, and downsampling plus cardinality control are the two levers that keep both storage and query cost bounded as it grows. Ingest is decoupled from storage by the durable buffer so a slow writer never blocks a reporting agent; storage is time-partitioned and compressed so writes are sequential appends and old blocks are immutable and cheap to downsample or drop; and the alerting engine reads only fresh, narrow windows so it stays real-time regardless of how much history exists. The TSDB is not a general database - it is a specialized append-mostly, aggregate-over-ranges engine, and every design choice follows from that.

Component Deep Dive

Four hard parts: (1) ingesting and storing a time-series firehose cheaply, (2) downsampling into a rollup hierarchy so old data is small, (3) controlling cardinality so a label explosion does not kill the cluster, and (4) evaluating alerts continuously and firing in seconds. I will walk each from naive to scalable.

1. Ingesting and Storing Time Series: Row Store -> Purpose-Built TSDB

We write ~1M samples/sec and read them back as range aggregations. The storage engine choice dominates everything.

Approach A: Store points as rows in a relational database (the naive instinct)

A table metrics(series_id, ts, value) with an index on (series_id, ts), one row per sample.

INSERT INTO metrics (series_id, ts, value) VALUES (...);
SELECT avg(value) FROM metrics WHERE series_id IN (...) AND ts BETWEEN ? AND ?;

Where it breaks: at 1M inserts/sec, a B-tree index on (series_id, ts) is death by random writes - every insert updates the index, pages thrash, and write amplification crushes you. Each sample is a full row with per-row overhead (a tuple header, the series_id repeated, the timestamp as 8 bytes, the value as 8 bytes) - call it 30-60 bytes per point on disk versus the ~1.5 bytes a TSDB achieves, a 20-40x storage blowup. And the read pattern is wrong for a row store: “average over a 5-minute range for 1,000 series” scatter-gathers across the index and reads full rows to extract one float column each. It works for a demo and falls over at real volume. A row store is optimized for random point reads and updates; monitoring is sequential appends and range scans.

Approach B: A time-series storage engine (append-only, compressed, time-partitioned)

Build (or use) a TSDB storage layout designed for this exact access pattern. Three ideas make it work:

  • Columnar, per-series compression. For each series, timestamps and values are stored as compressed columns, not rows. Timestamps use delta-of-delta encoding: samples are near-evenly spaced (every 10s), so the delta is ~10,000ms every time and the delta-of-delta is usually 0, which packs to a single bit. Values use XOR compression (the Gorilla scheme): consecutive gauge/counter values are similar, so value XOR prev_value has many leading/trailing zero bits that pack tightly. Together this is the ~16 bytes -> ~1.5 bytes/point win. Compression is the difference between a viable and an impossible system.
timestamps: 1751000000, +10000, +10000, +10000, ...
   delta-of-delta:      0,      0,      0     -> ~1 bit each
values:     42.0, 42.1, 42.1, 43.0, ...
   XOR with previous -> mostly-zero -> a few bits each
Result: ~1-2 bytes per (timestamp, value) point on disk.
  • Time partitioning into immutable blocks. Data is written into a mutable in-memory head block covering the current ~2 hours; when a window closes it is compressed and flushed to an immutable on-disk block. Blocks are named by their time range, so a query for a range simply picks the blocks it overlaps and ignores the rest - no scanning irrelevant time. Immutability makes compaction, downsampling, and deletion trivial (drop the whole block) and makes replication safe (blocks never change).
  • A write-ahead log for the head. The in-memory head is not yet durable, so every append is first written to a WAL on disk. A crash replays the WAL to reconstruct the head. This gives durability without paying random-write cost on the main store - the WAL is a sequential append too.

Ingest writers are sharded by series hash: each writer owns a slice of the series space, so a series always lands on the same writer and its points stay ordered and local. Writers are stateful (they hold the head block) but recoverable from the WAL and the buffer. This is the design of every serious TSDB (Prometheus TSDB, InfluxDB TSM, M3DB, VictoriaMetrics) - the shapes differ but the ideas are the same: compress per series, partition by time, append never update.

2. Downsampling: Keeping Old Data Small

Raw 10s data for a year is impossible to store or query. But you rarely look at 10s resolution for last quarter - you want the trend.

Approach A: Keep everything at full resolution (naive)

Store every 10s point forever and let queries scan whatever range they ask for.

Where it breaks: storage grows without bound - 1M points/sec is ~600B points/week even compressed, and a year is unthinkable. Worse, query cost scales with the range at full resolution: “last 90 days, per hour” over 1,000 series would scan 1,000 * 90 * 8,640 = ~780M raw points to produce ~2,160 output points, throwing away 99.7% of the work. You cannot serve a sub-second dashboard by reading three-quarters of a billion points. Full resolution forever is both a storage and a query disaster.

Approach B: A rollup hierarchy - trade resolution for size and speed

The compactor continuously downsamples raw data into coarser resolutions, and queries read the coarsest resolution that still answers the question:

Resolution   Interval   Retention     Points/series/day
raw          10s        7 days        8,640
1m rollup    1m         30 days       1,440    (6x smaller)
1h rollup    1h         1-2 years     24       (360x smaller than raw)
  • What a rollup stores. A downsampled point is not just an average - for each coarse bucket you store the aggregates you might need: count, sum, min, max (and for that you can derive avg = sum/count). This is critical: if you only stored the average, you could never later compute a correct sum or max over a wider range. Storing the sufficient statistics per bucket lets downstream aggregation stay correct across resolutions.
  • Histograms downsample by summing buckets. For latency percentiles you do not store a pre-computed p99 (percentiles are not averageable). You store the histogram bucket counts (le="100ms": 900, le="500ms": 990, ...) and downsample by summing the per-bucket counts, then compute the quantile at query time. This is why the histogram is a first-class metric type.
  • Query resolution selection. The query engine looks at the requested range and the panel width (how many pixels / points it can show) and picks the resolution: last hour -> raw; last week -> 1m; last quarter -> 1h. A “last 30 days per hour” query now reads ~720 points/series instead of ~260M across the fleet - the 360x win from the BoE.
  • Downsampling is idempotent and block-scoped. Because raw data lives in immutable time blocks, the compactor downsamples a closed block exactly once into a rollup block and never revisits it. Late data (a sample arriving after its block closed) is handled by a short grace window before a block is finalized, or lands in the next block - monitoring tolerates the rare misplaced point, which is exactly the consistency relaxation we bought in the NFRs.

The trade is explicit and permanent: you lose the ability to zoom into 10s detail on old data, in exchange for storing and querying it 360x cheaper. For operations that is the right trade - nobody debugs last March at 10-second granularity - and the retention tiers make it a policy knob per metric.

3. Cardinality Control: The Thing That Actually Kills You

Cardinality is the number of distinct series, and it is the master variable. Every hard limit in the system is really a cardinality limit.

Approach A: Let any label with any value through (naive)

Accept whatever labels the instrumented code emits. http_requests_total{path="/user/12345", user_id="u_998", request_id="req_abc"}.

Where it breaks: each unique label-value combination is a new series, and series count is the product of every label’s distinct values. A metric with service (100 values) x region (10) x status (5) is 5,000 series - fine. Add user_id with a million values and it is 5 billion series. The series index (which maps label matchers to series IDs and must be largely in RAM) explodes from ~15 GB to terabytes, the writers run out of memory tracking that many head series, queries that match on a label have to walk enormous postings lists, and the whole cluster tips over - all from one careless label added in a deploy. This is the single most common way real monitoring systems fall over, and “just store more” does not fix it because the index is memory-bound. High-cardinality labels are a footgun that must be disarmed at the door.

Approach B: Bound cardinality at ingest with limits, relabeling, and an efficient index

Treat cardinality as a resource to be budgeted and enforced:

  • The series index, done right. To resolve {service="checkout", status="500"} to series IDs you need an inverted index: for each label=value pair, a postings list of the series IDs that have it; a query intersects the postings lists of its matchers. This is exactly a search index (Prometheus uses a labels index; others embed Lucene-like structures). It is fast, but its size is proportional to cardinality - which is precisely why cardinality must be bounded, not just indexed.
  • Relabeling / metric relabeling at ingest. Before a sample is stored, run configurable rules that drop dangerous labels (user_id, request_id, session_id, full URLs with IDs in them), collapse high-cardinality values (bucket path="/user/12345" into path="/user/:id"), or drop entire metrics. This is the primary defense: kill the explosion before it enters the index.
  • Per-tenant / per-metric series limits. Each tenant (team, service) gets a series budget. The ingest gate tracks how many active series a tenant has; a sample that would create a new series beyond the cap is rejected and counted in a dropped_samples metric, and the tenant is alerted that they are hitting their limit. Rejecting is correct: better to lose one team’s runaway metric than to take down monitoring for everyone. This is a load-shedding decision - protect the shared system first.
  • Cardinality observability. The system monitors itself: top metrics by series count, series growth rate, and per-label cardinality, surfaced in a dashboard so an operator can see “metric X grew from 10K to 2M series after this morning’s deploy” and act. You cannot control what you cannot see.
  • Label value length and count caps. Reject samples with absurd label values (a 10KB label) or too many labels, which are almost always bugs.
ingest sample s:
   s = relabel(s)                       # drop/rewrite dangerous labels
   if s == null: dropped++; return      # metric explicitly dropped
   sid = series_id(s)                    # hash(name + sorted labels)
   if not index.exists(sid):            # this would be a NEW series
        if tenant.series_count >= tenant.limit:
             rejected_new_series++; alert_tenant(); return   # SHED LOAD
        tenant.series_count++
        index.add(sid, s.labels)
   append(sid, s.ts, s.value)

The mindset shift: cardinality is not a storage problem to solve later, it is an admission-control problem to enforce at the front door. Relabeling defuses the common accidents, per-tenant limits contain the rest and turn a cluster-wide outage into one team’s throttled metric, and self-monitoring makes the invisible visible. Every mature monitoring system that survived did so by getting strict about this.

4. Alerting: Continuous Evaluation Without Falling Behind

Alerts are why the system exists. A rule (“error rate > 1% for 5 minutes”) must be evaluated against fresh data on a loop and fire within seconds, and one incident must not become a thousand pages.

Approach A: Query the whole history on a cron and diff (naive)

A cron job every minute runs each alert’s query over a long range and checks the threshold, then sends an email if it is breached.

Where it breaks: three ways. First, it does not scale with rule count - re-running thousands of rules, each a heavy query, on a tight loop hammers the same read path the dashboards use, and evaluation falls behind real time exactly during an incident when it matters most. Second, it flaps - a metric bouncing across the threshold for one scrape fires and resolves repeatedly, paging on noise. Third, it has no grouping - if 500 hosts all breach at once (a shared dependency failed), you get 500 pages for one incident, which is how on-call engineers learn to ignore the pager. Naive alerting is either slow, noisy, or both.

Approach B: A dedicated evaluation engine + an alert manager that groups and dedupes

Split alerting into two stages: evaluation (is the condition true?) and notification management (who should be told, and how, without spamming).

  • Evaluation reads fresh, narrow windows only. Each rule is a query over a recent window (rate(errors[5m])), never the whole history, so evaluation cost is bounded by rule count times a small window, independent of retention. The evaluator runs on its own interval per rule (e.g. every 15s) and is sharded across rules so thousands of rules evaluate in parallel across a fleet - the alert path scales horizontally and is isolated from the dashboard read path so a heavy dashboard cannot starve alerting.
  • For-duration to kill flapping. A rule fires only after its condition has held continuously for a configured for duration (e.g. “> 1% for 5m”). The engine keeps per-alert state: pending when the condition first goes true, firing once it has held for the full duration, resolved when it goes false. A one-scrape blip never pages because it never survives the for window. This stateful debounce is the difference between a trustworthy pager and an ignored one.
per rule, each interval:
   v = evaluate(rule.expr, window)        # e.g. rate(errors[5m]) / rate(reqs[5m])
   if v breaches threshold:
        if state == inactive: state=pending, since=now
        else if now - since >= rule.for: state=firing -> emit ACTIVE alert
   else:
        if state == firing: emit RESOLVED
        state = inactive
  • The Alert Manager: group, dedupe, silence, route. Firing alerts do not go straight to a pager. They flow into an Alert Manager that:
    • Groups related alerts (all status=500 alerts for one service, or all alerts sharing a cluster label) into a single notification, so a shared-dependency failure is one page describing 500 affected hosts, not 500 pages.
    • Deduplicates identical alerts coming from redundant evaluators (you run the evaluator replicated for availability, so the same alert may fire twice - dedupe by alert identity).
    • Silences / inhibits: an operator can silence alerts during a known maintenance window, and an inhibition rule suppresses downstream alerts when a root-cause alert is already firing (“if the whole datacenter is down, do not also page for every service in it”).
    • Routes by labels and severity to the right channel and on-call - severity=page to PagerDuty, severity=ticket to Slack, with escalation.
  • Notification delivery is retried and stateful. Sending a page is a durable action: the manager retries failed deliveries and tracks acknowledgement, because a dropped page during an outage is the worst possible failure. Notifiers are idempotent per alert-incident so a retry does not double-page.
  • Availability of the alert path. The evaluator and alert manager are replicated and run independently of the storage cluster’s query load. If the metrics query path degrades, alerting on recent in-memory data still works. The rule of thumb: the thing that tells you production is broken must be more available than production.

Put together: sharded evaluation over fresh windows keeps alerting real-time regardless of history size; the for-duration debounce keeps it from flapping; and the alert manager’s grouping, dedup, and inhibition keep one incident to one page. That chain is what makes the pager trustworthy, and a trustworthy pager is the entire point of the system.

API Design & Data Schema

Ingest API (the write plane)

# Push model (agents / short-lived jobs / OTel)
POST /v1/write
  Content-Type: application/x-protobuf   (batched, snappy-compressed)
  body: [
    { name:"http_requests_total",
      labels:{ service:"checkout", region:"ap-south-1", status:"500" },
      samples:[ {ts:1751932800000, value:42}, {ts:1751932810000, value:43} ] },
    ...
  ]
  204 No Content   -> accepted & durably buffered. Async storage.
  429 Too Many Requests -> tenant over series/rate limit (with Retry-After)

# Pull model (scrape) - the system fetches instead
GET  http://<target>/metrics          # target exposes text/proto exposition
  -> ingest gateway scrapes on a schedule; scrape failure = "target down" signal.

Notes:
  - Ingest is fire-and-forget: 204 after the durable buffer append, not after storage.
  - Rejected new series over a tenant's cardinality cap return 429 and increment
    a `dropped_samples{reason="series_limit"}` self-metric.

Query API (the read plane)

# Range query (dashboards) - PromQL-style expression over a time range
GET /v1/query_range
    ?query=histogram_quantile(0.99, sum by (le,service) (rate(http_latency_bucket[5m])))
    &start=2026-07-09T00:00:00Z
    &end=2026-07-10T00:00:00Z
    &step=1h                         # server picks matching rollup resolution
  200 OK   (served from rollups + result cache)
  {
    "resolution":"1h",              # which rollup answered (raw|1m|1h)
    "series":[
      { "labels":{ "service":"checkout" },
        "points":[ {"ts":...,"value":0.482}, ... ] },
      ...
    ]
  }

# Instant query (alert evaluation, single-value panels)
GET /v1/query?query=<expr>&time=<ts>

# Metadata / cardinality introspection (self-monitoring)
GET /v1/labels                       # all label names
GET /v1/label/service/values         # values for a label
GET /v1/cardinality?by=metric&top=20 # top metrics by series count

Alerting API (the control plane)

POST /v1/rules
  { "name":"HighErrorRate",
    "expr":"sum by(service)(rate(errors[5m])) / sum by(service)(rate(reqs[5m])) > 0.01",
    "for":"5m",
    "labels":{ "severity":"page" },
    "annotations":{ "summary":"{{$labels.service}} error rate {{$value}}" } }

GET  /v1/alerts                       # current firing/pending alerts
POST /v1/silences                     # silence matching alerts for a window
  { "matchers":{ "cluster":"blr-1" }, "ends_at":"2026-07-10T04:00:00Z" }

Data model: the TSDB is not one table, it is an index + compressed blocks

Series index (inverted index, mostly in RAM):

series_id   = hash(metric_name + sorted(labels))     # stable 64/128-bit id
label index: for each (label=value) -> postings list of series_ids
   service="checkout" -> [sid_1, sid_9, sid_44, ...]
   status="500"       -> [sid_9, sid_44, sid_88, ...]
query {service="checkout", status="500"} = intersect the two postings lists
Size: ~proportional to CARDINALITY (10M series ≈ 10-20 GB). Bounded by admission control.

Time-series data blocks (immutable, compressed, time-partitioned):

block:  covers a fixed time window (e.g. 2h), on SSD; older blocks -> object store
  per series_id:
     timestamps  delta-of-delta encoded    (~1 bit/point steady state)
     values      XOR-compressed floats      (Gorilla)
  block also carries its own mini series-index (which series it contains)
Rollup blocks: same layout at 1m and 1h resolution, storing
     count, sum, min, max (+ histogram bucket counts) per bucket per series.

Write-ahead log (durability for the not-yet-flushed head):

append-only WAL on disk; replayed on crash to rebuild the in-memory head block.
Sequential writes only - no random I/O on the hot path.

Alert rule + state store (small, strongly consistent):

rules(rule_id, tenant, expr, for_duration, labels, annotations, channels)
alert_state(rule_id, fingerprint, state[inactive|pending|firing],
            active_since, last_eval)      # for the for-duration debounce

Why this split - an inverted index for label lookups, compressed time-partitioned blocks for the numbers, a WAL for durability, and a small relational store for rules/state: each structure does the one thing it is best at, and it is a purpose-built TSDB, not a relational database, because the workload (append-mostly, aggregate-over-ranges, cardinality-bounded) is nothing like OLTP. A row store would blow up storage 20-40x and cannot serve range aggregations at this scale; the raw blocks cannot answer a label-matcher query without the index; the rules need transactions the TSDB does not offer. Right tool per job.

Bottlenecks & Scaling

Where it breaks first, in order, and the fix for each:

1. Cardinality explosion (the number-one killer). One high-cardinality label (user_id, request_id) multiplies series into the billions, blowing up the RAM-bound series index and toppling the cluster. Fix: admission control at ingest - relabeling to drop/collapse dangerous labels, per-tenant series limits that reject (429) new series beyond a cap and shed one team’s runaway rather than the whole system, and self-monitoring of top metrics by series count. Cardinality is a front-door problem, not a storage problem.

2. Ingest write throughput (~1-2.5M samples/sec). No single node sustains it, and a B-tree row store dies on random index writes. Fix: stateless ingest tier appending to a durable buffer, sharded TSDB writers owning series by hash, with columnar per-series compression (delta-of-delta + XOR) making writes sequential appends. Scale writers by adding shards.

3. Query cost over long ranges. Aggregating a quarter of raw 10s data scans hundreds of millions of points per panel. Fix: downsampling into a rollup hierarchy (raw/1m/1h) with the query engine picking the coarsest resolution for the range, plus a result cache for repeated dashboard polls. 360x less work for a month-long query.

4. Hot series / hot shard. A few extremely high-throughput series (a global request counter) concentrate load on one writer shard. Fix: hash-partition by full series identity so series spread evenly; for a genuinely hot single series, split it at the source (add a low-cardinality shard label and sum at query time) - the controlled inverse of the cardinality problem.

5. Alert evaluation falling behind. Thousands of rules re-evaluated on a loop contend with dashboards and lag exactly during incidents. Fix: a dedicated, sharded evaluation fleet reading only fresh narrow windows, isolated from the dashboard read path, scaling horizontally with rule count. Evaluation cost is bounded by rule count, not retention.

6. Alert flapping and page storms. Threshold-crossing noise pages repeatedly; a shared failure pages for every affected host. Fix: for-duration debounce so a condition must hold before firing, and an Alert Manager that groups related alerts into one notification, deduplicates across replicated evaluators, and inhibits downstream alerts when a root cause is firing. One incident, one page.

7. Late and out-of-order samples. A buffered agent reports 09:59 data at 10:03, after that block/window is being finalized. Fix: a short grace window before a block is finalized and downsampled; genuinely late points land in a later block or are dropped and counted. Monitoring’s eventual-consistency tolerance (NFR) makes the rare misplaced point a non-event - a deliberate relaxation versus exactly-once billing systems.

8. Storage growth. Raw data at 1M points/sec is unbounded over a year even compressed. Fix: tiered retention (raw 7d on SSD, 1m 30d, 1h 1-2y tiered to object storage) driven by the compactor, which merges, downsamples, and deletes whole immutable blocks cheaply. Old data is coarse, small, and cold.

9. Writer/head loss. A TSDB writer crashes with an in-memory head block of un-flushed samples. Fix: write-ahead log replay rebuilds the head on restart, and the durable ingest buffer lets a replacement writer replay from the last committed offset. Combined with immutable flushed blocks, no committed data is lost.

10. The monitoring system’s own availability during an outage. Load spikes when production is on fire, and if monitoring dies you are blind and unpaged. Fix: replicate the ingest and alert paths, isolate alerting from dashboard query load, and reserve headroom sized for the incident spike (the 2.5x peak). The alerting path is engineered to be more available than the systems it watches - degrade dashboards before you ever degrade the pager.

Wrap-Up

The trade-offs that define this design:

  • Purpose-built TSDB over a relational database. We store points as compressed, time-partitioned, append-only blocks with an inverted series index, trading a familiar SQL model for 20-40x less storage and range-aggregation reads that a row store cannot serve at 1M points/sec. The workload is append-mostly and aggregate-over-ranges, so the engine is too.
  • Downsampling: resolution for size. We keep raw data for days and roll it up into 1m and 1h rollups for months and years, giving up the ability to zoom into old data at 10s detail in exchange for 360x cheaper long-range storage and queries. Nobody debugs last quarter at 10-second granularity.
  • Cardinality as admission control, not a storage afterthought. We defuse label explosions at the front door with relabeling and per-tenant series limits, shedding one team’s runaway metric rather than letting it topple the shared cluster. Series count is the master variable, and it is enforced, not hoped for.
  • Eventual consistency over exactly-once. Unlike a billing aggregator, monitoring tolerates a dropped or late sample among billions, and we spend that relaxation on timeliness - approximate-but-fresh data beats exact-but-late for operations.
  • A decoupled, debounced, grouped alerting path. Sharded evaluation over fresh windows keeps alerting real-time regardless of history, the for-duration kills flapping, and the alert manager’s grouping and inhibition keep one incident to one page - because a trustworthy pager is the whole reason the system exists.

One-line summary: a stateless ingest tier with cardinality admission control feeding a compressed, time-partitioned TSDB sharded by series, a compactor downsampling raw data into a rollup hierarchy so old data is small and cheap to query, and a sharded, debounced alerting engine reading fresh windows and routing grouped, deduplicated pages - trading exactly-once and full-resolution history for a system that ingests millions of samples per second, keeps cardinality bounded, and pages within seconds when production breaks.