Design a Feature Flag Service - System Design

A feature flag service sounds like a key-value store with a boolean in it. “Is new-checkout on? Return true or false.” That framing survives exactly until you ask the three questions that actually define the system. First: who is the flag on for - because “on” is rarely global; it is “on for 1% of users, ramping to 100%”, or “on for internal staff and users in Canada”, or “on for everyone except this one enterprise account that is paying us to be careful.” Second: how fast does a change take effect - because the entire reason flags exist is to turn a bad feature off now, and “now” means the kill switch has to reach every running instance of your app in well under a second, not on the next poll, not after a cache expires. Third: how much load does that put on your servers - because if you have 10 million SDK instances embedded in your customers’ backends and mobile apps, and each one asks a central server “what is the current flag config?” every time it evaluates a flag, you have built a self-inflicted DDoS. The real problem is this: evaluate flags locally inside millions of SDK instances with zero network latency on the evaluation path, support rich targeting and percentage-based gradual rollout that stays sticky per user, and propagate any flag change to every one of those millions of instances in under 100ms - without the central flag server collapsing under the fan-out.

Everything hard here - local evaluation, sticky percentage rollout, and above all real-time propagation to millions of listeners - serves two non-negotiable properties: evaluation is free and local (a flag check is a function call, never a network round trip, because it sits in the hot path of your customers’ code) and changes are near-instant and cheap to fan out (flipping a flag reaches everyone in under 100ms, and doing so costs the server almost nothing per connection). You do not query a flag server per evaluation; you ship the whole ruleset to the SDK and evaluate in-process. You do not poll for changes; you hold a persistent stream and push deltas the instant a flag changes. Let me build it properly.

Functional Requirements (FR)

In scope:

• Define and manage flags. A developer or PM creates a flag with a key (new-checkout), a type (boolean, string, number, JSON), a default value, and a set of targeting rules: who gets which variation. Flags can be toggled on/off, have their rules edited, and be archived.
• Evaluate a flag for a context. Given a flag key and an evaluation context (user id, email, country, plan tier, custom attributes), return the variation that context should receive - deterministically (the same user keeps the same variation across calls) and locally (no network call on the evaluation path).
• Gradual / percentage rollout. Roll a flag out to a fraction of users: 1%, then 5%, then 50%, then 100%. The set of users must grow monotonically - a user who was in the 1% stays in the 5% and the 50%; nobody flips back to the old behavior as you ramp.
• Rich targeting rules. Target by attribute (country is CA), by segment (beta testers), by individual override (always on for user_42), and combinations, with a fallback rollout for everyone else.
• Instant propagation / kill switch. A change to a flag (toggle, rule edit, rollout bump) must reach every connected SDK in under 100ms so a bad feature can be killed immediately.
• Multi-environment and multi-project. The same flag key has independent config in production, staging, dev. An SDK key scopes to one environment.
• Audit and flag state history. Who changed what flag when, with the ability to see and roll back a change.

Explicitly out of scope (say it so you own the scope):

• Experiment analysis / statistical significance. This service serves flags and can emit exposure events for an analytics system, but computing conversion lift and p-values is a separate experimentation platform I feed, not own.
• The flag management UI. The dashboard is a thin client over the management API; I design the API and the stores, not the front-end.
• Per-language SDK internals. I define the SDK contract (what it fetches, streams, caches, and evaluates) and the wire protocol, not each language implementation.
• Auth/billing/org plumbing. Assume an authenticated SDK key scopes to a project+environment, and an authenticated user edits flags. Move on.
• General remote config beyond flags. The same machinery could ship arbitrary config; I scope to flag evaluation and keep it honest.

The one decision that drives everything: flag evaluation must be a local, in-SDK computation over a cached ruleset, and config changes must be pushed over persistent streaming connections, not polled - because any per-evaluation network call adds latency to the customer’s hot path, and any polling model either wastes millions of requests or misses the sub-100ms kill-switch target. Unlike a config database you query (round trip per read) here you evaluate in-process from a cached ruleset; unlike a cache you poll (staleness bounded by the poll interval) here you hold a stream and get the delta the instant it changes. That asymmetry - evaluate locally, propagate by push - is the whole problem.

Non-Functional Requirements (NFR)

• Scale: ~10M concurrent SDK instances (server-side SDKs in customers’ backends, each a long-lived process; plus mobile/client SDKs). Hundreds of thousands of flags across all projects/environments. Flag evaluations happen in-process at the customer’s request rate - easily tens of millions per second aggregate - but they cost us nothing because they are local. Flag changes are rare (thousands/day across the whole platform) but each must fan out to a large slice of the 10M instances.
• Latency: flag evaluation effectively zero (a hash and a few comparisons, in-process, microseconds) because it sits in the customer’s request path. Change propagation under 100ms from “editor saves” to “every connected SDK has the new ruleset.” Initial SDK config fetch tolerates ~100-300ms and is cached to local disk for offline/restart.
• Availability: 99.99%+ on the serving path. If our entire backend is down, SDKs must keep evaluating from their last-cached ruleset and never block or error the customer’s app. A flag check must never throw into customer code - it fails safe to the caller-supplied default. The management/editing path can tolerate brief unavailability; the streaming/propagation path must be highly available.
• Consistency: evaluation is strongly deterministic (same context + same ruleset => same variation, everywhere, always). Config propagation is eventually consistent but fast - a change reaches all SDKs within ~100ms, and during that window some SDKs have the old ruleset and some the new. That brief divergence is acceptable; what is not acceptable is a user flipping variations as you ramp a rollout up (monotonicity is a hard invariant, guaranteed by the bucketing, not by propagation timing).
• Durability: flag definitions and targeting rules are the source of truth and must be durable (a lost flag ruleset breaks every app depending on it). The audit log must be durable (compliance). The streaming fan-out tier is stateless and holds no durable data - it is a conduit, regenerable instantly.
• The real budget is propagation latency and fan-out cost. It is fine for evaluation to be eventually consistent by ~100ms and for the initial fetch to take a few hundred ms; it is not fine to add latency to evaluation, to miss the sub-100ms kill switch, or to let 10M instances hammer a central server. The whole design evaluates locally and propagates by streaming push.

Back-of-the-Envelope Estimation (BoE)

Real numbers with the arithmetic, because they dictate the architecture.

Connected SDK instances (the fan-out target):

~10M concurrent SDK instances holding a live connection to us.
   (Server-side SDKs are long-lived processes; a big customer runs thousands of
    app servers, each an SDK instance. Client SDKs add more but are burstier.)
If each instance POLLED for config every 30s:
   10M / 30 = ~333,000 requests/sec of "give me the latest config", mostly
   returning "no change". Pure waste, and still up to 30s stale on a kill switch.
If each instead holds ONE persistent streaming connection:
   10M idle connections, near-zero bytes/sec until a flag changes.
=> persistent streaming, not polling. The question becomes "can one box hold
   ~1M idle connections?" (yes - this is a C10M / epoll problem), so ~10-20
   streaming nodes hold all 10M connections, each mostly idle.

Flag evaluation volume (why it must be local):

Customers evaluate flags in their own hot paths. Say the aggregate across all
   customers is ~50M flag evaluations/sec (every request in every customer app
   may check several flags).
If each evaluation were a call to US: 50M requests/sec to a flag server. Insane,
   and it would add a network RTT to every one of our customers' requests.
=> evaluation is IN-PROCESS from the cached ruleset. We serve ZERO of these.
   Our load is only (a) initial config fetch and (b) pushing deltas on change.

Change / propagation volume (rare but must be fast and wide):

Flag changes are rare: say ~5,000 changes/day across the whole platform.
   5,000 / 86,400 ≈ ~0.06 changes/sec average. Trivial write rate.
BUT each change must fan out to every SDK in that project+environment.
   A change to a popular flag may touch, say, 500k connected SDK instances.
   The server must push the delta to 500k connections in <100ms.
   500k small messages (~a few KB) = ~a few GB pushed in a 100ms burst, spread
   across the streaming nodes holding those connections.
The cost is not the write (0.06/sec); it is the FAN-OUT BURST per change.

Storage:

Flag definitions + rules: hundreds of thousands of flags * (rules, variations,
   segments) * a few KB each ≈ low single-digit GB. Tiny. Fits comfortably in a
   relational DB, and the per-environment ruleset fits in RAM on every node.
Per-environment config bundle (what an SDK fetches): all flags in one
   environment, compiled. Say a large environment has ~5,000 flags * ~1KB
   ≈ ~5MB uncompressed, ~1MB gzipped. Shippable whole; deltas are bytes.
Audit log: ~5,000 changes/day * 365 * a few KB ≈ a few GB/year. Append-only.
Evaluation/exposure events (optional, for analytics): if emitted, this is the
   only high-volume stream - batched and shipped to an event pipeline, out of
   the critical path (same shape as any event firehose).

Streaming-node capacity (the crux):

Target ~1M persistent connections per streaming node (achievable with epoll/
   async IO and tuned file descriptors; the classic C10M problem, now routine).
   10M connections / 1M per node = ~10 nodes to hold everyone; run ~20 for
   headroom and failure tolerance.
Idle cost per connection is a socket + a few KB of buffers - memory-bound, not
   CPU-bound, while idle. CPU spikes only during a fan-out burst.
On a change: the node looks up which of its connections subscribe to the changed
   environment and writes the delta to each. 500k targets across 20 nodes =
   ~25k writes/node, each a few KB - a sub-100ms burst, easily.

The takeaways: evaluation is a ~50M/sec firehose we must serve zero of by pushing the ruleset into the SDK and evaluating in-process; connections are ~10M persistent streams we hold idle across ~10-20 epoll-based streaming nodes rather than eating 333k polls/sec; changes are rare (0.06/sec) but each is a wide fan-out burst to hundreds of thousands of connections that must complete in under 100ms; and storage is tiny (low GBs of rules) because the data is small config, not user data. The hard parts are therefore: local evaluation + sticky rollout, the streaming fan-out tier that holds 10M connections and pushes deltas in <100ms, the propagation pipeline from editor to streaming node, and graceful degradation so a flag check never breaks a customer’s app.

High-Level Design (HLD)

The architecture splits along the fundamental seam of a flag service: the control/management plane (define and edit flags - rare writes, must be consistent and durable) is separate from the delivery plane (fetch the ruleset once, stream deltas forever, evaluate locally - 10M connections, must be fast and cheap). Config flows outward from a small durable store, through a fan-out tier, to every SDK, which caches it and evaluates in-process. The instant a flag changes, the change propagates through a pub/sub backbone to every streaming node, each of which pushes the delta to its relevant connections.

   MANAGEMENT PLANE (rare writes, consistent, durable)
   ┌──────────┐  edit flag   ┌──────────────┐  write   ┌──────────────────┐
   │ Dev / PM │──rule/toggle─>│ Flag Mgmt API│─────────>│ Flag Store (SQL): │
   │ (web UI) │  /rollout %   │ (validate,   │          │ flags, rules,     │
   └──────────┘               │  version++)  │          │ segments, audit)  │
                              └──────┬───────┘          └─────────┬────────┘
                                     │ publish change event        │ source of truth
                                     ▼                             │
                              ┌──────────────────────────┐         │
                              │  Change Propagation Bus   │         │
                              │  (pub/sub: Redis/Kafka,   │<────────┘ (compiled bundle
                              │   topic per environment)  │            cached by edge)
                              └───────────┬──────────────┘
   ─────────────────────────────────────  │ fan-in change, fan-out to nodes
   DELIVERY PLANE (10M connections, fast, cheap)
                         ┌─────────────────┴──────────────────┐
                         ▼                                     ▼
                ┌──────────────────┐                  ┌──────────────────┐
                │ Streaming Node 1  │   ...  ~20 nodes │ Streaming Node N  │
                │ holds ~1M persist │                  │ holds ~1M persist │
                │ connections; on   │                  │ connections       │
                │ change, pushes    │                  │                   │
                │ delta to subs     │                  │                   │
                └───────┬──────────┘                  └────────┬─────────┘
                        │ SSE / websocket (persistent)          │
            ┌───────────┴───────────┐                 ┌─────────┴──────────┐
            ▼                       ▼                 ▼                    ▼
     ┌─────────────┐        ┌─────────────┐    ┌─────────────┐     ┌─────────────┐
     │  SDK (app   │        │  SDK (app   │    │  SDK (app   │     │  SDK (app   │
     │  server)    │  ...   │  server)    │    │  server)    │ ... │  server)    │
     │ * fetch     │        │ caches      │    │ evaluates   │     │ fails safe  │
     │   ruleset   │        │ ruleset in  │    │ LOCALLY:    │     │ to default  │
     │   (initial) │        │ RAM + disk  │    │ bucket(hash │     │ if we are   │
     │ * stream    │        │             │    │ (ctx+salt)) │     │ unreachable │
     │   deltas    │        │             │    │  microsec   │     │             │
     └─────────────┘        └─────────────┘    └─────────────┘     └─────────────┘

   Initial ruleset fetch is served from a CDN/edge cache (GET /sdk/config),
   not from the Flag Store directly. Streaming holds the connection after.

Edit flow (define/change a flag):

1. A developer edits a flag via the Flag Management API, which validates it (variations well-typed, rules reference real segments, rollout percentages in range), bumps the flag’s version, writes the new state to the Flag Store (relational), and appends an audit record.
2. The API publishes a change event ({environment, flag_key, new_version} plus the compiled delta) to the Change Propagation Bus (a pub/sub system with a topic per environment).

Propagation flow (editor to every SDK, under 100ms):

1. Every Streaming Node subscribes to the propagation bus. When a change event for environment E arrives, each node looks up its locally-held connections that are subscribed to E and writes the delta down each of those persistent connections.
2. The SDK receives the delta over its open stream, applies it to its in-RAM ruleset atomically, and persists the updated ruleset to local disk. No fetch, no poll - the push is the update. End to end: editor save -> bus -> node -> socket write -> SDK apply, in tens of milliseconds.

Bootstrap flow (an SDK starts up):

1. On init, the SDK fetches the current compiled ruleset for its environment: GET /sdk/config with its SDK key, served from a CDN/edge cache (a versioned, cacheable blob) so 10M startups do not hammer the origin. It caches the ruleset in RAM and on local disk.
2. The SDK then opens a persistent streaming connection (SSE or websocket) to a Streaming Node and holds it open, receiving deltas for the life of the process. On reconnect it sends its current version so the node can send a catch-up delta or tell it to refetch.

Evaluation flow (customer code checks a flag - zero network):

1. The customer calls client.variation("new-checkout", context, default). The SDK evaluates locally against its cached ruleset: resolve individual overrides, then segment/attribute rules, then the percentage rollout via bucket(hash(flag_key + context_key + salt)), returning the matched variation. Microseconds, in-process, no I/O.
2. If any step fails (flag unknown, ruleset missing, malformed context), the SDK returns the caller-supplied default and never throws into customer code. Optionally it emits a batched exposure event to an analytics pipeline, off the hot path.

The key insight: evaluation is a pure local function over a cached ruleset, and propagation is a push over a persistent connection held open by a stateless fan-out tier. The management plane keeps a small ruleset consistent and durable; the delivery plane ships it once and streams deltas forever. Nothing on the evaluation path talks to us, and nothing about a change requires polling. That separation is what lets evaluation be free while a kill switch reaches 10M instances in under 100ms.

Component Deep Dive

The hard parts, naive-first then evolved: (1) flag evaluation and sticky percentage rollout - how to evaluate locally and ramp 1% to 100% without ever flipping a user back; (2) real-time propagation - how to get a change to 10M instances in under 100ms without melting a central server (the core challenge); (3) the targeting rule engine - how to express and evaluate “who gets what” fast and locally; (4) graceful degradation and bootstrap - how an SDK stays correct when we are unreachable and how 10M of them start up without a thundering herd.

1. Evaluation and gradual rollout: why a central flag query and stored assignments fail, then local hash bucketing

Naive approach: the SDK calls GET /evaluate?flag=new-checkout&user=u_42 on every flag check; the server looks up the flag, and for a percentage rollout keeps an assignments(user, flag, variation) table so a user stays sticky as you ramp.

Where it breaks:

• A network call per evaluation is fatal. Flag checks sit in the customer’s request path, firing ~50M/sec in aggregate. Routing each to us adds a network RTT to every one of our customers’ requests and asks us to serve 50M req/sec. Both are non-starters; the entire value of flags (cheap, everywhere checks) evaporates.
• Stored assignments explode and race. A percentage rollout over millions of users means millions of assignment rows per flag; bumping 1% to 5% writes more; two concurrent first-checks for the same (user, flag) race to insert. You are maintaining a huge table to remember something a hash can recompute for free.
• Ramp-up reshuffles users. If assignment is a random coin flip stored on first contact, extending the rollout from 1% to 5% has no principled way to keep the original 1% and only add new users - naive re-randomization flips some users out of the treatment, which is exactly the regression you were trying to avoid.
• Offline is impossible. An SDK that must call us to evaluate cannot evaluate during a network blip - so your customer’s app breaks when our network does.

Evolution: ship the ruleset to the SDK and evaluate locally; make percentage rollout a deterministic hash into stable buckets, storing nothing. The core trick, same as consistent assignment anywhere: a good hash of the same input always yields the same output, so stickiness needs no stored state - you recompute the bucket identically every time.

For a percentage rollout of flag "new-checkout" to a context (user) u_42:

  h = hash( "new-checkout" + ":" + "u_42" + ":" + flag_salt )   # xxHash / Murmur
  bucket = h mod 100000                                         # bucket in [0,100000)
  # rollout is 1%  => "on" iff bucket < 1000      (1% of 100000)
  # rollout is 5%  => "on" iff bucket < 5000
  # rollout is 50% => "on" iff bucket < 50000
  variation = (bucket < rollout_pct * 1000) ? "on" : "off"

  => u_42 lands in the SAME bucket every time, on every server, forever, with
     ZERO stored state and ZERO network calls. The rollout is just a threshold.

Monotonic ramp falls out of the threshold. Because the bucket for a user is fixed (it depends only on flag key + user key + salt, never on the rollout percentage), ramping the rollout only raises the threshold. A user with bucket 800 is “on” at 1% (threshold 1000) and stays “on” at 5%, 50%, 100% - the set of “on” users only grows. Nobody who was on turns off as you ramp. This is the whole reason to bucket by hash rather than flip a stored coin: ramp-up is monotonic by construction.

Monotonic ramp, same salt, same buckets, only the threshold moves:
  1%:  "on" iff bucket < 1000      user@800 -> ON,  user@3000 -> off
  5%:  "on" iff bucket < 5000      user@800 -> ON,  user@3000 -> ON
  50%: "on" iff bucket < 50000     both ON
  user@800 is ON at every stage; it NEVER flips back. No stored state, no flips.

Why the per-flag salt: if you hashed user_key alone, a user in a low bucket would be in the “on” group of every flag at low rollout - their inclusion across flags would be perfectly correlated, so two independent 1% rollouts would hit the exact same 1% of users, concentrating all early-rollout risk on one unlucky cohort. A per-flag salt re-randomizes each user’s bucket independently per flag, so being in the 1% of flag A says nothing about flag B. (If you deliberately want correlated rollouts - ramp a family of related flags together - you share a salt on purpose. That is the bucketBy/rollout-key knob.)

Why this is correct at scale: evaluation is O(1) CPU, zero I/O, zero storage - a hash, a modulo, and a threshold compare, over a ruleset already in RAM. 50M evaluations/sec is trivial because each is nanoseconds of local math on data we already shipped, with no shared state to contend on. Determinism gives stickiness and monotonic ramp for free - no locks, no assignment table, no transactions. The only thing the SDK needs from us is the small ruleset, fetched once and streamed thereafter. We replaced a per-evaluation query and a million-row assignment table with a stateless hash - the move that makes evaluation free.

2. Real-time propagation: why polling fails, then a streaming fan-out tier over a pub/sub backbone

This is the hard part the problem is really about: a flag change must reach ~10M SDK instances in under 100ms, and doing so must not overload a central server.

Naive approach: each SDK polls GET /sdk/config every N seconds; if the version changed, it downloads the new ruleset.

Where it breaks:

• Polling is a latency/load tradeoff with no good setting. Poll every 30s and your kill switch is up to 30s slow - unacceptable when a bad feature is burning money. Poll every 1s to get close to real-time and 10M instances generate 10M req/sec, almost all returning “no change” - a self-inflicted DDoS on your origin. There is no interval that is both fast and cheap; polling is structurally wrong for “rare changes, instant delivery.”
• Thundering refetch. Even at a modest interval, a popular flag change makes a large fraction of the 10M instances all refetch the full ruleset in the same window - a correlated load spike on your config origin.
• Wasted bandwidth. Shipping the full ruleset on every change (or every poll that detects one) moves megabytes when the change was a single flag’s single byte.

Evolution: hold a persistent streaming connection per SDK on a stateless fan-out tier, and push only the delta the instant a change is published to a pub/sub backbone. Invert the model: the SDK does not ask “did anything change?” millions of times; it holds one connection and we tell it the moment something does.

Editor saves a change
        │ Flag Mgmt API: write to Flag Store, version++, append audit
        ▼
   PUBLISH {env, flag_key, version, delta} to Change Propagation Bus
        │   (pub/sub; topic/channel PER ENVIRONMENT; e.g. Redis pub/sub or Kafka)
        ▼
   Every Streaming Node is SUBSCRIBED to the bus.
        │ A node receives the change event once.
        ▼
   Node consults its in-memory index: which of MY held connections subscribe to
   this environment?  (a per-env list of connection handles)
        │
        ▼
   Node writes the small DELTA down each of those persistent connections (SSE/ws).
        │
        ▼
   SDK receives delta on its open stream -> applies to in-RAM ruleset atomically
   -> persists to local disk.  End to end: tens of milliseconds.

Why persistent connections + pub/sub hits <100ms cheaply. The change is published once to the bus (0.06 writes/sec average - nothing). The bus fans it out to ~20 streaming nodes (a handful of messages). Each node already holds the relevant connections open, so delivery is just a socket write per connection - no connection setup, no auth, no DB lookup per instance. 500k target connections spread across 20 nodes is ~25k socket writes per node, each a few KB; a node doing async IO flushes that in well under 100ms. The expensive thing in the naive design (10M instances independently discovering and fetching the change) is replaced by one publish and a fan-out of cheap writes down connections that are already open.

Holding 10M connections: the C10M problem, solved by stateless epoll nodes. Each streaming node holds ~1M persistent connections. Idle connections are cheap - a socket plus a few KB of buffers - so a node is memory-bound while idle, not CPU-bound; CPU spikes only during a fan-out burst. Nodes are stateless: they hold no durable data, only a volatile index of (environment -> connection handles). If a node dies, its ~1M SDKs detect the dropped connection and reconnect (with jittered backoff) to another node, which re-indexes them. No data is lost because the node never owned any; the Flag Store and bus are the sources of truth.

Streaming Node internals (stateless, ~1M connections each):
  connections:  fd -> { sdk_key, environment, current_version }
  by_env:       environment -> set<fd>      (the fan-out index)
  on change(env, delta):
     for fd in by_env[env]:  async_write(fd, delta)   # the whole fan-out
  on connect(sdk):  auth sdk_key -> environment; add to maps; send catch-up if
                    sdk.version < current (so a reconnecting SDK never misses a change)
  on disconnect(fd): remove from maps

Deltas, versions, and catch-up. Each environment’s ruleset carries a monotonically increasing version. A change ships as a delta ({flag_key, new_rule, version}), not the whole bundle, so the push is bytes. When an SDK connects (or reconnects after a blip), it sends its last-seen version; the node sends either a targeted catch-up delta (if it can reconstruct the gap) or a signal to refetch the full compiled bundle from the CDN. This guarantees no missed changes across reconnects: the version is the sync point. Because changes are rare, catch-up is cheap and refetches are rare.

Scaling the fan-out and avoiding a hot environment. Most of the load is one environment’s fan-out burst. Since any node only writes to its own held connections, fan-out is naturally parallel across the ~20 nodes - adding nodes adds both connection capacity and fan-out throughput. If one environment is enormous (a single customer with millions of SDK instances), its connections are already spread across all nodes by the connection-balancing load balancer, so no single node owns a disproportionate share of any one environment. The pub/sub bus itself fans out one small message per change to ~20 subscribers - trivial, and shardable by environment if change rate ever grew.

Why this is correct at scale: we turned “10M instances each asking, forever” into “10M connections held idle, told once.” Propagation cost per change is one publish + a parallel burst of cheap socket writes, not a load spike of millions of fetches; latency is tens of ms because the connections are already open and the payload is a delta; and the fan-out tier is stateless and horizontally scalable - more nodes, more connections and more fan-out, with failures healing by reconnect. Streaming inverts the load: the server does near-zero until something changes, then pushes a small delta down connections it already holds - the only shape that is both sub-100ms and cheap at 10M instances.

3. The targeting rule engine: why a pile of if-statements fails, then an ordered rule list with segments evaluated locally

Naive approach: encode each flag’s targeting as ad-hoc server-side logic - a hard-coded sequence of checks per flag - and evaluate it wherever the flag is checked.

Where it breaks:

• Not shippable to the SDK. Hard-coded logic lives in our code; but evaluation must happen in the SDK, which only has data, not our functions. Targeting has to be data (a declarative ruleset) the SDK can interpret, not code only we can run.
• No consistency across languages. Ten SDK languages each re-implementing bespoke per-flag logic guarantees they disagree on edge cases - the same user gets different variations in your Go service and your Node service. Targeting semantics must be one spec, interpreted identically everywhere.
• Unmaintainable and unauditable. “Who is this flag on for?” has no answer you can read off a config; it is buried in imperative branches. Rollback and audit need declarative rules with versions.

Evolution: targeting is a declarative, ordered list of rules over the evaluation context, with reusable segments, compiled into the ruleset and interpreted identically by every SDK. Evaluation is: walk the rules top to bottom, first match wins, else fall through to the default rollout.

Flag "new-checkout" targeting (declarative, shipped to the SDK):
  variations: ["off", "on"]
  rules (ordered, first match wins):
    1. if user.key in [ "u_7", "u_42" ]            -> "on"   # individual overrides
    2. if user in segment "internal-staff"         -> "on"   # segment match
    3. if user.country in ["CA"] AND user.plan=="pro"
                                                   -> rollout("on":5%, "off":95%)
    4. (fallthrough) default rollout               -> rollout("on":1%, "off":99%)
  off_variation: "off"      # served when flag is globally OFF (kill switch)

Evaluation in the SDK (local, microseconds):
  if flag is OFF globally -> return off_variation            # kill switch wins
  for rule in rules:
     if rule.clauses all match context:
        return rule.fixed_variation
             OR bucket(hash(flag_key+context.key+salt)) -> rollout variation
  return default rollout variation

Segments are reusable, named sets of rules (internal-staff = email ends with @us.com OR user.key in [...]), defined once and referenced by many flags. They are part of the shipped ruleset, so segment membership is also evaluated locally. Large list-based segments (a million enterprise user ids) are shipped as a compact set (or a Bloom-filter-backed structure for “big segment” membership) so the SDK can test membership in-process without a lookup to us.

Clauses are the atoms: (attribute, operator, values) - country in [CA, US], email endsWith @us.com, plan == pro, age >= 18. The operator set is fixed and spec’d so every SDK evaluates them identically. The context the customer passes ({key, email, country, plan, custom: {...}}) is matched against clauses purely locally.

Why this is correct at scale: targeting is data, not code, so the exact same ruleset evaluates identically in every language and is fully auditable (“show me the rules” is a config read, and rollback is restoring a prior version). First-match-ordered rules with a fallthrough rollout express the real cases - overrides, segments, attribute targeting, and a percentage ramp for everyone else - and all of it runs in-process in microseconds because the whole ruleset, segments included, is cached in the SDK. Targeting as a declarative, versioned, locally-interpreted ruleset is what lets rich “who gets what” logic live at the edge without a single network call and without ten SDKs disagreeing.

4. Graceful degradation and bootstrap: why a hard dependency and a naive fetch fail, then fail-safe defaults and CDN-fronted bootstrap

Naive approach: the SDK requires a live connection to evaluate; on startup every instance fetches its config straight from the origin config service; a flag check throws if anything is wrong.

Where it breaks:

• We become a hard dependency of every customer app. If a flag check throws when we are unreachable, our outage becomes their outage - the exact opposite of what a flag system should do. A feature flag is best-effort control on top of a product that must keep working.
• Thundering herd on bootstrap. 10M instances starting up (or all reconnecting after a network partition heals) fetching the full ruleset from one origin is a correlated spike that knocks the origin over right when everyone needs it.
• Cold start blocks the app. If the SDK blocks application startup until it has fetched config, a slow fetch stalls the customer’s deploy.

Evolution: fail safe to caller-supplied defaults, bootstrap from a CDN with cached local disk fallback, and connect with jittered backoff. Three rules make us a soft dependency.

1. FAIL SAFE, NEVER THROW.
   client.variation(flag, ctx, default):
      try: evaluate locally from cached ruleset
      catch (no ruleset / unknown flag / bad context): return default
   A flag check ALWAYS returns a usable value - the caller's default if all else
   fails. Our total outage degrades customers to their coded defaults, not down.

2. BOOTSTRAP FROM CDN + LOCAL DISK CACHE.
   On init: GET /sdk/config from a CDN edge (versioned, cacheable blob) - 10M
   startups hit the CDN, not the origin; the origin serves the CDN, not clients.
   If the CDN is unreachable too: load the LAST ruleset from local disk (persisted
   on every delta). An SDK can start fully offline from its last-known-good config.
   Only a brand-new instance with neither CDN nor disk falls back to all-defaults.

3. CONNECT WITH JITTERED BACKOFF.
   After bootstrap, open the streaming connection. On disconnect, reconnect with
   exponential backoff + random jitter so a node failure or partition heal does
   NOT produce a synchronized 10M reconnect stampede. The LB spreads reconnects
   across nodes; jitter spreads them across time.

The streaming connection is additive, not required. Evaluation works the moment the ruleset is cached (from CDN or disk); the stream only keeps it fresh. So a network blip that drops the stream does not stop evaluation - the SDK serves its last ruleset (slightly stale, still correct and sticky) and reconnects in the background, getting a version-based catch-up delta so it never misses the change it slept through. The one hard invariant preserved throughout: a flag check is a pure local function that always returns, so we can be completely down and customers keep running on last-known-good config or their defaults.

Why this is correct at scale: bootstrap load lands on a CDN (built for 10M-client fan-out of a cacheable blob), not the origin; reconnect storms are broken up by jitter + LB spread; and the serving path has no hard dependency on us because evaluation is local and failures fall back to disk then to caller defaults. We are a soft dependency by design - the product keeps working when we do not - which is the only acceptable posture for a system embedded in every one of your customers’ request paths.

API Design & Data Schema

Traffic splits into a low-volume management plane (define/edit flags - humans), a cacheable bootstrap endpoint (SDKs fetch the ruleset once), and a persistent streaming channel (SDKs hold open for deltas). The evaluation “API” is deliberately absent as a network endpoint - it is an SDK-local computation.

REST / RPC endpoints

# --- Management plane: flag CRUD (low volume, human-facing, authenticated user) ---
POST /api/v1/projects/proj_9/environments/production/flags
  Body: { "key":"new-checkout", "type":"boolean",
          "variations":[ {"key":"off","value":false}, {"key":"on","value":true} ],
          "off_variation":"off", "default_rollout":{ "on":1, "off":99 },
          "rules":[
            {"clauses":[{"attr":"key","op":"in","values":["u_7","u_42"]}],
             "variation":"on"},
            {"clauses":[{"attr":"segment","op":"in","values":["internal-staff"]}],
             "variation":"on"} ] }
  -> 201 { "flag_key":"new-checkout", "version":1, "status":"off" }

PATCH /api/v1/.../flags/new-checkout   { "default_rollout":{ "on":5, "off":95 } }  # ramp
  -> 200 { "version":2 }   # only RAISES the rollout threshold; no user flips back.
  # publishes a change event to the propagation bus; SDKs updated in <100ms.

POST  /api/v1/.../flags/new-checkout/kill   { "reason":"error_spike" }
  -> 200 { "version":3, "status":"off" }   # kill switch: off_variation served to all.

GET   /api/v1/.../flags/new-checkout/audit
  -> 200 { "history":[ {"version":2,"by":"dev_3","change":"rollout 1->5","ts":...} ] }

PUT   /api/v1/projects/proj_9/segments/internal-staff
  Body: { "rules":[{"clauses":[{"attr":"email","op":"endsWith","values":["@us.com"]}]}] }
  -> 200   # segments are reusable across flags; shipped in the ruleset.

# --- Bootstrap: SDK fetches the compiled ruleset (cacheable, CDN-fronted) ---
GET  /sdk/config
  Headers: Authorization: <sdk_key>   (scopes to one project+environment)
           If-None-Match: "v1041"     (SDK's current version)
  -> 200 { "version":1041, "flags":[ ...compiled rules, segments, salts... ] }
     or 304 Not Modified   # served from CDN edge; origin rarely touched.

# --- Streaming: SDK holds this open for deltas (the propagation channel) ---
GET  /sdk/stream
  Headers: Authorization: <sdk_key>,  Last-Event-ID: 1041
  -> 200, Content-Type: text/event-stream   (Server-Sent Events; or a websocket)
     ... connection stays open ...
     event: put     data: { "version":1042, "flag":"new-checkout", "rule":{...} }
     event: put     data: { "version":1043, "flag":"banner", "off":true }
  # node pushes a delta the instant a change is published; SDK applies in-place.

# --- Optional: exposure events for analytics (batched, off the hot path) ---
POST /sdk/events   { "events":[ {"flag":"new-checkout","variation":"on",
                                 "context_key":"u_42","ts":...} ] }
  -> 202 Accepted   # fire-and-forget; feeds a separate analytics pipeline.

Evaluation has no runtime endpoint - the SDK computes the variation locally from the cached ruleset. This is the point: the highest-QPS operation (50M/sec) makes zero network calls.

Data store schemas

1. Flag Store - relational (SQL, e.g. Postgres). Small, low-write, must be consistent and durable; the source of truth for flag definitions, rules, segments, and audit. Relational because flags/variations/rules/segments are structured and relational, edited transactionally, and the volume is tiny (low GBs). SQL’s consistency is exactly what the management plane needs and its volume makes relational overhead free.

flags(
  flag_id PK, project_id, environment, key, type, status,     -- status: off/running
  off_variation, default_rollout_json, version, archived,
  created_by, updated_ts,
  UNIQUE(project_id, environment, key),
  INDEX(project_id, environment) )          -- to compile an env's bundle fast
variations(
  variation_id PK, flag_id FK, key, value_json )
rules(
  rule_id PK, flag_id FK, ordinal, clauses_json, variation_or_rollout_json )
                                            -- ordinal = first-match order
segments(
  segment_id PK, project_id, environment, key, rules_json, big_list_ref )
audit_log(
  audit_id PK, project_id, environment, flag_key, version,
  changed_by, change_summary, before_json, after_json, ts )   -- append-only

2. Compiled Ruleset Bundle - a versioned, cacheable blob (CDN object + in-SDK RAM/disk). The management API compiles all flags+segments for one environment into a compact bundle SDKs fetch and cache. Not a DB - a versioned artifact, served from a CDN, applied via streamed deltas thereafter.

{ "version":1041, "environment":"production",
  "segments":[ {"key":"internal-staff","rules":[...]} ],
  "flags":[
    {"key":"new-checkout","salt":"f_nc9","type":"boolean","status":"running",
     "variations":[{"key":"off","value":false},{"key":"on","value":true}],
     "off":"off","rules":[...ordered...],"default_rollout":{"on":1000,"off":99000}} ] }
   (~1MB gzipped for a large env; SDK holds in RAM, persists to disk, evaluates locally)

3. Change Propagation Bus - pub/sub, topic per environment (Redis pub/sub or Kafka). Carries change events from the management API to every streaming node. Not durable storage - a fan-in/fan-out conduit. Low message rate (changes are rare), tiny messages (deltas).

Channel: env:proj_9:production
  { version, flag_key, delta }     # one small message per change
  ~0.06 msgs/sec avg; ~20 subscribers (the streaming nodes). Trivial load.
  (Redis pub/sub for speed; or Kafka per-env topic if replay/durability of the
   change stream is wanted - deltas can be reconstructed from the audit log too.)

4. Streaming Nodes - stateless, in-memory connection state only. Hold ~1M persistent connections each; keep a volatile index environment -> {connection handles} and per-connection current_version. No durable data - if a node dies, SDKs reconnect elsewhere and re-index. This is the C10M fan-out tier.

In-memory per node (volatile, regenerable):
  connections: fd -> { sdk_key, environment, version }
  by_env:      environment -> set<fd>
  (holds no truth; Flag Store + bus are authoritative. Reconnect rebuilds state.)

Why these choices: flag definitions are small, structured, transactional, must be consistent - SQL is exactly right and the tiny volume makes it free. The ruleset bundle is read-by-10M-SDKs, small, cacheable - a versioned CDN blob, not a per-request DB read. Propagation is rare events needing instant wide fan-out - a pub/sub bus, not a queue the SDKs drain. The streaming tier is stateless connection-holding - in-memory only, regenerable by reconnect. And there is no store at all on the evaluation path because evaluation is computed from the cached ruleset, not stored or queried. Each layer gets the mechanism its access pattern demands - SQL for the small consistent truth, a CDN blob for bootstrap fan-out, pub/sub for instant propagation, stateless epoll nodes for the connections - and the hottest operation touches none of them.

Bottlenecks & Scaling

Where it breaks first, in order, and the fix for each:

1. Evaluation on the request path (breaks catastrophically if done wrong). ~50M evaluations/sec in customers’ hot paths - any network call per evaluation is fatal. Fix: evaluation is a local in-SDK computation (bucket(hash(flag_key + context_key + salt)) plus a rule walk) over a cached ruleset - zero I/O, zero stored state, deterministic. We serve none of these. The only network dependency is the small ruleset (fetched once, streamed thereafter). This removes the biggest wall by not building it.

2. Propagation latency and fan-out cost (the core challenge). A change must reach ~10M instances in <100ms without a central server melting. Fix: persistent streaming connections on a stateless fan-out tier + a pub/sub backbone. The change is published once (0.06/sec); ~20 streaming nodes each push a small delta down the connections they already hold. No polling, no refetch storm - one publish and a parallel burst of cheap socket writes. End to end: tens of ms. The server does near-zero until a change happens.

3. Holding 10M persistent connections. Idle connections must be cheap and the tier must survive node loss. Fix: ~1M connections per epoll/async-IO node, ~10-20 nodes. Idle connections are memory-bound (socket + buffers), CPU spikes only on a fan-out burst. Nodes are stateless; on node death SDKs reconnect (jittered backoff) to another node and re-index. No data is lost because nodes own no truth. Add nodes for more connections and more fan-out throughput at once.

4. Bootstrap thundering herd. 10M instances starting up or reconnecting after a partition heals could hammer the config origin. Fix: bootstrap GET /sdk/config is served from a CDN edge (versioned cacheable blob); the origin serves the CDN, not 10M clients. Reconnects use exponential backoff + jitter and spread across nodes via the LB, so there is no synchronized stampede. Local disk cache means a restart can skip the fetch entirely when its version is current.

5. Hot environment / hot flag fan-out. A single huge customer’s environment, or a flag on everything, concentrates the fan-out burst. Fix: that environment’s connections are already spread across all nodes by the connection-balancing LB, so no node owns a disproportionate share of any one environment’s fan-out; the burst parallelizes. The pub/sub message is one small delta to ~20 subscribers regardless of how many connections it ultimately reaches. If change rate on one environment ever grew, shard the bus by environment.

6. Missed changes across reconnects. An SDK whose stream dropped during a change must not stay stale. Fix: every environment ruleset carries a monotonic version; on (re)connect the SDK sends its last-seen version and the node sends a catch-up delta or a signal to refetch the bundle. The version is the sync point, so a change that happened during a disconnect is delivered on reconnect - no missed kill switches.

7. Large list-based segments. A “big segment” of a million user ids cannot be a network lookup per evaluation (back to the fatal per-eval call). Fix: ship big segments to the SDK as a compact membership structure (a sorted set or Bloom-filter-backed index with an authoritative fallback only for the rare Bloom positive that needs confirming). Membership is tested in-process; only pathologically huge segments use a bounded “big segment store” the SDK queries sparingly and caches, never on the common path.

8. Shard keys, stated plainly. Flag Store -> keyed by (project_id, environment) for compiling an environment’s bundle; tiny, effectively unsharded relational. Propagation bus -> topic/channel per environment (each node subscribes to the environments it holds connections for; shardable by environment if rate grows). Streaming tier -> connections balanced across nodes by the LB, indexed in-node by environment; there is no per-user shard because connections are stateless and interchangeable. Evaluation -> no shard key, because nothing is stored per user; the hash is the bucketing of users into rollout slots, computed at the edge. The environment is the unit of propagation; the user is bucketed by a stateless hash and never stored.

9. Single points of failure and graceful degradation. The management API and Flag Store are the only stateful truth; they are replicated (primary + replicas) and, being low-write, easy to keep available - and if they are fully down, no new changes propagate but every SDK keeps evaluating from its last ruleset. The CDN, bus, and streaming nodes are all stateless/regenerable. The critical resilience property: a flag check is a pure local function that always returns - last-cached ruleset, then local disk, then the caller’s default - so our total outage degrades customers to their coded defaults, never to an error or a hang. No component’s failure can blank or break a customer’s app.

10. Multi-region. The management plane runs centrally (edits are rare); the compiled bundle replicates to every region’s CDN edge so bootstrap is local everywhere. Streaming nodes run in every region, each subscribed to the global propagation bus (or a regionally-replicated mirror of it), so a change published once fans out to nodes in all regions and reaches SDKs locally - still under 100ms because the socket writes are region-local; only the one publish crosses regions. A user evaluated in any region gets the identical variation because evaluation is a pure hash of the same flag_key + context_key + salt - determinism makes evaluation automatically consistent across regions with zero per-user replication. The hardest multi-region problem (consistent per-user state everywhere) simply does not exist here; only the small ruleset replicates.

Wrap-Up

The trade-offs that define this design:

• Evaluate locally, do not query. Flag evaluation is a pure local function over a cached ruleset - a hash into stable buckets plus a rule walk - so there is no per-evaluation network call (which would add RTT to every customer request and ask us to serve 50M/sec) and no assignment table (which would be millions of rows per flag). A central evaluation endpoint was rejected on latency, load, and offline-correctness all at once.
• Propagate by push, not poll. Changes stream down persistent connections held open by a stateless fan-out tier over a pub/sub backbone, so a change is published once and reaches ~10M instances in tens of milliseconds via a parallel burst of cheap socket writes - instead of 10M instances polling forever (wasteful and still stale) or refetching in a correlated spike. This is the core win: the server does near-zero until a flag changes, then pushes a delta down connections it already holds.
• Sticky, monotonic rollout by hash, not by stored coin flip. Percentage rollout is a deterministic bucket and a moving threshold, so ramping 1% to 100% only adds users and never flips one back - stickiness and monotonicity for free, with no stored state and automatic cross-region consistency. Per-flag salts keep independent rollouts from concentrating risk on the same cohort.
• Targeting as declarative, versioned data. Rules and segments are shipped to the SDK and interpreted identically everywhere, so rich “who gets what” logic runs at the edge in microseconds, stays auditable and rollback-able, and never lets ten SDK languages disagree. Hard-coded server-side logic was rejected because evaluation must happen in the SDK from data, not from our functions.
• Soft dependency over hard. A flag check is a pure local function that always returns - last-cached ruleset, then disk, then the caller’s default - and bootstrap rides a CDN with jittered reconnects, so our total outage degrades customers to their coded defaults rather than taking their apps down. The engineering goes into a guaranteed-working default, never into strong consistency the problem does not need.

One-line summary: a feature flag platform that evaluates flags locally inside ~10M SDK instances by a stateless per-flag-salted hash over a cached, declaratively-targeted ruleset (zero network calls, sticky and monotonic rollouts, consistent across regions for free), bootstraps from a CDN, and propagates every change in under 100ms by publishing one delta to a pub/sub bus that a stateless fan-out tier pushes down the millions of persistent connections it already holds - so evaluation is free, rollouts never flip a user, and the kill switch reaches everyone almost instantly without a central server ever carrying the evaluation or polling load.