In a monolith, “call the inventory module” is a function call. In a fleet of 800 microservices spread over 15,000 instances that scale up and down, crash, get rescheduled, and move IPs every few minutes, “call the inventory service” is a surprisingly hard question: which of the inventory instances are alive right now, and what are their addresses? Hardcoding IPs is a non-starter - an autoscaler churns them constantly. A static load balancer in front of every service is operationally heavy and hides the health signal you actually need. The clean answer is a service discovery and registry system: a live, queryable map from a logical service name to the set of currently-healthy instances backing it.

This system sits on the hot path of nearly every inter-service call, directly or through a sidecar, so it has to be fast and almost always available. But its data is life-and-death for traffic routing: send requests to an instance that died two seconds ago and you get a wall of connection-refused errors and cascading retries. So the real design tension is a CAP tension. During a network partition, do you refuse to answer (and halt traffic) or hand back a possibly-stale instance list (and risk routing to a dead node)? Most of this design is about answering that question deliberately rather than by accident.

Let me do this the way I would in an interview: state the requirements, size it, sketch the architecture, then take the genuinely hard parts - the registry’s consistency model, health propagation via gossip versus Raft, and the read path at scale - from the naive version, break them, and evolve.

Functional Requirements (FR)

In scope:

  • Service registration. An instance (or an agent on its behalf) registers itself under a logical service name with its address, port, and metadata (version, zone, tags, weight). It can deregister on clean shutdown.
  • Health checking. The system continuously determines whether each registered instance is healthy, and only healthy instances are returned to callers. Health can come from active probes (HTTP/TCP/gRPC checks), passive signals, or instance-reported heartbeats.
  • Service discovery / lookup. A caller asks “give me the healthy instances of service inventory in zone us-east-1a” and gets back the current set, filtered by health and metadata. Supported over DNS (inventory.service.consul) and a richer HTTP/gRPC API.
  • Change notification / watch. A caller (or a sidecar proxy) subscribes to a service and is pushed updates when the healthy set changes - an instance joins, dies, or drains - without polling.
  • Key-value / config store. A small strongly-consistent KV store for feature flags, leader-election keys, and dynamic config, riding on the same cluster.
  • Multi-datacenter awareness. Discovery is primarily local to a datacenter, with the ability to resolve services in a remote DC when asked.

Explicitly out of scope (say this so you control scope):

  • Being the load balancer. We return the healthy instance set and let the client (or its sidecar) pick one and balance across them. We do not proxy the actual request traffic. That is a service mesh data plane; we are the control plane that feeds it.
  • General-purpose database. Values are tiny (an instance record is a few hundred bytes), the whole registry fits comfortably in memory. No blobs, no big rows, no analytical queries.
  • Deep application-level health. We check liveness and basic readiness. “Is the business logic correct” is the application’s job; we answer “can this instance take traffic right now.”
  • Cross-DC strong consistency by default. Each datacenter is its own failure domain. We do not run one global consensus group across the WAN on the hot path.

The single decision that drives everything: the registry has two very different kinds of data, and they want opposite consistency models. The catalog - which services exist and who is allowed to register - wants strong consistency (one source of truth, no split registrations). The health/liveness of thousands of churning instances wants high availability and fast propagation, and can tolerate being a second or two stale. Trying to force both through one strongly-consistent log is the classic mistake. The good designs split them.

Non-Functional Requirements (NFR)

  • Availability: very high on the read/discovery path (target 99.99%+). If discovery is down, nothing can call anything - it is a cluster-wide outage amplifier. Lookups must survive the loss of individual registry nodes and keep answering even during partitions, from a possibly-stale-but-recent view.
  • Consistency: tiered. The catalog and KV store are linearizable (via consensus). Health and membership are eventually consistent with a tight convergence bound - a dead instance should disappear from results within a few seconds everywhere.
  • Propagation latency: seconds, not minutes. A node death or a deregistration should be reflected cluster-wide within roughly 1-5 seconds at p99. This is the headline number; stale routing is the main pain.
  • Lookup latency: single-digit milliseconds. Discovery is on the request hot path. Served from a local cache / local agent, it should be sub-millisecond; a cache miss to a server, a few ms.
  • Scale: ~15,000 instances across ~800 services in one datacenter, churning at maybe hundreds of register/deregister/health-change events per second steady-state, with bursts into the thousands during a deploy or an autoscaling event. Tens of thousands of lookups per second, overwhelmingly cache hits.
  • Durability: the catalog and KV store survive the loss of a minority of server nodes (fsynced consensus log). Ephemeral health state does not need disk durability - it is reconstructed from live signals.
  • Fault model: crash-stop and network partitions, not Byzantine. Nodes crash, pause, or get partitioned; they do not lie.

Back-of-the-Envelope Estimation (BoE)

Let me ground the numbers, because they justify the split architecture.

Fleet and registry size:

Services                          ~800
Instances (total)                 ~15,000
Avg instances per service         ~19
Record per instance               ~300 bytes (id, ip, port, tags, health, zone)
Catalog + health state            15,000 * 300B  ≈ 4.5 MB
With KV config + indexes          well under ~100 MB

The entire live registry is a few megabytes. It lives in RAM on every node. There is no disk-seek read path. This is why it is a coordination/control-plane system, not a database: the data is tiny, the agreement and propagation are the expensive parts.

Health-check load (the real pressure):

Instances                         15,000
Health check interval             ~10 sec per instance
If the servers probed directly    15,000 / 10  = 1,500 checks/sec, centrally

1,500 active probes per second from a central cluster, each a network round trip to a different instance, plus the fan-out of connections - this is the number that breaks the naive centralized design and forces health checking to the edge. More on that in the deep dive.

Change / event rate:

Steady-state churn               ~hundreds of events/sec
                                 (deploys, autoscale, crashes, flaps)
Deploy / autoscale burst         thousands of events/sec for tens of seconds
Each event                       must propagate to all nodes that care

Lookup / discovery load:

Services make inter-service calls constantly, but discovery is cached.
Logical calls needing a lookup    ~tens of thousands/sec cluster-wide
Cache hit ratio (local agent)     > 99%  (instance set changes slowly vs call rate)
Actual lookups reaching a server  low thousands/sec

The key insight from the numbers: lookups vastly outnumber changes, and changes are what is hard. A given service’s instance set might change a few times a minute, but it is looked up thousands of times a second. So we cache aggressively at the edge and spend our engineering on propagating changes fast and reliably rather than on raw lookup throughput.

Watch fan-out:

A popular service (e.g. "auth")   called by hundreds of other services
Watchers on its instance set      up to ~hundreds of sidecars/agents
On one instance of auth dying     push a delta to all of them, within seconds

A single instance death on a popular upstream means a fan-out of a small delta to hundreds of watchers, fast. Fan-out of changes, not lookup volume, is the scaling problem we design for.

High-Level Design (HLD)

The architecture is a classic agent + server split, which is exactly how Consul is built, and the reasoning generalizes to etcd/ZooKeeper-based setups.

  • A small server cluster (odd number, 3 or 5) runs a Raft consensus group. It is the authoritative, strongly-consistent store for the catalog (registered services) and the KV store. This is the CP core.
  • A local agent runs on every host in the fleet (a sidecar/daemon). The agent registers the local instances, runs their health checks locally, and participates in a gossip membership protocol (SWIM-style) with all other agents and servers. Agents cache discovery data and answer local lookups.
  • Gossip spreads membership and health liveness across the whole fleet quickly and partition-tolerantly (the AP layer). Raft keeps the catalog authoritative (the CP layer). They are deliberately different protocols for the two different consistency needs.
  Host 1                         Host 2                        Host N (~thousands)
 ┌───────────────┐             ┌───────────────┐             ┌───────────────┐
 │ svc instance  │             │ svc instance  │             │ svc instance  │
 │     + AGENT   │             │     + AGENT   │             │     + AGENT   │
 │ - local health│             │ - local health│             │ - local health│
 │   checks      │             │   checks      │             │   checks      │
 │ - gossip peer │◄───gossip───►│ - gossip peer │◄──gossip───►│ - gossip peer │
 │ - lookup cache│   (SWIM)    │ - lookup cache│   (SWIM)    │ - lookup cache│
 └──────┬────────┘             └──────┬────────┘             └──────┬────────┘
        │ register / watch / lookup (RPC to a server)               │
        └───────────────┬───────────────────────────┬──────────────┘
                        ▼                           ▼
             ┌──────────────────── SERVER CLUSTER (3-5 nodes) ─────────────┐
             │  ┌──────────┐      ┌──────────┐      ┌──────────┐           │
             │  │ Follower │◄────►│  LEADER  │◄────►│ Follower │  Raft      │
             │  │ catalog  │ Raft │ catalog  │ Raft │ catalog  │  (CP:      │
             │  │ + KV(RAM)│      │ + KV(RAM)│      │ + KV(RAM)│  catalog,  │
             │  └────┬─────┘      └────┬─────┘      └────┬─────┘  KV)       │
             │    ┌──▼──┐           ┌──▼──┐           ┌──▼──┐               │
             │    │ WAL │           │ WAL │           │ WAL │  fsync+snap   │
             │    └─────┘           └─────┘           └─────┘               │
             │  Servers are also gossip members, so they learn health too.  │
             └──────────────────────────────────────────────────────────────┘

   Catalog writes -> Raft quorum (strong).   Health/liveness -> gossip (fast, AP).

Registration flow (write path):

  1. An instance starts. Its local agent registers it: a write RPC to a server with name, address, port, tags, and the health-check definition.
  2. The server leader appends the registration to the Raft log, replicates to a quorum of followers who fsync, commits, and applies it to the in-memory catalog. The service now officially exists with this instance.
  3. The agent begins running the instance’s health checks locally and marks the instance healthy/unhealthy in its own view, which gossip then spreads.

Health flow (the fast, AP path):

  1. Each agent runs the checks for its own local instances only - an HTTP GET to /health, a TCP connect, a gRPC health probe, every few seconds. No central prober.
  2. When an instance’s health changes, the agent updates its local state and gossips the change: it tells a few random peers, who tell a few random peers, and the update floods the fleet in O(log N) rounds - typically converging across thousands of nodes in a second or two.
  3. Agents (and servers) merge incoming gossip into their local view. A dead or failed instance is marked unhealthy and filtered out of lookups everywhere, fast, without touching Raft.

Discovery flow (read path):

  1. A caller resolves a service via its local agent (DNS query or HTTP/gRPC API). The agent answers from its local cache - the merged view of catalog (from the servers) plus health (from gossip). Sub-millisecond, no network hop off-host in the common case.
  2. The agent returns only instances that are both in the catalog and currently healthy, filtered by requested tags/zone.
  3. The caller (or its sidecar) load-balances across the returned set.

Watch flow:

  1. A sidecar watches a service. The agent maintains a long-lived watch; when its merged view of that service’s healthy set changes (from gossip or a catalog update), it pushes the delta to the sidecar.
  2. Catalog changes propagate from servers to agents via blocking queries / streaming; health changes propagate via gossip. Either can trigger a watch notification.

Component Deep Dive

Three hard parts, each taken naive-first: (1) the consistency split - why one store cannot do both jobs; (2) health propagation - centralized polling versus gossip versus Raft, and why gossip wins for liveness; (3) the read path and staleness at scale. I will break each before fixing it.

1. The registry store: why one consistency model is not enough

Everything hinges on how you store “who is registered and who is healthy.”

Naive: one strongly-consistent store for everything

Put the entire registry - catalog and live health - in a single Raft/ZooKeeper-style consensus store. Every registration, every deregistration, and every health-state change is a write that goes through consensus. Reads are linearizable. Beautifully simple, one source of truth.

Where it breaks:

  • Health churn overwhelms the consensus log. Health flaps. An instance under load fails a check, recovers, fails again. With 15,000 instances and checks every few seconds, you have a steady stream of health transitions. Every one becomes a consensus write - a quorum round trip plus fsync. The consensus group tops out around 10-20K writes/sec for the whole cluster, and you are burning that ceiling on ephemeral liveness flaps that do not even need durability. The catalog writes you actually care about get starved.
  • A partition halts health updates. Consensus is CP: the minority side of a partition cannot commit writes. So during a partition, instances on the minority side cannot update their health, and the majority side cannot learn that a minority instance died. Health, which is exactly the thing you most need to keep flowing during a partition, freezes. You route traffic to dead nodes because the registry could not record their death.
  • Propagation is bounded by consensus latency and read patterns. Even when healthy, pushing every health change through the leader and then out to all readers is slower and more centralized than it needs to be.

Forcing health through consensus is using a correctness hammer on an availability problem. Health does not need linearizability; it needs speed and partition-tolerance.

Better: split the catalog (CP) from health/liveness (AP)

Two stores, two models, matched to the data:

DataExampleConsistencyMechanismWhy
Catalog“service auth exists, these instances are registered”Strong (linearizable)Raft consensusOne truth, no split registration, durable
KV / configfeature flags, leader-election keysStrong (linearizable)Raft consensusCorrectness-critical, low write rate
Health / liveness“instance 3f2a is failing its check”Eventual, fast-convergingGossip (SWIM)High churn, must survive partitions, staleness tolerable
Membership“agent on host 812 is alive”Eventual, fast-convergingGossip (SWIM)Same
  • The catalog changes slowly (an instance registers once at startup, deregisters once at shutdown) and must be authoritative, so consensus is the right cost - low write rate, strong guarantees.
  • Health changes constantly and must keep flowing during partitions, so gossip is the right tool - no quorum, no single point, floods the fleet in O(log N) rounds.
  • A lookup merges the two: the authoritative catalog set, intersected with the currently-healthy set from gossip. An instance is returned only if it is both registered (catalog) and healthy (gossip).

This split is the single most important design decision. It is why the system can be both strongly consistent about what exists and highly available about what is alive. Consul does exactly this: Raft for the catalog/KV, Serf/gossip for membership and health.

2. Health propagation: centralized checks vs gossip vs Raft

This is the focus of the problem - propagate health changes globally within seconds while surviving partitions.

Naive A: central health checker polling every instance

The server cluster actively probes all 15,000 instances on a schedule.

Where it breaks:

  • Probe load and fan-out. 15,000 instances at a 10s interval is 1,500 probes/sec from a handful of servers, each a fresh connection to a different host across the network. The servers become a probe factory, and the probe traffic itself is a large, bursty load that competes with the consensus and lookup work.
  • It does not scale with the fleet. Double the instances, double the central probe load. The thing that should be cheap (checking a local process) is made expensive by doing it from far away.
  • Network blind spots. The server’s view of an instance’s health is really “can the server reach it,” which conflates instance health with server-to-instance network health. A partition between servers and a healthy instance looks like instance death.
  • Single point of pressure. If the central checker falls behind, the whole fleet’s health goes stale at once.

Naive B: push every health change through Raft

Covered above - health churn blows the consensus write budget and freezes during partitions. Rejected.

Better: local checks + gossip (SWIM)

Two moves together.

Move 1 - push health checks to the edge. Each agent checks only its own local instances. A check becomes a loopback or same-host call - cheap, accurate (it is genuinely testing the local process, not the network to it), and perfectly scalable: 15,000 instances means 15,000 agents each doing a trivial amount of work. The central servers run zero instance probes.

Move 2 - spread liveness with gossip (SWIM protocol). Agents form a gossip membership group. SWIM (Scalable Weakly-consistent Infection-style Membership) gives two things:

  • Failure detection. Each agent periodically picks a random peer and pings it. No ack? It asks k other random peers to ping that peer indirectly (to rule out a one-off network blip between just those two). If none can reach it, the peer is marked suspect, then dead after a timeout. Indirect probing is what keeps the false-positive rate low - you do not declare a node dead because of a single bad link.
  • Dissemination. State changes (a node joined, a node is suspect/dead, an instance went unhealthy) ride along on the ping/ack messages (piggybacked) and flood the cluster infection-style. Each node tells a few, who tell a few. Convergence is O(log N) rounds; across thousands of nodes that is a second or two.
SWIM failure detection (why it does not false-positive):

  A pings B directly ────X  (no ack within timeout)
  A asks C, D, E to ping B indirectly:
       C ─ping─► B ─ack─► C   (B is actually alive, A->B link was bad)
  -> A learns B is alive, does NOT mark it dead.   False positive avoided.

  If C, D, E all also fail to reach B:
  -> A marks B SUSPECT, gossips "B suspect"
  -> after suspicion timeout with no refute, B -> DEAD, gossiped fleet-wide
  -> B's instances drop out of every lookup within ~1-2s

Why gossip beats both naive options for liveness:

  • Partition-tolerant. There is no quorum and no leader on the health path. Each side of a partition keeps detecting and spreading health within itself. When the partition heals, states merge. Health never fully freezes.
  • Scales with the fleet, not against it. Gossip load per node is roughly constant (a fixed number of messages per round) regardless of cluster size; the fleet does the work of spreading its own health.
  • Fast. O(log N) convergence hits the seconds target even at tens of thousands of nodes.
  • No central bottleneck. No node is special for health.

The cost / trade-off, stated honestly: gossip is eventually consistent. For a brief window (sub-second to a couple seconds) different nodes can disagree about whether an instance is healthy. That is acceptable for routing - a client that gets one stale dead instance in its set fails over to the others and the staleness clears in a second. It would not be acceptable for the catalog, which is exactly why the catalog is on Raft and only liveness is on gossip.

When would you use Raft for health instead? If you had a small cluster (tens of nodes, like the database replicas in an etcd-coordinated system) and wanted strongly-consistent membership with linearizable “is this node in or out” semantics - e.g. consensus-group membership changes. For a 15,000-node application fleet with high churn, gossip is the right tool; for a 5-node database cluster’s own membership, Raft is. Know which regime you are in. The headline: gossip for big, churny, availability-first fleets; Raft for small, correctness-first membership.

3. The read path and staleness at scale

Discovery is on the hot path of every inter-service call. It must be fast and must degrade gracefully.

Naive: every lookup is an RPC to the server cluster

Each caller, on each call, asks a server “who backs auth?”

Where it breaks:

  • Lookup volume crushes the servers. Tens of thousands of lookups/sec funneled to 3-5 servers, each doing a linearizable read (a leader round trip for freshness). The servers become the bottleneck for the entire fleet’s ability to make calls.
  • Latency on the hot path. Every call now pays a network round trip to a server before it can even start. That is latency tax on literally everything.
  • Server outage = total outage. If the servers are briefly unavailable (leader election, partition), no one can look anything up and the whole mesh stalls, even though all the instances are fine.

Better: local agent cache + stale-but-available reads + watches

  • Answer lookups from the local agent’s cache. The agent already holds the merged catalog+health view (catalog pulled from servers and kept fresh via streaming/blocking queries; health arriving via gossip). A lookup is a local, in-memory, sub-millisecond read. No off-host hop on the common path. This is why lookup volume stops being a server problem - it is absorbed at the edge where it scales with the fleet.
  • Serve stale during server unavailability. Because the agent caches, a brief server outage does not stop lookups - the agent returns its last-known catalog, and gossip keeps health fresh independently. The read path stays up even when the CP core is mid-election. This is the deliberate availability choice: a slightly stale instance list beats no answer at all for routing.
  • Watches / blocking queries instead of polling. An agent does not poll the servers for catalog changes; it issues a blocking query (long-poll with a version/index) that returns only when the data changes past the version the agent already has. One outstanding query per watched service, server responds on change. This collapses “tens of thousands of polls/sec” into “a response only when something actually changes,” which is the same edge-triggered idea that keeps watch systems scalable.
  • DNS and API interfaces. Expose discovery as DNS (for zero-code clients: auth.service.dc1 resolves to healthy instance IPs, short TTL) and as a richer HTTP/gRPC API (for clients that want tags, weights, and streaming updates). DNS is easy to adopt but coarse (A records, TTL-bound staleness); the API gives health-filtered, metadata-rich, push-updated results. Offer both.

Staleness budget, made explicit. Total staleness on the read path = gossip convergence (1-2s for health) + catalog propagation (blocking-query round trip, tens of ms) + any client-side DNS TTL. Keep DNS TTLs low (a few seconds) and prefer the streaming API for sidecars that need the tightest freshness. The design target - a dead instance gone from results within a few seconds everywhere - is met by gossip, not by the catalog path.

API Design & Data Schema

API

# --- Registration (write, hits the CP catalog via Raft) ---
PUT /v1/agent/service/register
  body: {
    "id": "auth-7f3a", "name": "auth", "address": "10.2.4.11", "port": 8443,
    "tags": ["v2", "zone:us-east-1a", "weight:100"],
    "check": { "type": "http", "path": "/healthz", "interval": "10s", "timeout": "2s",
               "deregister_after": "60s" }
  }
  200: { "registered": true }

DELETE /v1/agent/service/deregister/{service_id}
  200: { "deregistered": true }

# --- Discovery (read, served from local agent cache) ---
GET /v1/health/service/{name}?passing=true&tag=zone:us-east-1a
  200: {
    "service": "auth",
    "instances": [
      { "id": "auth-7f3a", "address": "10.2.4.11", "port": 8443,
        "health": "passing", "tags": ["v2","zone:us-east-1a","weight:100"] }
    ],
    "index": 88213          # version for blocking queries
  }

# DNS interface (zero-code clients)
#   dig auth.service.dc1   ->  A records of healthy instances, low TTL

# --- Watch (blocking query: returns only when index advances) ---
GET /v1/health/service/{name}?index=88213&wait=30s
  200 (on change): { ...updated instance set..., "index": 88240 }
  200 (on timeout): same index, client re-issues   # long-poll, edge-triggered

# --- KV / config (strongly consistent, Raft) ---
PUT /v1/kv/{key}?cas=<version>     # compare-and-set
  body: <= 512 KB value
  200: { "ok": true, "version": 41 }
GET /v1/kv/{key}?consistent=true
  200: { "value": "...", "version": 41 }

# --- Health report (agent -> local, gossiped; not a server write) ---
# Internal: agent updates local check status, gossip disseminates. No Raft write.

The passing=true filter and the index for blocking queries are the load-bearing parts: health-filtered results plus edge-triggered updates are what make discovery both correct and cheap.

Data / storage schema

Three stores, each matched to its consistency need.

Catalog (in Raft-replicated state machine, in memory on every server, fsynced log on disk):

service {
  name            string      # "auth", logical service name (indexed)
  id              string      # "auth-7f3a", unique instance id (primary key)
  address         string      # "10.2.4.11"
  port            int
  tags            set<string> # ["v2","zone:us-east-1a","weight:100"]
  meta            map         # arbitrary small metadata
  check_def       struct      # how to health-check this instance
  modify_index    int64       # Raft log index of last change (for blocking queries)
}
# Index: primary on id; secondary on name (lookups are by name);
#        secondary on tags/zone for filtered discovery.

Health / membership (in memory, per node, NOT durable, maintained by gossip):

member {
  node_id         string      # agent/host id
  status          enum        # alive | suspect | dead
  incarnation     int64       # refutation counter (see below), monotonically increasing
  instances       map<id, health>   # local instances this node owns + their status
  last_seen       timestamp   # local clock, for suspicion timeouts
}
# health = passing | warning | critical
# Reconstructed from live gossip, never read from disk.

The incarnation number is the anti-flap mechanism in SWIM: when a node is falsely suspected, it refutes by broadcasting a higher incarnation for itself, which supersedes the stale “suspect” gossip. This is how a live node that got briefly mislabeled clears its own name without a central authority.

KV / config (Raft-replicated, same engine as catalog):

kv { key string (PK), value bytes (<=512KB), version int64, modify_index int64 }

On disk, per server:

WAL:  append-only Raft log of catalog + KV ops; fsynced before commit. Durability.
Snapshot: periodic dump of catalog+KV state + last_included_index; truncates WAL.
# Health/membership is NEVER persisted - it is live-signal-derived and
# would be stale/wrong after a restart anyway. On restart a node rejoins gossip.

Store choice: a custom in-memory state machine - Raft log for catalog/KV, gossip-maintained membership table for health - not SQL and not a generic NoSQL store. Reasons:

  • The catalog needs linearizable, durable, totally-ordered writes - a consensus log, not a single-writer SQL DB that would itself be the SPOF.
  • Health needs partition-tolerant, high-churn, fast-converging dissemination - gossip, which no database offers.
  • The dataset is tiny and hot (a few MB, all RAM reads). A disk-oriented DB’s strengths are irrelevant and its lack of ephemerality/gossip is disqualifying.

This is exactly why Consul, etcd, and ZooKeeper are their own category rather than “just use Postgres.”

Bottlenecks & Scaling

Where it breaks first, in order, and the fix for each.

1. Catalog write throughput on the single Raft group (breaks first if you misroute health to it). All catalog/KV writes serialize through one leader and one fsynced log - a hard ceiling near 10-20K writes/sec. The primary defense is architectural: keep health off this path entirely (gossip), so only true register/deregister events hit Raft, which is a tiny fraction of total events. Beyond that: batch and pipeline at the leader (group many ops into one AppendEntries + fsync), and for very large multi-DC estates, shard by datacenter - each DC runs its own Raft group and only federates for cross-DC lookups. Never run one global Raft group across the WAN on the hot path.

2. Lookup volume on the servers. Tens of thousands of lookups/sec would bury 3-5 servers. Fix: serve lookups from the local agent cache, so lookup load scales with the fleet (each host answers its own) and the servers see only cache-fill blocking queries, which fire only on change. This is the main reason the agent exists.

3. Gossip convergence and message load at very large scale. At tens of thousands of nodes, naive all-to-all gossip would be noisy, and convergence could stretch. Fix: SWIM’s design already bounds per-node load (fixed fan-out per round). Tune the probe interval and indirect-probe count for your size. For huge or multi-region fleets, use gossip with network topology awareness (Serf’s LAN vs WAN pools): a dense LAN gossip pool per DC for fast local convergence, and a sparser WAN gossip pool of just the servers for cross-DC membership, so WAN links are not flooded.

4. Flapping instances creating health churn. An overloaded instance that fails and recovers repeatedly generates a storm of state changes and watch notifications. Fix: debounce at the edge - require N consecutive failures before marking critical, and a cooldown before marking healthy again. Use the SWIM suspect phase and incarnation refutation to avoid promoting a transient blip to a death. Optionally apply deregister_after so a long-critical instance is removed from the catalog, stopping the flap entirely.

5. False-positive failure detection under network blips. A single bad link should not evict a healthy instance and cause a needless failover. Fix: indirect probing (ask k peers to confirm before declaring suspect) plus the suspect -> dead timeout with a refutation window. This is SWIM’s core correctness feature and directly reduces spurious churn.

6. Server unavailability during leader election. If the Raft leader dies, catalog writes stall for an election timeout. Fix: this is a bounded, deliberate CP pause (150ms-1s, randomized to avoid split votes). Critically, the read/discovery path stays up throughout because agents serve from cache and health keeps flowing over gossip. Registrations queue briefly; lookups do not stop. Keep the ensemble at 5 to survive 2 server failures.

7. Stale cache on the read path. An agent could return an instance that died moments ago. Fix: this is the accepted trade - staleness is bounded to the gossip convergence window (1-2s). Mitigate downstream with client-side health awareness: callers that hit a dead instance evict it locally and retry another, so one stale entry costs a single fast failover, not an outage. Keep DNS TTLs low and prefer streaming watches for the tightest freshness.

8. The partition decision, made explicit. During a partition between agents and servers: agents keep serving discovery from cache and keep gossiping health within their partition - availability-first, because halting all routing is worse than mild staleness. During a partition within the server cluster: the minority side cannot commit catalog writes (CP), so registrations on the minority side fail, but lookups everywhere still work. We chose CP for the catalog (small, must-be-correct) and AP for discovery+health (hot path, must-stay-up). That split is the whole point - it lets us refuse to be wrong about what exists while refusing to be unavailable about what is alive.

Wrap-Up

The trade-offs that define this design:

  • Two consistency models, deliberately. The catalog and KV are CP (Raft) - one authoritative truth, durable, no split registration. Health and membership are AP (gossip/SWIM) - fast, partition-tolerant, eventually consistent. Forcing both through one store is the central mistake; splitting them is the central insight.
  • Health checks at the edge, not the center. Each agent checks only its own local instances, so health checking scales with the fleet instead of against it, and a check measures the instance rather than the network path to a central prober.
  • Gossip for propagation, consensus for authority. Gossip floods liveness in O(log N) rounds with no leader and no quorum, hitting the seconds target and surviving partitions; Raft keeps the catalog correct. Right tool for each job, and knowing which regime you are in (big churny fleet vs small correctness-critical membership) is the senior judgment call.
  • Cache at the edge, serve stale when you must. Lookups are answered locally and sub-millisecond; during a server outage the agent returns its last-known set rather than nothing. A slightly stale instance list beats no answer on the routing hot path, and client-side failover makes the staleness cheap.
  • Availability-first for discovery, consistency-first for the catalog. The partition behavior is chosen, not accidental: never be wrong about what exists, never be unavailable about what is alive.

One-line summary: a per-host agent runs local health checks and gossips liveness (SWIM) for fast, partition-tolerant propagation, while a small Raft server cluster holds the authoritative catalog and KV store - lookups are served from the agent’s merged, cached, health-filtered view so discovery stays sub-millisecond and available even when the consensus core is mid-election, with a dead instance disappearing from results everywhere within a couple of seconds.