“Nearby Friends” looks like the Uber problem wearing a different hat - a map, some dots, find the ones near me. It is not. Uber asks “out of 5 million strangers moving around, who is the single nearest one,” and answers it with a spatial nearest-neighbour index. Nearby Friends asks a different and in some ways nastier question: “out of my friends specifically, which ones are within 5km of me right now, and keep that answer live as both of us walk around.” The set you care about is not “everyone near this point” - it is the intersection of two things that both change every few seconds: the social graph (who is my friend, who opted in) and live geography (who is physically close). The hard part is the fan-out: when I move, which friends need to be told, and when a friend moves, do I need to know? Do that naively for 50M concurrently-moving people each with hundreds of friends and you generate billions of pointless distance checks a second.

That single question - maintaining a live, mutually-consented, distance-filtered projection of a social graph onto physical space, as both the graph membership and the positions change continuously - is the spine of this design. Everything hard hangs off it: how you ingest 50M location streams without melting a database, how you decide who to notify without scanning every friendship on every ping, how you route a location update from my phone to my friend’s phone when the two of us are connected to different WebSocket servers, and how you enforce “friends only, opt-in” so a location never leaks to someone who is not allowed to see it. Let me build it properly.

Functional Requirements (FR)

In scope:

  • Broadcast my location. While I have the feature enabled and the app in the foreground, my phone streams my position (lat, lng) every few seconds.
  • See nearby friends live. I open the “Nearby Friends” view and see the subset of my friends who are (a) sharing their location with me, and (b) within a radius (default 5km), each with their approximate distance, updating in real time as they and I move.
  • Real-time updates. When a friend crosses into or out of my radius, or moves within it, I see it within a couple of seconds, not on a manual refresh.
  • Opt-in and per-friend visibility. Location sharing is off by default. A user explicitly enables it, and sharing is mutual and directional: I only see friends who share with me, and they only see me if I share with them. I can pause sharing, or share with a subset of friends.
  • Presence. A friend who closes the app or disables sharing disappears from my view promptly.

Explicitly out of scope (say this so you own the scope):

  • The friendship system itself. Friend requests, accept/reject, and the social graph’s write path are a separate service. I consume “is A a friend of B” and “does A share location with B” as lookups against a friend-graph service; I do not design friend request flows.
  • Yelp’s places / business search. The “Yelp” half - reviews, business listings, POI proximity search over static places - is a different, read-heavy geospatial problem (indexing static points by S2/geohash for a nearby-places search). That is well-trodden ground; here I focus on the genuinely distinctive part, Nearby Friends, which is about moving people you have a relationship with, not static businesses.
  • Historical location / timeline. We do not build a “where was my friend at 3pm” history. Location is ephemeral and live-only; storing history is a privacy liability and a separate product.
  • Maps rendering and routing. Tiles and turn-by-turn are client/CDN concerns.
  • Messaging, notifications infra. We hand “friend X is now nearby” to a notification service; we do not build push infra.

The one decision that drives everything: this is a write-heavy, fan-out-heavy system where the unit of work is not “find the nearest point” but “deliver my movement to exactly the right small set of people, filtered by a live social graph and a distance threshold, and vice versa.” Unlike Uber, there is no scan for a global nearest neighbour and no strongly-consistent assignment; the entire difficulty is targeted, privacy-scoped, real-time fan-out at scale. That is why the pub/sub delivery layer and the who-to-notify filter, not a spatial index alone, are the heart of the design.

Non-Functional Requirements (NFR)

  • Scale: 200M total users, ~50M concurrently online with the feature active, each streaming a location ping every few seconds. Average ~200 friends per user; a heavy tail up to a few thousand.
  • Latency: a friend’s movement should surface on my screen within ~2s end to end (their ping to my screen). The “who is nearby” view on open should populate in under ~500ms.
  • Availability: 99.9%+ on the location and delivery path. This is a social feature, not life-critical; a brief gap where a friend’s dot freezes for a few seconds and then jumps is acceptable. Degrade gracefully - a dropped ping self-heals on the next one.
  • Consistency: eventual consistency everywhere. A friend’s position being 5s stale is fine. There is no “assign exactly one” operation, so we never need strong consistency on the hot path. The one thing that must be correct, not just eventually consistent, is privacy: we must never show a location to someone not authorized, even transiently. A stale position is fine; a stale permission that leaks is not - so permission revocation must propagate fast and fail closed.
  • Durability: live location state is ephemeral and non-durable by design - it lives seconds, and a lost ping is re-sent. Privacy settings and the friend graph are durable and authoritative. We deliberately do not durably store the stream of positions.
  • Freshness: a friend’s tracked position should be at most one ping interval (~5s) stale.

Back-of-the-Envelope Estimation (BoE)

Real numbers with the arithmetic, because they dictate the architecture.

Location updates (the write firehose):

50M concurrent users, each pinging every ~10s while the view is open
= 50,000,000 / 10
= 5,000,000 location writes/sec average
Peak (2x, evening social hours) ≈ 10,000,000 writes/sec

That 5M-10M writes/sec is the number that kills a naive design, exactly as in Uber - a sustained firehose of tiny updates to a hot dataset that is overwritten seconds later.

The fan-out (the number that actually defines this problem):

Naive fan-out: on each ping, notify every online friend.
Avg friends = 200. Fraction online = 50M / 200M = 25%.
Online friends per user ≈ 200 * 0.25 = 50.
Deliveries/sec if we notify all online friends =
    5,000,000 pings/sec * 50 = 250,000,000 deliveries/sec.

250M message deliveries/sec is absurd - and almost all of it is waste, because the overwhelming majority of a person’s friends are in a different city and will never be within 5km. The whole game is cutting that 250M/sec down to the tiny fraction that is actually a friend who is actually nearby.

With a geohash pre-filter (only pair friends who share/adjacent cells):
For a given user, online friends within a ~10km pre-filter box are
    typically 0-3, occasionally more in a home city.
Real "nearby" deliveries/sec ≈ 5,000,000 * ~2 = ~10,000,000 deliveries/sec.

So the pre-filter takes us from 250M/sec of blind fan-out to ~10M/sec of useful deliveries - a 25x cut, and the single most important optimization in the design.

Storage:

Live location state (in memory, overwritten in place):
  per user: user_id 8B, lat/lng 16B, updated_at 8B, cell 8B, status 1B
          ≈ 40B (round to ~80B with overhead)
  50M concurrent * 80B ≈ 4 GB.  Fits in RAM across a small fleet.

Friend graph (durable, read on the hot path):
  200M users * 200 friends * (8B friend_id + a share-flag byte)
          ≈ 200M * 200 * 10B = 400 GB.  Sharded KV, cached hot.

Privacy / share settings:
  Directional share edges ≈ same order as friend edges, a few hundred GB,
  but the "does A share with B" answer is cached alongside the friend list.

Bandwidth:

Ingress: 10M pings/sec (peak) * ~100B (payload + framing) ≈ 1 GB/sec inbound.
Egress: ~10M useful deliveries/sec * ~80B ≈ 0.8 GB/sec, spread across
        the WebSocket fleet and edge-terminated.

The takeaways: a 5M-10M writes/sec firehose that must not touch a durable DB; a live index of only ~4GB that fits in memory; a fan-out that is 250M/sec if done blindly and ~10M/sec if geo-pre-filtered; and a 400GB friend graph that must be read (from cache) on the ingest path to know who to notify. The fan-out filter and the pub/sub delivery layer, not storage, are the design drivers.

High-Level Design (HLD)

The architecture separates three planes: a location plane (a high-write, in-memory position store fed by a streaming ingest path), a delivery plane (WebSocket servers holding user connections plus a pub/sub bus that routes a movement to the right friends’ connections), and a graph/privacy plane (the authoritative friend + share settings, cached hot next to the ingest path so “who may see this ping” is an O(1) lookup). A user’s ping flows into the location plane, is matched against their friend graph, geo-filtered, and delivered through the pub/sub bus to their nearby friends.

   ┌──────────────┐      location ping / ~10s        ┌──────────────┐
   │  User A phone │ ───────────────────────────────▶ │  User B phone │
   │ (WSS conn)    │ ◀─────────────────────────────── │ (WSS conn)    │
   └──────┬────────┘   "friend nearby" delivery        └──────┬───────┘
          │                                                   │
   ┌──────▼───────────────┐                           ┌───────▼───────────┐
   │  WebSocket Server 1   │   ... N edge servers ...  │ WebSocket Server K │
   │ (holds A's conn)      │                           │ (holds B's conn)   │
   └──────┬────────────────┘                           └───────▲───────────┘
          │ ping                                              │ deliver
   ┌──────▼───────────────┐                                   │
   │  Ingest / Location    │                                   │
   │  Service              │──── who-to-notify ───────────────┐│
   │  ┌─────────────────┐  │                                  ││
   │  │ In-mem location  │  │   ┌────────────────────┐        ││
   │  │ store (user->pos,│  │   │ Friend+Privacy cache│        ││
   │  │  by user shard)  │  │   │ (user -> [friend,    │        ││
   │  └─────────────────┘  │   │  shares_with?] )     │        ││
   └──────┬────────────────┘   └─────────┬──────────┘         ││
          │ publish(user_A moved)         │ read              ││
   ┌──────▼───────────────────────────────▼──────────────────▼┴┐
   │            Pub/Sub Bus (Redis / sharded, channel=user_id)   │
   │  publish to channel:A  -> fans to subscribers (A's friends  │
   │  who are online and pre-filtered to nearby cells)           │
   └──────┬──────────────────────────────────────────────────────┘
          │
   ┌──────▼─────────┐   ┌──────────────────┐   ┌────────────────────┐
   │ Friend Graph    │   │ Privacy / Share   │   │  Presence (online / │
   │ Store (sharded  │   │ Settings (sharded │   │  offline, TTL       │
   │ KV, by user_id) │   │ KV, directional)  │   │  heartbeat)         │
   └─────────────────┘   └──────────────────┘   └────────────────────┘

Location update flow:

  1. User A’s phone holds a persistent WebSocket to an edge WebSocket Server and pushes {lat, lng, ts} every ~10s.
  2. The WS server forwards the ping to the Ingest / Location Service, which updates A’s live position in the in-memory location store (overwrite in place, compute A’s geohash cell) and looks up A’s friend + privacy list from the hot cache.
  3. The service computes the notify set: friends who (a) share with A directionally, (b) are online, (c) sit in A’s cell or an adjacent cell (the geo pre-filter). For each, it does a final exact-distance check against 5km.
  4. It publishes A’s new position to the pub/sub channel channel:A. The nearby, authorized, online friends are subscribed to that channel; the bus routes the message to whichever WS servers hold those friends’ connections.
  5. Each friend’s WS server pushes the update down that friend’s socket. Nothing durable is written on the hot path.

Opening the Nearby Friends view (subscription setup):

  1. When A opens the view (or comes online), A’s client tells its WS server. The service loads A’s friend/share list, and for each friend who shares-with-A, subscribes A to that friend’s channel (so A receives their movements) and ensures A publishes on A’s own channel (so they receive A’s).
  2. It also does an initial pull: query the location store for those friends’ current positions, geo-filter to 5km, and return the starting snapshot so the view populates immediately rather than waiting for the next ping.

Presence / teardown: a heartbeat marks A online with a short TTL; when A backgrounds the app or the socket drops, the TTL lapses (or an explicit “going offline” frame arrives), A’s channel goes quiet, and A is unsubscribed from friends’ channels. Friends see A’s dot vanish.

The key insight: the location plane is memory + stream and eventually consistent; delivery is a targeted pub/sub fan-out scoped by a live friend graph and a geohash pre-filter; and the only thing that must be strictly correct is the privacy filter, which we evaluate on every publish and fail closed. We never scan the world and never take a lock - we just publish each movement to precisely the small set allowed to see it.

Component Deep Dive

The hard parts, naive-first then evolved: (1) ingesting the location firehose, (2) the who-to-notify fan-out - the core problem, (3) routing a movement across a distributed WebSocket fleet, (4) friends-only opt-in privacy.

1. Ingesting 5M-10M location updates/sec

Naive approach: write each ping to a database row. The phone does UPDATE user_location SET lat=?, lng=?, ts=? WHERE user_id=? every 10 seconds.

Where it breaks:

  • Write throughput. 5M-10M UPDATEs/sec against any relational store is hopeless - each is a row write, index update, WAL append, and replication event, all on data whose useful lifetime is 10 seconds.
  • Write amplification for nothing. You durably persist, fsync, and replicate positions you overwrite seconds later. Every byte of that durability is wasted.
  • HTTP overhead. A fresh HTTP request every 10s for 50M users means constant connection setup, TLS, and auth - request/response is the wrong shape for a continuous stream.

First evolution: write to Redis instead, over HTTP. Better - Redis can absorb the writes and there is no durability tax - but you have coupled ingest directly to the store (a slow store backs up onto phones), you still pay HTTP per ping, and a single Redis is a hot spot at millions of writes/sec.

The answer: persistent connections at the edge, a streaming shock absorber, and the live state in memory.

  • WebSocket at the edge. Each phone holds one WebSocket to a WS server and streams frames. No per-ping setup; the same socket carries pings up and “friend nearby” deliveries down, which is exactly what a bidirectional social-location feature needs.
  • Kafka keyed by user_id as a shock absorber. WS servers publish pings to a partitioned Kafka topic keyed by user_id, guaranteeing per-user ordering (a stale position never overtakes a fresh one) and absorbing bursts. If the Location Service falls behind, pings queue in the log instead of overwhelming it or blocking phones. (In the lowest-latency variant the WS server calls the Location Service directly and uses Kafka only for the durable-ish presence/graph events; either way the principle is decoupling ingest from index update.)
  • In-memory location store, sharded by user_id. Consumers apply pings to a sharded in-memory map user_id -> {lat, lng, cell, ts}. The whole live set is ~4GB, so it fits in RAM across a handful of nodes. An update is an O(1) overwrite plus a cell recompute, not a B-tree write.
Phone --WSS--> WS Server --produce--> Kafka(topic=locations, key=user_id)
Kafka --consume--> Location Service shard:
    store[user_id] = {lat, lng, ts}
    cell_new = geohash(lat, lng, precision6)
    if cell_new != store[user_id].cell:
        cellIndex[cell_old].remove(user_id)   # optional secondary index
        cellIndex[cell_new].add(user_id)
        store[user_id].cell = cell_new
    -> then run the who-to-notify step (deep dive 2)

This state is not durable, deliberately - a lost position self-heals on the next ping, and a crashed Location Service node rebuilds its shard by replaying the last minute of Kafka (retention need only be a couple of minutes for this purpose). We trade durability we do not need for the throughput we absolutely need - the same bargain as Uber’s location plane.

2. The who-to-notify fan-out (the core problem)

Given that user A just moved, who must be told? The answer is the intersection of three sets, and computing it cheaply, 5M times a second, is the whole design.

Naive approach: on each ping, scan A’s friends and compute distance to each. Load A’s 200 friends, for each fetch their current position, compute haversine, if within 5km and they share with A, deliver.

Where it breaks:

  • Fan-out blowup. 5M pings/sec * 200 friends = 1 billion friend-lookups and haversine computations/sec, and 250M/sec of deliveries even after filtering to online - and almost all of those friends are in another city, so it is pure waste (see BoE).
  • Repeated work. A and B are friends. A’s ping computes distance(A,B); a second later B’s ping recomputes distance(B,A). Every pair is evaluated from both ends, doubling the work.
  • Celebrity tail. A user with 5,000 friends turns one ping into a 5,000-way fan-out. A handful of such users can hot-spot a shard.

First evolution: pub/sub per user - stop scanning, let subscription do the work. Give each user a channel channel:{user_id}. When a user comes online and opens the view, they subscribe to the channels of all friends who share with them. On each ping, the Location Service simply PUBLISH channel:A {pos}; the pub/sub layer delivers to whoever is subscribed. This flips the model: instead of the publisher scanning friends on every ping, the subscription graph is built once when a user comes online and reused for thousands of pings.

But two problems remain, and they are why pub/sub alone is not the finished answer:

  • You still deliver to far-away friends. If A subscribes to all 50 online friends’ channels, A receives 50 friends’ pings every 10s even though 48 of them are in other cities. The receiver computes distance and discards - work done, bytes sent, for nothing. That is the 250M/sec problem still lurking, just moved to the subscriber side.
  • Subscription churn. With 50M users each subscribing to dozens of channels, the subscription set is huge and must be maintained as people go online/offline.

The answer: a geohash pre-filter that narrows delivery to friends who could plausibly be within 5km, layered on top of pub/sub. The insight: two friends can only be “nearby” if their geohash cells are the same or adjacent. So we do not deliver a movement to a friend unless the two are geographically co-located, and we make that check cheap.

Two ways to combine geo and pub/sub, and I will use the first as the primary with the second as an optimization:

(a) Publish-side geo filter. On A’s ping, the Location Service already knows A’s cell. It intersects A’s share-with-A friend list with the set of users currently in A’s cell or its 8 neighbours (a bounded lookup on the in-memory cellIndex), and publishes A’s position only to that small intersection - typically 0-3 people. Concretely, A publishes to per-user channels of just the nearby friends, or to a cell-scoped channel that only nearby friends subscribe to.

on_ping(A, pos):
    A.cell = geohash6(pos)
    nearby_users = union(cellIndex[c] for c in [A.cell] + neighbours(A.cell))
    friends = friendCache[A]                # {friend_id: shares_with_A?}
    targets = [u for u in nearby_users
                 if u in friends and friends[u].shares_with_A and online(u)
                 and haversine(pos, store[u]) <= 5km]
    for t in targets:
        publish(channel = t, msg = {from: A, pos})   # deliver to t's socket

The intersection nearby_users ∩ friends is the trick: nearby_users is small (people physically in a ~1km cell cluster), friends is a hashset, so the intersection is cheap and returns only the people who are both near A and friends of A. This replaces “scan 200 friends” with “scan the handful of people in my cell and keep the ones who are friends” - and in a sparse area that handful is tiny; in a dense area we cap it (see bottlenecks).

(b) Subscription-side cell channels (the pull-and-resubscribe model). Alternatively, model it as: each online user subscribes to the channel of their current geohash cell, and publishes their position to that cell channel. When A moves into cell X, A subscribes to cell:X; everyone publishing in cell:X reaches A, who filters to “is this a friend who shares with me, within 5km.” As A walks across a boundary, A unsubscribes from the old cell channel and subscribes to the new one. This bounds each user’s subscriptions to ~1 cell channel (plus neighbours) instead of one-per-friend, which scales better for users with many friends - at the cost of receiving non-friends’ updates in a dense cell (so you filter, and cells must be small enough that the cell population is bounded).

Approach Work per ping Handles many-friends Handles dense areas Notes
Scan all friends O(friends) badly (celebrity) fine 1B lookups/sec, mostly waste
Pub/sub per friend channel O(1) publish badly (many subs) fine still delivers to far friends
Pub/sub + publish-side geo filter O(cell pop) well needs cap on hot cells primary design
Cell-channel subscribe O(1) publish well needs small cells + filter bounds subs, filters non-friends

Avoiding double work. Because delivery is directional (A publishes A’s movement; B publishes B’s), each pair’s distance is naturally evaluated once per mover. We do not need a symmetric “pair” table - A moving notifies B, B moving notifies A, and both are cheap cell-scoped intersections. The “enter/leave 5km” transitions are computed by comparing the current distance against the previous state the receiver holds for that friend, so we can emit a distinct “friend entered range” event rather than just position updates.

The elegant part: we turn “who among my friends is near me” from a per-ping friend scan into a per-ping intersection of a tiny cell population with a cached friend hashset, delivered through a subscription layer built once per session. The geohash pre-filter is what collapses 250M/sec of blind fan-out into ~10M/sec of useful delivery.

3. Routing a movement across a distributed WebSocket fleet

A publishes on WS server 1; A’s nearby friend B is connected to WS server K. How does A’s movement reach B’s socket?

Naive approach: one big WebSocket server (or sticky routing so friends share a server). Put all connections on one machine, or hash users to servers such that friends land together.

Where it breaks:

  • One server cannot hold 50M sockets. A single box tops out at ~1M connections; you need a fleet of dozens to hundreds, so friends are inevitably scattered across servers.
  • You cannot co-locate friends. The friend graph is dense and overlapping - A is friends with people who are friends with disjoint others. There is no partition of users that keeps all friend pairs on the same server. So cross-server delivery is unavoidable.

The answer: a pub/sub bus that decouples “who published” from “which server holds the subscriber.”

  • A channel registry maps user_id -> the WS server holding that user's socket, kept in a fast store (Redis) with a TTL heartbeat. When B connects to server K, K registers conn:B -> K.
  • Delivery via Redis Pub/Sub (or a sharded message bus). When the Location Service wants to deliver A’s movement to B, it publishes to channel:B. Server K, which holds B’s socket, is subscribed to the channels of the users it serves; it receives the message and pushes it down B’s socket. A does not need to know where B is - the bus routes it.
B connects to WS server K:
    registry.set(conn:B -> K, TTL=30s, refreshed by heartbeat)
    K subscribes to bus channel:B   (K listens for anything addressed to B)

deliver A's movement to B:
    bus.publish(channel:B, {from:A, pos})
    -> Redis routes to K (the subscriber) -> K writes to B's WebSocket
  • Sharding the bus. A single Redis pub/sub instance cannot carry 10M messages/sec. Shard the bus by hash(target_user_id) across many Redis nodes (or use a purpose-built fan-out service). Each WS server subscribes, on each of the bus shards, to the channels of the users it holds. Delivery for target B goes to bus_shard[hash(B)], which only the servers holding B-like users listen on.
  • Reconnect / failover. WS connections are stateless relative to this registry: if server K dies, its users’ sockets drop, phones reconnect (with backoff to avoid a thundering herd) to a new server, which re-registers them and re-subscribes. In the gap, a few of that user’s friend updates are missed - acceptable, they self-heal on the next ping.

This is the same lesson as any chat/presence system: at fleet scale you cannot co-locate related connections, so you route messages through a pub/sub bus keyed by the recipient, and keep a registry of where each recipient currently lives.

4. Friends-only opt-in privacy (the one thing that must be correct)

Naive approach: filter visibility on the client, or check friendship once at subscribe time. Send all nearby users to the client and let the app show only friends; or verify “is B a friend of A who shares with A” only when A opens the view.

Where it breaks:

  • Client-side filtering leaks. If the server ships a non-friend’s location and the client is supposed to hide it, a modified client or a packet capture sees the leaked position. Privacy filtering must be server-side and authoritative.
  • Stale permissions. If we check “does B share with A” only at subscribe time and B later revokes, A keeps receiving B’s location because the subscription persists. Revocation must take effect within seconds and fail closed.
  • Directionality and asymmetry. Sharing is directional: B may share with A but A not with B. And it is per-friend or per-list (“share with close friends only”). A single “is friend” boolean is not enough.

The fix: evaluate a directional share permission on every publish, from a hot-cached authoritative source, with fast fail-closed revocation.

  • Directional share edges. The authoritative model is a set of directional edges: share(from=B, to=A) = true means B allows A to see B’s location. Stored in the privacy service, sharded by the sharing user. The friend cache next to the Location Service holds, per user, the set {friend_id -> shares_with_me?} so the who-to-notify step (deep dive 2) can filter in O(1) without a network hop.
  • Evaluate on publish, not just on subscribe. The publish-side filter checks share(from=A, to=target) for each candidate target at publish time, using the cached edge. So even if a subscription lingers, a movement is not delivered unless the current permission allows it. Subscriptions are a routing optimization; the permission check is the gate.
  • Fail closed on cache miss or uncertainty. If the share status for a pair is not in cache or is ambiguous, we do not deliver and we fetch the authoritative answer - the default is “do not show,” never “show and check later.” A missed delivery is a frozen dot for a second; a wrongly-delivered location is a privacy breach.
  • Fast revocation. When B revokes sharing with A (or unfriends, or goes “invisible”), the privacy service emits an invalidation on a control stream that (a) evicts the share(B->A) entry from the friend cache and (b) tells B’s channel machinery to stop delivering to A and pushes A a “friend went offline” so B’s dot disappears. Because the permission is re-checked on every publish against the now-updated cache, the next ping (within ~10s) is already filtered; the explicit invalidation makes it near-instant.
  • Coarsening as defense in depth. We can round displayed positions to a coarser precision (for example, show “within 200m” rather than an exact point) so even authorized friends do not get pinpoint coordinates, reducing the blast radius of any leak. Exact distance is computed server-side; the client gets an approximate location and a distance bucket.

The point: position data is eventually consistent and disposable, but the permission that gates it is safety-critical - so we make the permission check a cheap, cached, per-publish gate that fails closed and invalidates fast, and we never trust the client to hide anything.

API Design & Data Schema

The location path is streamed frames over WebSocket; settings and the initial snapshot are REST.

Location streaming (phone <-> backend)

// persistent WebSocket, one frame every ~10s while the view is open
{ "type":"LOCATION", "lat":19.0821, "lng":72.7411, "ts":1752... }
// server -> phone: a friend's movement or a range transition
{ "type":"FRIEND_UPDATE", "friend_id":"u_77", "approx_lat":19.081,
  "approx_lng":72.741, "distance_bucket":"<1km", "ts":1752... }
{ "type":"FRIEND_ENTERED_RANGE", "friend_id":"u_77", "distance_m":820 }
{ "type":"FRIEND_LEFT_RANGE",    "friend_id":"u_77" }
{ "type":"FRIEND_OFFLINE",       "friend_id":"u_77" }

REST (client-facing)

POST /api/v1/nearby/enable
  body: { share_scope: "all" | "list:close_friends" | "off" }
  -> { status: "sharing_enabled", scope: "all" }

GET /api/v1/nearby/friends?radius_m=5000
  -> initial snapshot on opening the view
  -> { friends: [
        { friend_id:"u_77", name:"Riya", approx_lat, approx_lng,
          distance_m:820, last_seen_s:4 }, ... ] }

POST /api/v1/nearby/share
  body: { friend_id:"u_77", allow:false }   // revoke sharing with one friend
  -> { friend_id:"u_77", shares_with_them:false }

POST /api/v1/nearby/pause
  -> { status: "paused" }                    // go invisible immediately

Data stores - the right tool per plane

1. Live location store - in-memory, sharded by user_id, NOT a database. user_id -> {lat, lng, cell, ts, status}, ~4GB, rebuildable from ~1 minute of the Kafka location topic. Deliberately non-durable because the data lives seconds. Optionally a secondary cell -> set<user_id> index in the same memory for the publish-side geo intersection.

In-memory (per user shard):
  loc[user_id] = { lat, lng, cell(geohash6), ts, status }
  cellIndex[cell] = Set<user_id>            # for cell-population intersection
  Shard by: hash(user_id)  -> spreads the write firehose evenly

2. Friend + privacy cache - in-memory / Redis next to the Location Service. Per user, the list of friends and the directional share flags, so the who-to-notify filter is O(1) with no network hop. Read-through from the authoritative graph/privacy stores; invalidated by a control stream on friend/share changes.

friendCache[user_id] = { friend_id -> { shares_with_me: bool } , ... }
  invalidation: control-stream event on friend add/remove or share change
  fail-closed: a miss defaults to "not shared" until refreshed

3. Friend graph store - sharded NoSQL KV / wide-column (Cassandra / a graph store), by user_id. The adjacency list is read-heavy and huge (~400GB); a KV/wide-column store scales reads horizontally and shards cleanly on user_id. No cross-user transaction is needed, so relational buys nothing.

Table: friendships          (partition key = user_id)
  user_id     BIGINT   PARTITION KEY
  friend_id   BIGINT   CLUSTERING KEY
  created_at  TIMESTAMP
  Query: all friends of user_id = one partition read.
  Shard by: user_id (a user's whole friend list co-located).

4. Privacy / share settings - sharded KV, directional edges, by sharing user. Authoritative source of “does B allow A.” Durable and consistent; changes emit invalidations.

Table: share_edges          (partition key = from_user_id)
  from_user_id BIGINT  PARTITION KEY   -- the person sharing
  to_user_id   BIGINT  CLUSTERING KEY  -- the person allowed to see
  scope        STRING                  -- "all" | list id
  updated_at   TIMESTAMP
  Query: does from share with to = one point read (cached).

5. Presence + connection registry - Redis with TTL heartbeat. conn:{user_id} -> ws_server_id and online:{user_id} with a ~30s TTL refreshed by heartbeat. Ephemeral; drives routing and “friend offline.”

6. Settings / profile - relational, cached. Slow-changing account data (name, avatar, default scope). Modest volume, read through a cache.

Why NoSQL/in-memory throughout the hot path and no relational store on it: like Uber’s location plane and unlike its trips, Nearby Friends has no transactional entity per request - no fare, no “assign exactly one,” no ledger. Every hot-path operation is a read of static-ish graph data (cached) or an overwrite of ephemeral location (in memory). The only durable, authoritative data is the friend graph and share settings, which are read-mostly and shard cleanly by user. That is why the design leans on in-memory stores, sharded KV, and a pub/sub bus, and why eventual consistency is acceptable everywhere except the fail-closed privacy gate.

Bottlenecks & Scaling

Where it breaks first, in order, and the fix for each:

1. The location write firehose (breaks first). 5M-10M writes/sec will destroy any durable DB. Fix: never write locations to a durable store on the hot path. Stream through Kafka keyed by user_id into an in-memory, user-sharded location store. Accept non-durability - a lost ping self-heals; a crashed shard rebuilds from a minute of Kafka retention.

2. Blind fan-out (the defining bottleneck). Notifying all online friends is 250M deliveries/sec, almost all waste. Fix: the geohash publish-side pre-filter - intersect the small cell population around the mover with the mover’s cached friend set - cuts delivery to only friends who could be within 5km, ~10M useful deliveries/sec, a 25x reduction. The pre-filter is the single most important optimization.

3. Celebrity / high-degree users. A user with thousands of friends turns one ping into a huge fan-out. Fix: fan-out is already bounded by the cell population, not the friend count, because we only deliver to friends physically nearby - a celebrity’s thousands of friends are still mostly far away. For the residual (a popular person in a dense cell with many nearby friends), cap deliveries per ping and prioritize closest/most-recently-interacted friends; the rest see a slightly delayed update on a later ping.

4. Hot cells - stadiums, concerts, festivals. Thousands of friends packed into one geohash cell make the cell-population intersection expensive and the fan-out large. Fix: adaptive cell precision (use a finer geohash level in dense areas so a cell never holds tens of thousands of users), sub-shard hot cells onto dedicated nodes, and cap + sample the delivered set (you do not need sub-second updates for 500 nearby friends; a bounded, prioritized subset is fine). Rate-limit per-user update frequency in dense contexts.

5. Friend-graph reads on the ingest path. Looking up the friend + share list on every ping would hammer the graph store. Fix: the hot friend+privacy cache next to the Location Service serves those reads from memory; the authoritative store is touched only on cache miss or invalidation. A user’s friend list is small and slow-changing, so it caches extremely well.

6. Cross-server delivery scale. 10M deliveries/sec across a fleet where friends are scattered. Fix: a sharded pub/sub bus keyed by recipient plus a connection registry (user -> ws server); each WS server subscribes only to the channels of the users it holds. Delivery routes to the right bus shard and only the relevant servers receive it - no broadcast.

7. Presence flapping and reconnect storms. Mobile clients drop and reconnect constantly; a server failure reconnects millions at once. Fix: TTL heartbeats with hysteresis (do not mark offline on the first missed beat), exponential backoff with jitter on reconnect to avoid a thundering herd, and rebuild subscriptions lazily on reconnect. Missed updates during a blip self-heal on the next ping.

8. Privacy correctness under change (the safety bottleneck). A revoked or unfriended permission must not leak location. Fix: evaluate the directional share edge on every publish from the hot cache, fail closed on any miss or ambiguity, and invalidate fast via a control stream so revocation takes effect within seconds. Never filter on the client; coarsen displayed coordinates as defense in depth.

9. The initial-snapshot thundering herd. Everyone opening the view at once (evening) triggers many friend-list + position pulls. Fix: the snapshot is a bounded in-memory read (cached friend list + in-memory positions, geo-filtered), not a scan; cache the friend list, and coalesce the snapshot with the subscription setup so opening the view is one cheap operation.

10. Single points of failure and multi-region. Fix: WS servers, Location Service shards, and the bus are stateless relative to Kafka/registry and rebuild from the log or on reconnect. Kafka, Redis, and the graph/privacy KV stores are replicated. Deploy the stack per geographic region and route users to their home region, so the latency-critical ping-to-delivery loop stays local - and since two friends who are within 5km are by definition in the same region, cross-region delivery is only needed for the rare traveling friend, handled by cross-region subscription forwarding. No single box whose loss stops the feature; at worst a shard rebuilds while a few dots freeze briefly.

Wrap-Up

The trade-offs that define this design:

  • In-memory streaming location plane over a durable database. We refuse durability for location because it lives seconds - trading “we might lose a ping” (self-healing on the next one) for the only architecture that survives a 5M-10M writes/sec firehose, keeping it entirely off any relational store.
  • Targeted pub/sub fan-out over a per-ping friend scan. The subscription graph is built once per session and reused for thousands of pings, so a movement is a publish, not a scan - and delivery routes through a recipient-keyed bus because friends cannot be co-located on one server.
  • Geohash pre-filter over blind delivery. Intersecting the tiny cell population around a mover with their cached friend set collapses 250M/sec of blind fan-out into ~10M/sec of useful delivery - the one optimization that makes the whole thing affordable, and the thing that distinguishes this from “nearest neighbour.”
  • Eventual consistency for position, strict fail-closed correctness for permission. Everything about where a friend is can be seconds stale; the permission to see them cannot leak, so we gate every publish on a cached directional share edge that fails closed and invalidates fast, and we never trust the client to hide anything.
  • Bound work by physical locality, not social degree. Fan-out is capped by cell population, not friend count, so celebrities and dense venues degrade gracefully via adaptive cell precision, capping, and sampling rather than melting a shard.

One-line summary: a write-and-fan-out-heavy system where 50M phones stream location through Kafka into an in-memory, user-sharded store, each ping is intersected against a cached friend graph and a geohash cell population to find the handful of authorized friends actually within 5km, movements are delivered through a recipient-keyed sharded pub/sub bus across a WebSocket fleet, presence is TTL-driven, and a fail-closed per-publish privacy gate makes “friends only, opt-in” safe - all eventually consistent except that one permission check.