Design GitHub (Code Hosting + PRs) - System Design

“Host code and let people review changes” sounds like a CRUD app with a diff view. It is not. Under the hood you are running a fleet that stores 100M Git repositories, each a content-addressed object database with its own history, serving the Git wire protocol (which is a chatty, stateful negotiation, not a REST call), computing diffs and merges on demand, fanning out webhooks to CI on every push, and - the genuinely hard part - indexing and searching across all public source code on earth so a query for a function name returns in a few hundred milliseconds. Every one of those is a different system with a different bottleneck.

The core tension: a Git repository is a stateful, mutable, per-repo blob that a handful of users write to, but the platform around it - search, PRs, notifications, the API - is a massively-read, fan-out, multi-tenant system. You cannot shard the whole product one way. The repo storage wants to be pinned and replicated per-repo; the search index wants to be sharded by content across everything; the metadata (PRs, comments, reviews) wants a relational database. So this design is really four systems bolted together at the seams, and the interview is about getting each seam right. Let me build it properly.

Functional Requirements (FR)

In scope:

• Repository hosting. Create, clone, fork a repo. Serve git push and git fetch/clone/pull over the Git smart HTTP and SSH protocols. Store full history, branches, tags. Private and public visibility.
• Push / fetch (the Git data path). Accept packs of new objects on push, run the pre-receive/update/post-receive hook chain, update refs atomically, and serve the negotiated set of objects on fetch. This is the highest-volume, most latency-sensitive path.
• Pull requests. Open a PR from a branch (or a fork) into a base branch, compute the diff and the mergeability, show the file-by-file changes, and merge (merge commit, squash, or rebase) with a locking guarantee so two merges do not race.
• Code review. Line-level and file-level comments, review threads, approvals / change-requests, required reviewers, and re-review on new pushes to the PR branch. Comments must anchor to a specific line of a specific blob even as the branch moves.
• CI hooks / webhooks. On push, PR open/update, and other events, deliver a webhook to registered consumers (CI, bots) reliably, with retries, ordering guarantees per repo, and status reporting back onto the commit/PR (the checks UI).
• Code search across all public code. Search function names, symbols, and arbitrary substrings across every public repo (billions of files), with filters (language, repo, path), returning ranked results in a few hundred ms. Also per-repo search inside private repos for authorized users.

Explicitly out of scope (say it so you own the scope):

• Issues, projects, wikis, discussions, the social graph (followers, stars feed). Same relational-metadata shape as PRs; I mention where they plug in but do not design them.
• Actions/CI execution itself - the runner fleet that actually builds and tests. I design the webhook + status contract that triggers and reports CI, not the compute that runs it.
• Packages, Pages, Codespaces, the LFS large-file store as their own products. I note where LFS diverges from normal object storage.
• Auth/authz internals (OAuth apps, fine-grained tokens, SSO). I assume an identity + permission service exists and gate every path on it, but its design is a separate problem.

The decision that drives everything: a Git repository is a per-repo stateful data structure, not a stateless record, so repo storage is pinned-and-replicated per-repo while everything derived from it (search, PR metadata, events) is a separate, differently-sharded system fed asynchronously. Get that split right and the rest follows.

Non-Functional Requirements (NFR)

• Scale: 100M repositories, ~500M code pushes/year, ~100M users. Billions of source files in the public corpus for search. Clone/fetch traffic dwarfs push (reads » writes on the Git path too, driven by CI cloning on every build).
• Latency: interactive Git ops (a small fetch, a ref update) p99 under a few hundred ms; a git clone is bounded by pack size and network, not our overhead. PR diff render p99 < 1s for a normal PR. Code search p99 a few hundred ms to first page. Webhook delivery started within ~1s of the push landing.
• Availability: 99.95%+ for the Git data path and the API. A repo being unavailable blocks a team’s entire workflow (they cannot deploy). Search can degrade (stale or slower) without taking the product down - it is important but not on the critical write path.
• Consistency: a ref update must be strongly consistent and atomic - a push either fully lands and is visible, or it does not; two pushes to the same branch must serialize, and a PR merge must not race another merge onto the same base. Object storage is content-addressed so objects are immutable and can be eventually replicated, but refs (the mutable pointers) demand a consensus/lock. Search and derived counts are eventually consistent.
• Durability: code is sacred. A repo’s object database must survive disk and node failure - synchronous replication across nodes/AZs, backups, no acknowledged push ever lost. Losing a customer’s git history is an existential failure.
• Multi-tenancy and isolation: 100M repos share a fleet. One giant/abusive repo (a monorepo with millions of files, or a fork bomb) must not starve neighbors. Per-repo resource limits and hot-repo handling are first-class.

Back-of-the-Envelope Estimation (BoE)

Real numbers with the arithmetic, because they set the architecture.

Push QPS (the write path):

500M pushes/year
= 500,000,000 / (365 * 86,400)
≈ 16 pushes/sec average
Peak (workday, CI-driven bot pushes, ~20x) ≈ 300-400 pushes/sec

Push QPS is modest. Pushes are rare relative to reads and each is heavy (receive a pack, run hooks, update a ref). The write path is about correctness and atomicity, not raw QPS.

Fetch / clone QPS (the read path, the real Git volume):

Every CI build clones or fetches. Assume each push triggers CI on
~3 downstream consumers, plus human pulls, plus mirror/fork syncs.
Rough: ~30-50x the push rate in fetch operations, and clones are
much heavier than pushes (they ship the whole reachable object set).
Say ~5,000-10,000 fetch/clone ops/sec at peak, bandwidth-dominated.

Reads dominate even on the Git path, and they are bandwidth-heavy (shipping packs). This is why fetch must be served from local replicas near the reader and heavily cached, and why CI clone patterns (shallow/partial clones) matter to the whole fleet’s load.

Repository storage:

100M repos. Highly skewed size distribution:
  median repo   ~ a few MB (packed)
  p99 repo      ~ hundreds of MB to GB
  a few monsters (monorepos, ML repos) ~ tens/hundreds of GB
Assume a blended average of ~50 MB packed per repo (generous; most are tiny).
100M * 50 MB = 5 PB of packed git object data.
With 3x replication for durability => ~15 PB raw.
Plus LFS/large binaries handled separately in object storage (blob store),
which can dwarf the git data for repos that use it.

Multiple petabytes, replicated 3x. This does not fit one machine or even one rack - repos are distributed across a large fleet of storage nodes, each holding a subset, with per-repo placement.

Code search index (the hard one):

Public source files: on the order of a few billion files.
Assume ~3B searchable files, average ~10 KB of text = ~30 TB of raw source.
A trigram / positional inverted index typically runs a few multiples of the
raw text (postings + symbol index + metadata):
  ~3-5x => ~100-150 TB of index.
Sharded across, say, hundreds of index nodes to keep per-shard RAM/SSD sane
and to fan a query out for parallelism. Index is rebuilt/updated incrementally
as pushes land on default branches.

100+ TB of index, sharded across hundreds of nodes, fanned-out per query. Search is its own distributed system with its own storage budget, larger than you expect and completely separate from the git storage fleet.

PR / metadata volume:

Assume ~1 PR per ~5 pushes -> ~100M PRs/year, each with commits,
review comments, status checks. Comments: say ~10M review comments/day.
Metadata row storage is small per item (~KB) but relationally rich and
queried constantly (the PR page, the API). Terabytes/year in a sharded
relational store - trivial size, but high query fan-in.

Webhook / event volume:

Each push -> PR-update + N webhook deliveries + status updates + search
index update + notification fan-out. Say ~10-20 downstream events per push.
16 pushes/sec avg * ~15 = a few hundred events/sec avg, thousands at peak,
each needing at-least-once delivery with retries. Modest QPS, strict
reliability + per-repo ordering requirements.

The headline: push QPS is tiny but each push is a heavy, atomic, correctness-critical operation; fetch/clone is bandwidth-dominated and read-heavy; git storage is multiple petabytes pinned per-repo; and code search is a separate 100+ TB sharded index that is the real engineering centerpiece. Four systems, four sharding strategies.

High-Level Design (HLD)

The spine: clients speak Git (SSH/HTTPS) or REST/GraphQL. A routing layer maps owner/repo to the storage node(s) that physically hold that repo’s object database. Git storage nodes own the on-disk repo and run the hook chain; refs are updated through a consensus/lock so pushes serialize. A push emits an event onto a bus that fans out to the PR/metadata service (relational), the search indexer, the webhook dispatcher, and notifications. Code search is its own sharded index cluster queried via a fan-out coordinator. Object storage and LFS blobs sit behind the storage nodes.

                         ┌───────────────────────────────┐
                         │           Clients             │
                         │  git (ssh/https), web, API    │
                         └───────┬───────────────┬───────┘
                     git proto   │               │  REST / GraphQL
                         ┌───────▼───────┐  ┌────▼─────────────┐
                         │  Git Frontend │  │   API Gateway    │
                         │ (ssh/https,   │  │ authz, rate-limit│
                         │  auth, route) │  │  routing         │
                         └───────┬───────┘  └───┬────────┬─────┘
              route owner/repo   │              │        │
              -> storage node    │        reads │        │ search
                         ┌───────▼──────────────▼───┐    │
                         │     Repo Router / Catalog │    │
                         │  (owner/repo -> shard set,│    │
                         │   placement, primary)     │    │
                         └───────┬───────────────────┘    │
                                 │ RPC to the repo's nodes │
        ┌────────────────────────▼─────────────────┐      │
        │            Git Storage Fleet              │      │
        │  node holds subset of repos as real .git  │      │
        │  dirs; 3x replicated; ref updates via a    │      │
        │  per-repo consensus/lock; runs hook chain  │      │
        └───────┬───────────────────────────┬───────┘      │
                │ post-receive event         │ LFS/blobs    │
        ┌───────▼───────────────┐    ┌───────▼───────┐      │
        │   Event Bus (Kafka)   │    │ Object / Blob │      │
        │  per-repo ordered      │    │ Store (LFS,   │      │
        └──┬──────┬──────┬──────┘    │  packfiles)    │      │
           │      │      │           └───────────────┘      │
    ┌──────▼─┐ ┌──▼────┐ ┌▼──────────┐   ┌──────────────────▼───┐
    │ PR /   │ │Webhook│ │  Search    │   │   Code Search Cluster │
    │metadata│ │Dispatch│ │ Indexer    │──▶│  sharded trigram/     │
    │ (SQL,  │ │(retry, │ │(build/update│  │  symbol index, fan-out│
    │sharded)│ │ order) │ │ postings)   │  │  query coordinator    │
    └────────┘ └───┬───┘ └───────────┘   └───────────────────────┘
                   │ deliver + status
              ┌────▼─────┐
              │ CI / bots │  (external consumers, report checks back)
              └──────────┘

Push flow (the correctness-critical path):

1. Client runs git push over SSH/HTTPS. The Git Frontend authenticates and asks the Repo Router which storage nodes hold owner/repo and which is the current primary.
2. The client and the primary storage node negotiate: the client sends a packfile of the objects the server lacks. The node writes the objects into the repo’s object database (content-addressed, so they are immutable and de-duplicated by hash).
3. The node runs the pre-receive / update hooks (policy: protected branches, size limits, secret scanning gate). If any reject, the push fails atomically - no ref moves.
4. The node updates the ref (e.g. refs/heads/main) through a per-repo consensus/lock so this update serializes against any concurrent push. The ref update is atomic and durably replicated to the replica nodes before ack.
5. The node acks the push and emits a post-receive event onto the bus (repo, ref, old sha, new sha, pusher).
6. Downstream, asynchronously: the PR service updates any open PRs on that branch and recomputes mergeability; the search indexer queues a re-index if it was the default branch; the webhook dispatcher enqueues deliveries to CI; notifications fan out. None of this blocks the push ack.

Fetch / clone flow (the read path):

1. Client git fetch/clone. Frontend authenticates, router picks a replica node near the client (reads served off replicas, not just the primary).
2. Client and node run the ref-advertisement + want/have negotiation: client says what refs it wants and what it already has; server computes the minimal set of reachable objects.
3. Server streams a packfile of exactly those objects (often served from a cached pack for popular clones like a fresh CI clone of main). Shallow/partial clone flags bound the object set.

PR / review flow:

1. POST /repos/{o}/{r}/pulls with head and base branches. The PR service records the PR (relational) and computes the diff by asking the storage node for the merge-base and the tree diff between base and head.
2. Reviewers comment on lines; each comment stores a stable anchor (blob sha + line, plus diff hunk context) so it survives new pushes. New pushes to the head branch (via the event bus) re-render the diff and mark stale reviews.
3. On merge, the PR service asks the storage node to perform the merge (merge commit / squash / rebase) under the same per-repo ref lock, so the base ref cannot move underneath the merge.

The key structural insight: the storage node is the only thing that touches the actual git object database, and refs move only under a per-repo lock; everything else consumes an event stream and is eventually consistent. The write path is small and strongly-consistent; the derived world is large and asynchronous.

Component Deep Dive

The hard parts, naive-first then evolved: (1) repository storage and placement, (2) atomic ref updates and PR merges under concurrency, (3) code review anchoring, (4) reliable webhooks with ordering, (5) code search across all public code.

1. Repository storage and placement

The job: physically store 100M repos (multiple PB), serve push/fetch with low latency, survive node loss, and keep one giant repo from hurting neighbors.

Naive approach: one big shared filesystem / one database of blobs

Put every repo as a .git directory on a giant shared network filesystem (or stuff git objects as rows in a central database), and let any app server operate on any repo.

/repos/{owner}/{repo}/.git/...   on one big NFS/cluster FS
any app server -> operate on any repo over the network mount

Where it breaks:

• Git is chatty and locality-sensitive. A single git operation touches thousands of small object files and does a lot of stat/random reads. Over a network filesystem this is death by round-trips - clone latency explodes.
• No isolation. A monorepo doing a huge repack, or an abusive fork bomb, saturates the shared FS and every repo on it slows down. One tenant sets everyone’s latency.
• The central-DB-of-objects variant loses all of git’s on-disk optimizations (packfiles, delta compression, reachability walks) and turns a git clone into millions of row reads. It also throws away git’s own content-addressed dedup.
• Scaling and failover of one giant FS is a nightmare; it is a single blast radius for petabytes.

A git repo wants to live as a real, local, packed .git directory on a specific machine. Fighting that is the mistake.

Evolved approach: repos pinned to storage nodes as real git dirs, N-way replicated, routed via a catalog, with hot-repo handling

Treat each repo as a unit that is placed on a small set of storage nodes (a primary + replicas), stored as an actual on-disk .git directory so git’s own machinery works at full speed.

Repo Catalog (the routing brain):
  owner/repo -> repo_id
  repo_id    -> { primary_node, replica_nodes[], generation }
  placement chosen by hashing repo_id into the node ring, with
  rebalancing when nodes join/leave.

Storage node:
  holds a subset of repos as local .git dirs on fast SSD
  serves push (primary) and fetch (any replica)
  replicates objects + ref updates to replicas

Why each piece matters:

• Real local git dirs mean clone/fetch/repack run at native speed - git’s packfiles, delta chains, and bitmap indexes all work. No network filesystem in the object path.
• Placement by repo_id, not by owner, spreads a single large org’s many repos across the fleet instead of piling them on one node. The catalog is the indirection layer that lets us move a repo (rebalance, evacuate a dying node) without changing its URL.
• N-way replication (e.g. 3x) with the primary owning writes. Objects are content-addressed and immutable, so replicating them is a straightforward copy; the subtle part is refs (next deep dive). Reads (fetch/clone) are served from any healthy replica near the client, spreading the bandwidth-heavy read load.
• Per-repo resource limits and isolation. Each repo gets CPU/IO quotas on its node; a runaway repack is cgroup-limited so it cannot starve neighbors. A repo that grows monstrous can be placed on a dedicated node.
• Hot-repo handling. A popular public repo (think a framework everyone clones in CI) gets extra read replicas and a cached “clone pack” - a pre-built packfile of main served directly from cache/CDN, so the millionth shallow clone of the day does not recompute the pack. Popularity is a read-scaling problem solved by more replicas + pack caching, orthogonal to write placement.
• Maintenance: background git gc/repack and bitmap generation keep repos fast to serve, scheduled off-peak and rate-limited so maintenance never competes with live traffic.

The trade-off: pinning repos to nodes means a failed node requires promoting a replica and re-replicating, and rebalancing moves data - but in exchange every git operation runs locally at full speed with real isolation, which the shared-FS approach can never match.

2. Atomic ref updates and PR merges under concurrency

The job: a ref (refs/heads/main) is the one mutable thing in an otherwise immutable object store. Two pushes to the same branch, or a push racing a PR merge, must serialize - and the winner’s update must be durable before ack. This is the strong-consistency core of the whole system.

Naive approach: write the ref file, replicate lazily

The primary node just writes the new sha into the ref (git’s default is a file or the reflog/packed-refs), acks, and copies to replicas in the background.

push: verify old_sha == current; write new_sha to refs/heads/main; ack
replicate ref to replicas asynchronously

Where it breaks:

• Lost updates on failover. Primary acks a push, then dies before the ref reached the replicas. A replica is promoted - now the acked commit is gone. That is losing customer code, the cardinal sin.
• Split-brain. During a network partition two nodes both think they are primary and both accept pushes to main with divergent history. Now you have two conflicting truths for one ref and no way to reconcile without data loss.
• Merge races. Two PRs both compute mergeability against main at sha A, both merge, both write their result assuming base was A. The second silently clobbers or produces a broken history. The classic “compare-and-swap on the ref” is missing.

Refs are the one place eventual consistency is unacceptable. Writing-then-replicating gets code lost.

Evolved approach: ref updates via per-repo consensus (quorum), compare-and-swap semantics, and merges under the same lock

Make the ref update a consensus operation across the repo’s replica set. A ref only advances when a quorum has durably persisted the new value.

Ref update (push or merge) on repo R, ref main, expected old_sha:
  1. objects for the push are streamed to primary AND replicated to the
     replica set (content-addressed, safe to copy eagerly).
  2. propose ref update {main: old_sha -> new_sha} to the repo's
     consensus group (Raft-style quorum over the N replicas, or a
     transactional ref backend like a reftable replicated by consensus).
  3. the update is a COMPARE-AND-SWAP: it commits only if current == old_sha.
     - concurrent push with a stale old_sha -> rejected ("fetch first").
  4. commit requires a quorum ack (e.g. 2 of 3) -> durable across nodes
     BEFORE we ack the client. No acked ref can be lost to a single-node
     failure, and failover promotes a node that has the committed value.

Why this is right:

• Quorum durability before ack means an acked push survives any single node failure - the committed ref lives on a majority, and leader election can only pick a node that has it. No lost commits, no split-brain (a minority partition cannot reach quorum, so it cannot accept writes).
• Compare-and-swap on the ref is exactly the primitive git’s protocol already wants (old_sha is sent by the client). It makes concurrent pushes serialize: one wins, the other is told to fetch and retry. No lost updates.
• PR merge is just a ref update with extra steps, under the same CAS. The PR service computes the merge commit (or squash/rebase result) against the current base sha, then attempts a CAS ref update base: old_sha -> merge_sha. If another merge or push moved the base in the meantime, the CAS fails and the merge is recomputed/retried against the new base. This is what makes “merge” safe without a global lock - it is an optimistic compare-and-swap, serialized by consensus, scoped to the single repo.
• The hook chain runs before the ref commits. Pre-receive/update hooks (protected-branch rules, required status checks, secret scanning) are evaluated before the CAS commits, so a rejected push never moves the ref at all - the atomicity is real, not cosmetic.
• Scope is per-repo. Consensus is expensive, but it is only over one repo’s tiny replica set (3 nodes) and only for the rare write (16 pushes/sec globally). We are not running one giant Raft; we are running 100M tiny independent consensus groups, one per repo, so it scales horizontally with the fleet.

The trade-off: a quorum write adds a little latency to each push versus a local file write, and running per-repo consensus is operationally heavier - but it is the only way to guarantee “no acked code is ever lost and no two writers diverge,” which is non-negotiable for a code host.

3. Code review: anchoring comments to a moving target

The job: a reviewer comments on “line 42 of auth.go.” Then the author pushes three more commits that add lines above 42 and edit the file. The comment must still point at the right code, the thread must stay coherent, and the diff must re-render. Anchoring to a raw line number is naive and wrong.

Naive approach: store (file path, line number) on the tip commit

Record the comment as {path: "auth.go", line: 42} against whatever the branch tip is.

comment: { pr_id, path, line, body }   # line = 42 on current tip

Where it breaks:

• Line numbers shift. The author adds 5 lines above line 42; the comment now points at unrelated code. The review reads as nonsense.
• The file gets renamed or the region deleted. Line 42 no longer exists. The comment is orphaned with no defined behavior.
• No sense of “outdated.” When the code the comment referred to changes, the reviewer needs to know the comment is now stale - a bare line number cannot express that.
• Force-push rewrites history entirely; the tip sha the comment implicitly assumed is gone.

A line number is a coordinate in a document that is constantly being edited underneath it. It cannot be the anchor.

Evolved approach: anchor to (blob sha, line) with diff-hunk context, then re-position on each push

Anchor each comment to an immutable point in history plus enough context to re-locate it as the branch moves.

review_comment:
  pr_id
  commit_sha        -- the head commit the comment was made against
  path
  blob_sha          -- content hash of the file version commented on (immutable)
  original_line     -- line in THAT blob
  diff_hunk         -- the surrounding hunk text (context for re-anchoring)
  side              -- LEFT (base) or RIGHT (head) of the diff
  in_reply_to       -- thread linkage
  position          -- current computed line on the latest diff (recomputed)

The mechanism:

• blob_sha is immutable and content-addressed, so the comment always knows exactly which bytes it was made against - that never changes even if the branch is force-pushed. The comment is never truly lost; worst case it becomes “outdated” and is shown against the historical blob.
• On each new push to the PR branch, recompute the diff and try to re-anchor the comment: map its original_line in the old blob to a line in the new blob using the diff between them (a line-mapping over the hunks). If the mapping succeeds, the comment moves to the new line and stays “current.” If the surrounding code changed materially or was deleted, mark the thread outdated and show it collapsed against the original blob context (the diff_hunk we stored lets us render it even though it is gone from the tip).
• side (LEFT/RIGHT) distinguishes a comment on removed code (base side) from added code (head side), which a single line number cannot.
• Reviews and approvals are tied to a commit sha. An approval is “approved at sha X.” When a new commit lands, the approval can be shown as “approved a previous version,” and branch-protection rules can require re-approval of the new head - because we recorded which version was approved.
• Threads survive force-push because they are keyed on blob content, not on a mutable ref. A force-push changes the tip but the blobs the comments reference still exist as objects (retained via the reflog/keep for the PR), so threads render.

Why this is right: content-addressed blob shas give comments an immutable ground truth, the stored hunk context lets us both re-anchor to the moving tip and render orphaned threads, and tying approvals to specific shas makes “this was reviewed” a precise, enforceable statement rather than a vibe. Review integrity is a versioning problem, and git already hands us the version identifiers.

4. Reliable webhooks and CI hooks with per-repo ordering

The job: every push must reliably trigger downstream consumers (CI, bots) with at-least-once delivery, bounded retries, and - critically - per-repo ordering so a “branch deleted” event never arrives before the “branch pushed” it supersedes. And a slow or dead consumer must not back up the whole system.

Naive approach: POST the webhook synchronously from the push path

When the push lands, the storage node directly HTTP POSTs the webhook to the consumer and waits.

post-receive: for each subscriber: http_post(url, event)   # inline, blocking

Where it breaks:

• A slow consumer blocks the push. If the CI endpoint takes 5s (or hangs), the user’s git push hangs with it. Delivery must never be on the push’s critical path.
• No retries / lost events. The consumer is briefly down -> the event is dropped -> CI never runs -> a broken commit ships. At-least-once needs durable queuing and retry, which inline POST has none of.
• No ordering. Fire-and-forget concurrent POSTs can arrive out of order; the consumer sees “deleted” before “created.”
• Fan-out amplification. A push with many subscribers turns one push into many synchronous calls; one bad subscriber degrades everyone.

Delivery is a distributed reliability problem, and the push path is the wrong place to solve it.

Evolved approach: durable event log, async dispatcher with per-repo ordered partitions, retries with backoff and DLQ, status reported back

Decouple with a durable event bus and a dedicated dispatcher:

push -> emit event to bus (Kafka), partitioned by repo_id
        => all events for one repo land in ONE partition = ordered.
Dispatcher consumes per partition:
  for each subscriber of that repo/event:
     enqueue a delivery attempt (durable)
     POST with timeout; on success mark delivered
     on failure -> retry with exponential backoff (e.g. 1s,10s,1m,...)
     after N failures -> dead-letter queue + surface to the repo owner
Status/check results flow back via the API onto the commit + PR
  (the "checks" UI), keyed by (commit_sha, check_name).

Why this is right:

• Push acks immediately; delivery is fully async off the durable log. A dead consumer never touches the user’s push.
• Partition by repo_id gives per-repo total ordering for free (Kafka guarantees order within a partition). Events for a repo are processed in sequence, so “push” precedes “delete.” Different repos are independent partitions and parallelize.
• At-least-once with durable retries + backoff. The event is persisted before we attempt delivery, so a transient consumer outage just delays it; retries with exponential backoff and a cap, then a dead-letter queue the owner can inspect/replay. Consumers must be idempotent (we send a delivery id) because at-least-once can double-deliver.
• Slow-consumer isolation. Each subscriber has its own delivery queue/worker pool; one hung endpoint fills its own queue and gets circuit-broken, not the shared path. Fan-out is bounded and back-pressured per subscriber.
• Status back-channel. CI reports results via POST /repos/.../check-runs keyed by (commit_sha, check_name); branch protection reads those to allow/block merge. This closes the loop - the webhook triggers CI, CI reports checks, checks gate the merge (deep dive 2’s hook chain reads them).
• Ordering vs throughput knob: for consumers that do not need ordering, we can parallelize within a repo; for those that do (most CI), we keep the single ordered partition. It is a per-subscriber choice.

The trade-off: at-least-once means consumers must dedupe, and per-repo ordering caps parallelism within a hot repo - but we get reliable, ordered, isolated delivery that never endangers the push, which fire-and-forget can never offer.

5. Code search across all public code

The job: a query for parseConfig (or a symbol, or an arbitrary substring, with regex) must search across billions of files in every public repo and return ranked results in a few hundred ms. This is the flagship hard problem and it is unlike normal text search.

Naive approach: git grep on demand, or a standard word-based inverted index

Either shell out to git grep across repos at query time, or feed source into a normal full-text search engine that indexes words.

query time: for each repo: git grep pattern   # across billions of files
   -- or --
index source into a word-tokenized inverted index (like doc search)

Where it breaks:

• On-demand grep does not scale at all. Grepping billions of files per query is minutes-to-hours, not milliseconds. There is no way to make brute-force scan interactive across the whole corpus.
• Word tokenization is wrong for code. Code is not prose. parseConfig, parse_config, ParseConfig, config.parse() - a word tokenizer mangles identifiers, splits on the wrong boundaries, and cannot do substring or regex search (.*Config), which is what code search actually needs. Searching for a partial identifier or a symbol across camelCase/snake_case fails.
• Scale of updates. 500M pushes/year each change files; the index must update incrementally, not rebuild. A naive engine chokes on the write volume and the corpus size.

Code search needs substring/regex over identifiers at corpus scale - a fundamentally different index than prose search.

Evolved approach: sharded trigram inverted index + symbol index, fan-out query with a candidate-then-verify pipeline, indexing default branches only

Build a purpose-built code index around trigrams (and a separate symbol index), sharded across the corpus, queried by fan-out.

Indexing (per file on the default branch):
  1. break the file content into overlapping 3-grams:
       "parseConfig" -> "par","ars","rse","seC","eCo","Con","onf","nfi","fig"
  2. inverted index: trigram -> posting list of (file_id, positions)
  3. separate SYMBOL index: parse the file (ctags/tree-sitter) to extract
     definitions (functions, classes, vars) -> symbol -> file_id, so
     "definition of parseConfig" is a direct lookup, ranked above matches.
  4. store file metadata: repo, path, language, default-branch commit.
Only index the DEFAULT branch of PUBLIC repos for global search
(indexing every branch of every repo is wasteful and mostly noise).
Private-repo search is scoped per-repo/org and authz-filtered.

Sharding: partition the index by content (hash of file_id / repo bucket)
across hundreds of nodes so each holds a slice and queries parallelize.

Query (substring or regex "src.*Config"):
  1. derive the required trigrams from the query
     ("Config" -> "Con","onf","nfi","fig"; regex -> trigram sets via the
      regex's mandatory substrings).
  2. fan out to all shards: each intersects the trigram posting lists to
     get CANDIDATE files that COULD match (trigram match is necessary,
     not sufficient).
  3. VERIFY: run the actual regex/substring only against the small
     candidate set (not the whole corpus) to eliminate false positives.
  4. each shard returns its top-k ranked hits; a coordinator MERGES the
     per-shard top-k into a global top-k.
  5. rank by symbol-definition > exact match > path/repo popularity
     (stars) > recency, and AUTHZ-FILTER to what the user may see.

Why this is right:

• Trigrams make substring and regex search tractable. Any substring of length >= 3 implies a set of trigrams that must all be present; intersecting those posting lists cheaply narrows billions of files to a few thousand candidates without scanning content. Regex is reduced to the mandatory trigrams it implies, then verified. This is the key trick that turns “grep the world” into an index lookup.
• Candidate-then-verify bounds the expensive exact-match work to the tiny candidate set, so p99 stays in the hundreds of ms even for regex.
• A separate symbol index handles the most common intent (“take me to the definition”) directly and ranks definitions above incidental text matches - code search is mostly navigational.
• Sharding by content + fan-out gives horizontal scale and parallelism: 100+ TB of index across hundreds of nodes, each query touching all shards in parallel and merging top-k. Add corpus -> add shards.
• Index only default branches of public repos cuts the corpus and the update rate massively while covering ~all of what users actually search. Push events on the default branch (from the event bus, deep dive 4) enqueue incremental re-index of just the changed files - no full rebuilds.
• Authz filtering at query time for private/scoped search: the global public index is safe to search broadly; private results are filtered to the user’s accessible repos (or served from per-repo indexes) so search never leaks private code.
• Freshness is eventual and that is fine (an NFR we set): a file becomes searchable seconds-to-minutes after the push, off the critical path. Search degrading or lagging never blocks a push or a clone.

The trade-off: the index is large (100+ TB) and eventually consistent, and default-branch-only means non-default branches are not globally searchable - but in exchange we get interactive substring/regex/symbol search across billions of files, which no on-demand or word-index approach can deliver.

API Design & Data Schema

API

Git data path (the protocol, not REST):

# Smart HTTP (git clone/fetch/push under the hood)
GET  /{owner}/{repo}.git/info/refs?service=git-upload-pack   -> ref advertisement (fetch)
POST /{owner}/{repo}.git/git-upload-pack                     -> want/have negotiation, returns packfile
GET  /{owner}/{repo}.git/info/refs?service=git-receive-pack  -> ref advertisement (push)
POST /{owner}/{repo}.git/git-receive-pack                    -> receive packfile, run hooks, CAS ref update
# SSH: git-upload-pack / git-receive-pack over an ssh channel, same negotiation

REST / GraphQL (the product surface):

POST /api/v3/user/repos            { name, visibility }        -> 201 repo
POST /api/v3/repos/{o}/{r}/forks                               -> 202 fork queued

POST /api/v3/repos/{o}/{r}/pulls   { title, head, base }       -> 201 pull_request
GET  /api/v3/repos/{o}/{r}/pulls/{n}                           -> PR + mergeable state
GET  /api/v3/repos/{o}/{r}/pulls/{n}/files                     -> paginated diff (per-file)
POST /api/v3/repos/{o}/{r}/pulls/{n}/merge   { merge_method }  -> 200 merged | 409 base moved / not mergeable

POST /api/v3/repos/{o}/{r}/pulls/{n}/comments
     { commit_sha, path, side, line, body, in_reply_to }       -> 201 review comment
POST /api/v3/repos/{o}/{r}/pulls/{n}/reviews
     { commit_sha, event: APPROVE|REQUEST_CHANGES|COMMENT }    -> 200 review

POST /api/v3/repos/{o}/{r}/hooks   { url, events[], secret }   -> 201 webhook subscription
POST /api/v3/repos/{o}/{r}/check-runs
     { commit_sha, name, status, conclusion }                 -> 201 check (CI reports back)

GET  /api/v3/search/code?q=parseConfig+language:go+repo:o/r    -> ranked code results (paginated)

Conventions: the merge endpoint returns 409 when the base ref moved (the CAS failed) - the client retries; diffs and search are paginated (a PR or result set can be huge); webhooks carry a delivery id + HMAC signature (the secret) so consumers can dedupe and verify; every path is authz-gated by the permission service.

Data stores

Different access patterns, different stores. Be explicit.

1. Git object + ref storage - the storage fleet, NOT a database. Repos live as real on-disk .git directories (packfiles + refs) on storage nodes, 3x replicated, refs updated via per-repo consensus. This is deliberately not SQL/NoSQL - git’s content-addressed packfile format with delta compression and reachability bitmaps is a purpose-built store that any general database would do worse. The “schema” is the git object model itself (blobs, trees, commits, tags addressed by SHA).

Storage node layout:
  /data/{repo_id}/objects/pack/*.pack   -- delta-compressed object packs
  /data/{repo_id}/objects/*             -- loose (recent) objects
  /data/{repo_id}/refs (reftable)       -- mutable refs, updated by CAS/consensus
Replication: object copy (immutable, easy) + ref consensus (quorum)
LFS / large binaries -> external object/blob store (S3-like), pointer in git.

2. Repo catalog / routing - relational, strongly consistent. The map every request needs: owner/repo -> repo_id -> placement. Small, hot, must be correct.

Table: repositories
  repo_id       BIGINT  PRIMARY KEY
  owner_id      BIGINT
  name          STRING
  visibility    STRING   -- public|private|internal
  default_branch STRING
  primary_node  STRING
  replica_nodes STRING[]
  generation    INT      -- bumped on placement change
  Index: (owner_id, name) UNIQUE, repo_id (pk)

3. PR / review metadata - relational (PostgreSQL/MySQL), sharded by repo_id. Rich relations (PRs -> commits -> comments -> reviews), transactional integrity, constant querying by repo. Classic relational job; small data, high query fan-in. Shard by repo_id so all of a repo’s PR data is co-located and a PR page is a single-shard query.

Table: pull_requests
  pr_id BIGINT PK, repo_id BIGINT, number INT, author_id BIGINT,
  head_sha CHAR(40), base_branch STRING, base_sha CHAR(40),
  state STRING(open|closed|merged), mergeable BOOL, created_at TS
  Index: (repo_id, number) UNIQUE, (repo_id, state)
  Shard by: repo_id

Table: review_comments
  id BIGINT PK, pr_id BIGINT, commit_sha CHAR(40), path STRING,
  blob_sha CHAR(40), original_line INT, side STRING, diff_hunk TEXT,
  position INT NULL, in_reply_to BIGINT NULL, author_id BIGINT, body TEXT
  Index: (pr_id), (pr_id, path)
  Shard by: repo_id (co-located with the PR)

Table: reviews
  id BIGINT PK, pr_id BIGINT, reviewer_id BIGINT,
  commit_sha CHAR(40), state STRING(approved|changes_requested|commented)
  Index: (pr_id), Shard by: repo_id

4. Code search index - sharded trigram + symbol store (custom / search engine). 100+ TB, sharded by content hash across hundreds of nodes, queried by fan-out coordinator. Not a general database - a purpose-built inverted index (postings on SSD, hot trigrams in RAM).

Shard holds:
  trigram -> posting list (file_id, positions)     -- substring/regex
  symbol  -> [file_id...]                           -- definitions
  file_id -> { repo_id, path, language, commit_sha, visibility }
Shard by: hash(file_id)  (content-spread, uniform)
Query: fan-out to all shards -> candidate trigram intersect -> verify -> merge top-k

5. Event bus - Kafka, partitioned by repo_id. Durable, ordered-per-repo event log feeding PR updates, indexing, webhooks, notifications. Retained long enough to replay a consumer.

Topic: repo-events   partition_key = repo_id
Event: { repo_id, ref, old_sha, new_sha, pusher_id, type, ts, delivery_id }

6. Webhook delivery store - relational/KV, per subscriber. Subscriptions plus durable delivery attempts with status for retry/DLQ/replay.

Table: webhooks  { id, repo_id, url, events[], secret, active }
Table: deliveries { id, webhook_id, event_id, status, attempts, next_retry_at, response_code }
  Index: (status, next_retry_at)  -- the retry scanner

7. Check runs / CI status - relational, keyed by (commit_sha, name). What CI reports back; branch protection reads it to gate merges.

Table: check_runs { id, repo_id, commit_sha, name, status, conclusion, details_url, ts }
  Index: (repo_id, commit_sha)

Why this split: git objects/refs demand git’s own content-addressed store (nothing general beats it, and refs need per-repo consensus, not a global transaction); catalog, PR metadata, and check runs need relational integrity and are small, so sharded SQL by repo_id co-locates a repo’s world for cheap single-shard reads; code search needs a purpose-built sharded trigram index at a totally different scale and consistency (eventual); and events need a durable ordered log. No single database spans these; the seams are the design.

Bottlenecks & Scaling

Where it breaks first, in order, and the fix for each.

1. Fetch/clone bandwidth and hot public repos (breaks first). A wildly popular repo cloned by CI millions of times a day saturates its node’s bandwidth and recomputes packs. Fix: extra read replicas for hot repos, served near the client, plus a cached pre-built clone pack for the default branch (served from cache/CDN) so repeat shallow clones do not recompute. Encourage shallow/partial clones from CI. Read load is a replica-and-cache problem, decoupled from the write primary.

2. The monorepo / giant-repo tenant. One repo with millions of files or a huge history does expensive repacks and huge diffs, starving neighbors. Fix: per-repo resource quotas (cgroups) and isolation on the storage node, dedicated nodes for giants, rate-limited background maintenance, and diff/pack size limits with pagination. One tenant cannot set the fleet’s latency.

3. Ref-update / merge contention on a hot branch. Many bots pushing/merging to main of a busy repo serialize on the per-repo consensus. Fix: the contention is inherent (you cannot parallelize writes to one ref without diverging history), so make it cheap: optimistic CAS with fast retry (recompute merge against new base and re-CAS), keep the consensus group small (3 nodes), and push non-conflicting work (diff render, checks) off the lock. Different refs/repos never contend - the lock is scoped to a single ref’s advance.

4. Storage node failure / rebalancing. A node dies holding thousands of repos’ primaries. Fix: quorum replication means a replica already has every acked ref; promote a replica to primary via the catalog (bump generation), then re-replicate to restore N-way redundancy in the background. The catalog indirection makes moving repos invisible to clients (same URL). No acked data lost because ack required quorum durability.

5. Search index update lag and size. 500M pushes/year each dirty files; the 100+ TB index must stay fresh and fit. Fix: incremental indexing off the event bus (re-index only changed files on default-branch pushes), default-branch-public-only to bound the corpus, and shard by content to scale storage and parallelize queries. Search is explicitly eventually consistent, so lag is acceptable and never blocks writes; degrade gracefully (serve slightly stale) rather than fail.

6. Search query fan-out cost. Every query hits every shard; a heavy regex or a common trigram set is expensive across hundreds of shards. Fix: candidate-then-verify (trigram intersect narrows before any content scan), per-shard top-k with early termination, cache popular queries, and reject/limit pathological regexes (require minimum mandatory trigrams). Rank cheaply (symbol > exact > popularity) and paginate.

7. Webhook fan-out and slow consumers. A dead or slow CI endpoint could back up delivery. Fix: durable event log + async per-subscriber queues with backoff and DLQ + circuit breaking (deep dive 4). A bad consumer fills only its own queue; the push never waits. Per-repo partition ordering preserved.

8. PR metadata hot shard. A hugely active repo concentrates PR/comment writes on one shard. Fix: shard by repo_id spreads across repos; within a mega-repo, read replicas for the read-heavy PR pages, cache rendered diffs, and keep the write path (comments) small. The PR page is read-dominated, so caching and replicas absorb it.

9. The repo catalog as a single point of dependency. Every git op and API call resolves owner/repo -> placement through it. Fix: the catalog is small, heavily cached (read replicas + edge cache), and highly replicated. Placement changes are rare (generation-versioned), so aggressive caching is safe with cache-busting on generation bump. It is read-mostly and tiny, so it is easy to make fast and redundant.

10. Cross-region latency and disaster recovery. A repo’s storage lives in some region; global users and DR need coverage. Fix: place replicas across AZs/regions so a regional outage promotes a surviving replica; serve reads from the nearest replica; the catalog is globally replicated. The event bus and search are regionally deployed with cross-region replication for DR.

Shard keys, stated plainly: git storage -> placement by repo_id (primary + N replicas, refs via per-repo consensus); repo catalog -> repo_id (replicated, cached); PR/review/check metadata -> repo_id (co-locate a repo’s world); code search index -> hash(file_id) (content-spread, fan-out query); event bus -> repo_id partitions (per-repo ordering); webhook deliveries -> per subscriber. Never shard git storage by owner (concentrates a big org on one node) - shard by repo_id.

Wrap-Up

The trade-offs that define this design:

• A git repo is a pinned, per-repo, stateful object database - not a stateless record. Repos live as real .git directories on storage nodes, N-way replicated, routed through a catalog. Everything derived from them (search, PR metadata, events) is a separate system fed asynchronously off an event bus. Four systems, four sharding strategies, joined at the seams.
• Refs move only under per-repo consensus with compare-and-swap. The one mutable thing in a content-addressed store gets quorum durability before ack (no acked code ever lost, no split-brain) and CAS semantics (concurrent pushes and PR merges serialize, one wins, the other retries). We run 100M tiny consensus groups, not one giant lock.
• Code review anchors to content, not line numbers. Comments key on immutable blob shas plus stored hunk context, then re-anchor to the moving tip on each push and mark stale threads outdated - so review integrity survives edits, force-pushes, and renames.
• Webhooks are a durable, per-repo-ordered, retrying, isolated delivery system off the push path - at-least-once with backoff and a DLQ, slow consumers circuit-broken, and a status back-channel that closes the CI-gates-merge loop.
• Code search is a bespoke sharded trigram + symbol index with candidate-then-verify query fan-out, indexing only public default branches incrementally off the event bus - the only way to make substring/regex/symbol search across billions of files interactive.

One-line summary: a fleet of per-repo, quorum-replicated git object stores with CAS ref updates for the strongly-consistent write path, wrapped by an eventually-consistent derived world (sharded relational PR metadata, a durable ordered event bus feeding retrying webhooks, and a 100+ TB sharded trigram code-search index) - four systems bolted at the seams, each sharded the way its own access pattern demands.