The whole point of a file-transfer service sounds like a solved problem: put the file on a server, hand out a URL, let people download. That works until the file is 10GB and a million people want it in the same hour. Now do the arithmetic the interviewer is waiting for: 10GB times 1,000,000 downloads is 10 petabytes of egress, and if half of them show up in the first hour you need roughly 10 PB / 3600s ≈ 2.9 TB/sec of outbound bandwidth from your origin. No single origin, no CDN tier you would willingly pay for, serves 2.9 TB/sec for one file. The server-centric model does not scale sub-linearly with popularity - it scales linearly, and popularity is exactly when it collapses.

BitTorrent’s insight is to invert the cost curve: the more people want a file, the more capacity there is to serve it, because every downloader is simultaneously an uploader. A peer that has fetched a 16MB chunk immediately becomes a source for that chunk to other peers. The origin only needs to inject the file into the swarm once; the swarm then multiplies the bandwidth among its own members. The hard problems all fall out of that inversion: how do peers find each other with no central server, how does a file get split so peers can trade pieces out of order, how do you stop freeloaders who download but never upload, and how do you verify a piece that came from a stranger is not corrupt or malicious. Let me build it properly.

Functional Requirements (FR)

In scope:

  • Distribute one large file (or a set of files) to many peers. A publisher makes a 10GB+ file available; millions of peers download it over time. This is the core.
  • Every peer uploads while it downloads. A peer serves the pieces it already has to other peers concurrently with fetching the pieces it still needs. Download and upload are symmetric roles on the same connection set.
  • Peer discovery without a central data server. A new peer joining a file’s swarm must find other peers who have pieces, using only lightweight coordination (a tracker and/or a DHT), never a server that holds the file bytes.
  • Out-of-order, parallel piece exchange. The file is split into pieces; a peer downloads pieces from many peers at once, in whatever order it can get them, and reassembles.
  • Integrity verification. Every piece is content-verified so a peer cannot be poisoned by a corrupt or malicious piece from an untrusted peer.
  • Incentive to share. The protocol must make uploading the rational choice, so the swarm does not collapse into pure leechers.
  • Resumable transfers. A peer that disconnects mid-download resumes from the pieces it already verified, not from zero.

Explicitly out of scope (say this so you own the scope):

  • Content legality, DRM, and takedowns. The transport is content-agnostic; policy layers sit above it and I will not design them.
  • The actual disk/file-system layout on each peer. How a client writes pieces to local storage is a client-implementation detail.
  • NAT traversal internals. Real clients need hole-punching / UPnP / relays to connect peers behind NAT. I will name where it hooks in but not design the STUN/TURN machinery.
  • Streaming/media playback ordering. BitTorrent can be biased toward sequential pieces for streaming, but the base design optimizes for whole-file completion, not playback.
  • Encryption of content at rest / privacy of who-has-what. Protocol encryption exists to dodge ISP throttling; the swarm membership is inherently public.

The one decision that drives everything: this is a bandwidth-multiplication problem, not a storage or request-routing problem. Unlike a CDN (push bytes from your edge to clients) or a feed (fan out records), here the clients themselves are the infrastructure and the design’s whole job is to make peer upload capacity dominate origin capacity. Correctness means every peer ends with a byte-identical, hash-verified copy; success means the origin’s contribution stays near-constant no matter how many peers join.

Non-Functional Requirements (NFR)

  • Scale: a single popular file draws a swarm of up to ~1M concurrent peers; the system as a whole tracks millions of files (torrents) and hundreds of millions of peers globally. Files range from a few MB to 50GB+. A tracker/DHT must handle peer churn of thousands of joins-and-leaves per second per hot swarm.
  • Throughput / latency: there is no single “request latency” target; the metric is effective download rate per peer, which should approach the peer’s own downlink capacity once the swarm is warm. Time-to-first-piece (peer discovery + first verified piece) should be under a few seconds. Swarm-wide, the origin (the initial seeder) should contribute a near-constant number of bytes regardless of swarm size - that is the definition of scaling.
  • Availability: the swarm must survive continuous peer churn - peers join and vanish constantly. No single peer’s departure loses data as long as the set of pieces held collectively (the “availability” of each piece) stays >= 1 copy across live peers. The tracker/DHT itself must be highly available but holds only tiny metadata, so it is cheap to replicate.
  • Consistency: the file content is immutable and content-addressed - a torrent describes a fixed set of bytes identified by a hash. There is no mutable shared state to reconcile; “consistency” here means integrity: every downloaded piece must hash to its known value or be discarded. Peer lists from the tracker/DHT are eventually consistent and best-effort (a stale peer entry just fails to connect).
  • Durability: durability is statistical, not absolute. The file persists as long as at least one complete copy exists across all live peers (at least one seeder, or enough partial peers to collectively hold every piece). The publisher keeps an “origin seeder” up to guarantee the file never falls below 1 full copy.
  • Security/trust: peers are mutually untrusted. Integrity must not depend on trusting the sender; it depends on the content hashes in a signed/known metadata file.

Back-of-the-Envelope Estimation (BoE)

Real numbers with the arithmetic, because they justify the whole P2P approach.

The bandwidth that kills the server model:

File size        = 10 GB
Downloaders      = 1,000,000  over a launch window
Total bytes      = 10 GB * 1e6 = 1e7 GB = 10 PB (10,000,000 GB) of transfer
If 50% arrive in the first hour:
   5e6 GB / 3600 s ≈ 1,389 GB/sec ≈ 1.4 TB/sec sustained origin egress

At a generous $0.02/GB CDN egress, 10 PB is 1e7 GB * $0.02 = $200,000 for one file’s launch. And 1.4 TB/sec is far beyond what a single origin serves. This is the number that forces P2P.

What P2P does to the origin cost:

Origin only needs to inject each piece ONCE into the swarm, then peers multiply it.
Ideal origin upload = ~1 full copy = 10 GB (plus a safety margin of a few copies).
Say the origin seeds ~5 full copies before the swarm self-sustains:
   5 * 10 GB = 50 GB total origin egress  vs  10 PB server-model egress.
   Reduction factor ≈ 10 PB / 50 GB = 200,000x.

The origin’s job collapses from petabytes to tens of gigabytes. That is the entire value proposition, quantified.

Piece / chunk sizing:

Piece size (classic BitTorrent) = 256 KB to 4 MB. Use 1 MB for a 10GB file.
Pieces per file = 10 GB / 1 MB = 10,240 pieces.
Each piece has a SHA-256 hash in the metadata: 32 bytes.
Metadata (.torrent) piece-hash section = 10,240 * 32 B ≈ 320 KB.

A 320KB metadata file describes a 10GB payload - the coordination data is ~0.003% of the content. That is why the tracker/DHT is cheap: it never touches the bytes.

Tracker / DHT load:

Hot swarm       = 1,000,000 peers.
Each peer re-announces to the tracker every ~30 min (1800 s) to refresh its entry.
Announce QPS per swarm = 1e6 / 1800 ≈ 556 announces/sec.
Plus joins/leaves churn: say 2,000 events/sec at peak for a hot swarm.
Each announce/response is tiny (peer wants a list of ~50 peers):
   50 peers * 6 bytes (IPv4 + port) = 300 bytes response.
Tracker egress for the swarm ≈ 2,000/sec * 300 B ≈ 600 KB/sec.  Trivial.

Even a single tracker node handles a hot swarm’s coordination in kilobytes per second, because it only ever ships peer addresses, never file bytes. Multiply by millions of files and you shard trackers / use the DHT, but the per-swarm cost is negligible.

Per-peer connection and memory footprint:

A peer keeps ~20-80 active peer connections (a small subset of the swarm).
Piece bitfield for 10,240 pieces = 10,240 bits ≈ 1.3 KB per peer, per connection.
Per-connection state (buffers, request queue) ≈ tens of KB.
80 connections * ~50 KB ≈ 4 MB of peer-side state. Nothing.

Storage on the origin/tracker side:

Tracker stores, per swarm: info_hash -> set of {peer_ip, port, last_seen}.
1e6 peers * ~20 bytes/entry = 20 MB per hot swarm in RAM.
Millions of swarms, but most are cold (few peers). Say aggregate 500 GB of
peer-list state across all swarms -> sharded Redis, easily held in memory.

The takeaways: the origin egress drops by ~5 orders of magnitude (10 PB to ~50 GB), the coordination layer moves only kilobytes because it ships addresses not bytes, and per-peer state is a few MB. Every hard number points the same way - the scarce resource is aggregate peer upload capacity and how fairly it is used, not server bandwidth, storage, or CPU. That is where the design lives.

High-Level Design (HLD)

The architecture has three planes: a thin coordination plane (tracker + DHT) that only maps a file to the peers holding it, a metadata artifact (the .torrent / magnet link) that describes the file’s pieces and their hashes, and the data plane which is the peer swarm itself trading pieces directly. No component in the data plane is a server the publisher pays for at scale - the peers are the data plane.

  PUBLISH TIME                          DISCOVERY / COORDINATION PLANE
  ┌──────────────┐                      ┌───────────────────────────────┐
  │  Publisher    │  create .torrent    │   Tracker(s)  (stateless-ish,  │
  │  (origin      │  (piece hashes +    │   sharded by info_hash)        │
  │   seeder)     │   tracker/DHT URLs) │   info_hash -> {peer list}     │
  └──────┬───────┘        │             └───────────────┬───────────────┘
         │ seed all       │ .torrent /                  │ announce / get-peers
         │ pieces         │ magnet link                 │
         ▼                ▼                              │        ┌──────────────┐
  ┌──────────────────────────────┐                      │        │  DHT (global │
  │  Metadata: .torrent / magnet  │                      │◄──────►│  Kademlia,   │
  │  info_hash, piece_size,       │                      │        │  no central  │
  │  piece_hashes[], file_len     │                      │        │  tracker)    │
  └──────────────────────────────┘                      │        └──────────────┘
                                                         │
  DATA PLANE  (the swarm - peers trade pieces directly, no origin bytes)
  ┌─────────────────────────────────────────────────────────────────────────┐
  │                                                                           │
  │   ┌────────┐        ┌────────┐        ┌────────┐        ┌────────┐        │
  │   │ Peer A │◄──────►│ Peer B │◄──────►│ Peer C │◄──────►│ Peer D │        │
  │   │ has:   │  piece │ has:   │  piece │ has:   │  piece │ SEEDER │        │
  │   │ 1,4,7  │  swap  │ 2,4,9  │  swap  │ 3,7,9  │  swap  │ has ALL│        │
  │   └───┬────┘        └───┬────┘        └───┬────┘        └───┬────┘        │
  │       └──────────────── every peer uploads AND downloads ───┘            │
  │        each holds a bitfield of which of the 10,240 pieces it has        │
  └─────────────────────────────────────────────────────────────────────────┘

Publish flow (one time):

  1. The publisher chops the file into fixed-size pieces (say 1MB), computes a hash of each piece, and builds a metadata file: info_hash (a hash of the metadata itself, the swarm’s global ID), piece size, the ordered list of piece hashes, file length and names, and the tracker URL(s) / DHT flag.
  2. The publisher runs an origin seeder: a peer that already has 100% of the pieces and announces itself to the tracker/DHT.
  3. The publisher distributes the metadata (a .torrent file or a compact magnet link carrying just the info_hash and trackers - the metadata itself can then be fetched from peers via the DHT).

Download flow (every peer):

  1. A new peer parses the metadata, learns the info_hash and the piece hashes, and creates an empty bitfield (10,240 bits, all zero - “I have no pieces”).
  2. It announces to the tracker (GET /announce?info_hash=...&peer_id=...) and/or does a DHT get_peers(info_hash) lookup, and receives a random subset of ~50 peers currently in the swarm.
  3. It opens connections to a handful of those peers, does a handshake (exchange info_hash + peer_id), and each side sends its bitfield so both know who has what.
  4. Using a piece-selection strategy (rarest-first), the peer requests missing pieces - broken into 16KB blocks - from whichever connected peers have them, in parallel across many peers at once.
  5. Every arriving piece is hash-verified against the metadata’s expected hash. Verified -> stored, bitfield bit set, and a HAVE message is broadcast to all connected peers (“I now have piece i”). Corrupt -> discarded, re-requested from a different peer.
  6. Simultaneously, the peer serves the pieces it already has to peers that request them, governed by choking (an incentive mechanism, below).
  7. When the bitfield is full (all 10,240 verified), the peer becomes a seeder and keeps uploading (ideally) until it leaves.

The key insight: the tracker/DHT never sees a file byte - it only answers “who is in this swarm.” The bytes flow peer-to-peer, and because each downloaded piece instantly becomes uploadable, the swarm’s serving capacity grows with its size. The origin seeds a few copies and then can even leave, as long as the swarm collectively still holds every piece.

Component Deep Dive

The hard parts, naive-first then evolved: (1) peer discovery without a central server; (2) chunking and piece selection so out-of-order parallel download works and rare pieces survive; (3) the choking/incentive mechanism that defeats freeloaders; (4) integrity against corrupt and malicious peers.

1. Peer discovery: from central tracker to DHT

Naive approach: one central server holds the whole file and a list of who is downloading. New peers ask it for the file and a peer list.

Where it breaks:

  • If the server holds the file, it is the server model we already proved dies at 2.9 TB/sec. The entire premise fails the moment the coordinator also serves bytes.
  • Single point of failure and censorship. One box means one place to overload, block, or take down. Kill it and the swarm cannot grow.

Evolution A: a lightweight tracker that holds only peer addresses, never bytes. The tracker is a tiny HTTP (or UDP) service keyed by info_hash. Peers announce themselves (“I am at IP:port, downloading info_hash, I have X% done”) and get back a random slice of the current peer set. It never touches the file.

Tracker state:  info_hash -> { (peer_ip, port, last_seen, is_seeder), ... }
announce(info_hash, peer_id, ip, port, left_bytes):
    add/refresh this peer in the set (TTL ~45 min; expire dead peers)
    return random sample of ~50 peers from the set

This is cheap (kilobytes/sec, from the BoE) and scales by sharding trackers on info_hash - each tracker owns a slice of the file space. But it still has a coordination single-point-of-failure per swarm: if the tracker for a hot file dies, new peers cannot find the swarm even though the swarm’s bytes are fully distributed.

Where the tracker still breaks:

  • Tracker outage isolates newcomers. Existing peers keep trading, but no new peer can join, and the swarm slowly dies as peers leave and are not replaced.
  • It is a censorship / liability chokepoint. One address to block.

Evolution B: a Distributed Hash Table (Kademlia DHT) - trackerless discovery. The peer list itself becomes distributed data stored across the peers, using a structured overlay. Each peer has a random 160-bit node ID. The DHT maps info_hash -> list of peers and stores that mapping on the DHT nodes whose IDs are closest (by XOR distance) to the info_hash.

Kademlia:
  node_id, info_hash both 160-bit. distance(a,b) = a XOR b.
  Each node keeps a routing table of k-buckets: known nodes bucketed by
  distance, so it knows many close nodes and few far ones (O(log N) knowledge).

  To find peers for a file:
    get_peers(info_hash):
      query the ~k nodes in my routing table closest to info_hash
      each returns either the peer list (if it stores it) OR
        closer nodes to ask next
      iteratively hop closer until you reach the nodes storing the peer list
      -> O(log N) hops to resolve, N = total DHT nodes (tens of millions)

    announce_peer(info_hash, port):
      store "I am a peer for info_hash" on the k nodes closest to info_hash

Now discovery has no central component at all: the “tracker” is spread over the same peers that hold the file, self-healing as nodes join and leave (k-buckets refresh, stored keys re-replicate to the new closest nodes). A magnet link carries just the info_hash and DHT bootstrap; even the metadata (piece hashes) is fetched from peers over an extension protocol. Real clients use both: trackers for fast warm-start, DHT + PEX (peer exchange, where connected peers gossip each other’s peer lists) for resilience and trackerless operation.

Dimension Central tracker Kademlia DHT
Central point yes (per swarm) none
Lookup cost O(1) request O(log N) hops
Failure mode swarm can’t grow if it dies self-healing
Warm-start speed fast slower first lookup
Censorship resistance low high
Who uses it classic .torrent magnet links, modern clients

The decision: tracker for speed, DHT for survival, PEX to fatten peer lists cheaply - run all three. The tracker gives sub-second warm-start; the DHT guarantees that no single box’s death can isolate a swarm whose bytes are already fully distributed.

2. Chunking and piece selection: rarest-first

Naive approach: download the file sequentially, byte 0 to end, from whichever peer will send it.

Where it breaks:

  • No parallelism. Pulling one contiguous stream from one peer caps you at that peer’s upload rate. The whole win of P2P - fetching different parts from many peers at once - is thrown away.
  • You have nothing to upload for a long time. Until you finish the early bytes, you cannot serve any later part, so you contribute almost nothing back and the swarm’s capacity does not grow with you.
  • The endgame stalls. The last few bytes come from whoever happens to have them; if that peer is slow, you wait.

Evolution A: split into fixed-size pieces, download out of order, in parallel. The file is N pieces (10,240 of 1MB each). A peer requests different missing pieces from different connected peers simultaneously, each piece further split into 16KB blocks pipelined per connection (so a slow round-trip does not idle the link). As soon as any single piece is complete and hash-verified, it is uploadable - so a peer starts contributing within seconds, not after finishing.

But which piece to request first matters enormously.

Where naive piece order (in-order or random) breaks:

  • Rare pieces vanish. Suppose one piece exists on only a single seeder. If everyone downloads in-order or randomly, that rare piece stays rare; if the lone holder leaves before enough peers copy it, the swarm can never complete the file - it is permanently missing a piece even though thousands of peers have all the others. This is the swarm-death failure mode.
  • Upload load is lumpy. Everyone wanting the same common early pieces means the few peers holding the rare later pieces are underused.

Evolution B: rarest-first piece selection. Each peer tracks, from the bitfield and HAVE messages of its neighbors, how many of its connected peers have each missing piece. It requests the rarest piece first - the one held by the fewest neighbors.

For each missing piece p:  availability[p] = count of connected peers that have p
Request order: ascending availability (rarest first), ties broken randomly.

Effect: the scarcest pieces get replicated fastest, so no piece drifts toward
extinction. The single-seeder's rare piece is grabbed early and spread, so the
seeder can leave and the swarm still holds a complete collective copy.

Two important refinements:

  • Random-first-piece exception. A brand-new peer with nothing has no pieces to trade, so it grabs its first piece at random (fastest to obtain, to become a participant quickly) and switches to rarest-first afterward.
  • Endgame mode. For the last few missing pieces, the peer broadcasts requests for their remaining blocks to all peers that have them and cancels the duplicates as they arrive - so the download does not stall on one slow peer at the finish line.

Rarest-first is the mechanism that makes the swarm’s durability statistical-but-robust: piece availability is actively equalized, so the collective copy survives churn even when no single peer has everything.

3. Choking: the incentive that defeats freeloaders

Naive approach: upload to whoever asks, equally. Be a good citizen, serve all requesters.

Where it breaks:

  • Freeloaders win. A peer can configure itself to download greedily and upload nothing (a “leech”). If honest peers upload to everyone regardless, leeches get full speed for free, honest peers’ upload is diluted, and the rational strategy becomes don’t upload - which, if everyone adopts it, collapses the swarm to zero serving capacity. This is a tragedy-of-the-commons.
  • Your scarce upload slot is wasted on peers who will never reciprocate.

Evolution: tit-for-tat choking. Each peer has a small number of upload slots (say 4 “unchoked” peers it will actively send to). It decides who to unchoke based on who uploads to it fastest - reciprocity, not fairness.

Every 10 seconds (rechoke round):
  measure download rate received FROM each connected peer over the last window
  UNCHOKE the top-k peers by the rate they gave ME  (default k = 4)
  CHOKE everyone else (refuse to send them pieces right now)

Every 30 seconds (optimistic unchoke):
  additionally unchoke ONE random choked peer, regardless of their rate
  -> lets you discover new good partners, and gives newcomers (who have
     nothing yet, so can't reciprocate) a foot in the door to get started

The mechanics and why they work:

  • Reciprocity is enforced locally, with no global reputation system. Each peer independently rewards the peers giving it the most bandwidth. A leech that uploads nothing gets choked by everyone and drops to a trickle (only the occasional optimistic-unchoke crumb). Uploading becomes the rational, self-interested strategy - the only way to reliably get unchoked by good peers.
  • Optimistic unchoke bootstraps newcomers and discovers better peers. A peer with zero pieces cannot reciprocate yet; the periodic random unchoke gives it a first piece so it can enter the tit-for-tat economy. It also probes for peers who might be faster than your current top-4.
  • Seeders have no one to reciprocate with, so they choke differently: they unchoke the peers that download from them fastest (maximizing the swarm’s overall throughput), rotating slots so many peers benefit.
  • Anti-snubbing. If a peer that was unchoking you suddenly stops sending, you snub it (deprioritize) and lean on optimistic unchokes to find a replacement.

Choking is the game-theoretic heart of BitTorrent: it makes selfish peers cooperate by tying the bandwidth you receive to the bandwidth you give, with just enough randomness (optimistic unchoke) to onboard newcomers and explore. No central bandwidth accounting, no payments - just a local, per-peer, 10-second reciprocity loop.

4. Integrity against corrupt and malicious peers

Naive approach: trust the bytes each peer sends; assemble them.

Where it breaks:

  • One malicious peer poisons the file. A peer can send garbage or subtly altered bytes for a piece. Without verification you assemble a corrupt file, and worse, you then serve that corrupt piece to others - poison spreads through the swarm.
  • Silent bit-rot / transmission errors also corrupt pieces with no way to localize the damage.

Evolution A: per-piece cryptographic hashes in the metadata. The metadata file lists the expected hash of every piece (that 320KB hash section from the BoE). When a peer finishes downloading piece i, it hashes the bytes and compares to piece_hashes[i].

on piece i complete:
    if hash(piece_bytes) == metadata.piece_hashes[i]:
        accept, set bitfield bit i, broadcast HAVE i, serve it onward
    else:
        discard the whole piece, penalize/disconnect the sender,
        re-request piece i from a DIFFERENT peer

Because the file is content-addressed - the info_hash is itself a hash over the metadata (including all piece hashes) - a peer that has the correct magnet link’s info_hash can trust the piece-hash list, and thus verify every piece against a sender it does not trust. Integrity does not require trusting peers; it requires trusting the hash. A malicious peer’s corrupt piece simply fails verification, is thrown away, and the sender gets penalized. Poison cannot spread because no peer forwards an unverified piece.

Evolution B: block-level detail and Merkle trees (BitTorrent v2). Two refinements matter at scale:

  • Verify at piece granularity, request at block granularity. You can only hash-check a whole piece, so a single bad 16KB block wastes the whole 1MB piece re-download. v2 addresses this with per-file Merkle trees: the metadata stores only the Merkle root, and each block arrives with a Merkle proof, so you can verify each block independently and localize corruption to the exact bad block - no whole-piece re-fetch, and much smaller metadata for huge files (a root hash instead of thousands of piece hashes).
  • Metadata authenticity. The piece hashes protect content integrity, but who signs the metadata? For untrusted distribution you pair the torrent with a publisher signature over the info_hash, so peers know the hash list itself came from the real publisher, not an impostor swarm.

The integrity model in one line: content-addressing turns “trust the peer” into “trust the math” - every piece (or block) is verified against a hash rooted in the metadata’s info_hash, so corrupt or malicious bytes are detected on arrival, discarded, and never propagated.

API Design & Data Schema

Most of the system is a peer wire protocol (peers talking to peers) plus a thin tracker/DHT API. There is no request/response app server holding content.

Tracker API (HTTP or UDP)

GET /announce?info_hash=<20B>&peer_id=<20B>&ip=<addr>&port=<n>
             &uploaded=<n>&downloaded=<n>&left=<n>&event=<started|stopped|completed>
  -> register/refresh this peer in the swarm; report progress.
  Response (bencoded): {
     "interval": 1800,                      // re-announce every 30 min
     "peers": [ {"ip":"1.2.3.4","port":6881}, ... ~50 random peers ],
     "complete": 4210,                       // seeders in swarm
     "incomplete": 88123                     // leechers in swarm
  }

GET /scrape?info_hash=<20B>
  -> swarm stats without joining: { "complete": .., "incomplete": .., "downloaded": .. }

DHT RPCs (Kademlia, over UDP)

ping(node_id)                         -> liveness check
find_node(target_id)                  -> k closest nodes to target_id (routing)
get_peers(info_hash)                  -> peer list if stored here, else k closer nodes
                                         + a write "token"
announce_peer(info_hash, port, token) -> store "I am a peer for info_hash" here

Peer wire protocol (the data plane, over TCP or uTP)

handshake:  <pstr "BitTorrent protocol"><reserved 8B><info_hash 20B><peer_id 20B>
            (both sides verify info_hash matches; mismatch -> drop)

Then length-prefixed messages:
  bitfield        : which pieces I have (one bit per piece)         [sent right after handshake]
  have <i>        : I just completed and verified piece i
  interested / not_interested : I do / don't want a piece you have
  choke / unchoke : I will not / will now serve you (the incentive control)
  request <i,off,len>  : send me block (piece i, offset off, length len=16KB)
  piece   <i,off,data> : here is that block
  cancel  <i,off,len>  : withdraw a request (used in endgame)
  port    <n>          : my DHT port (for DHT bootstrapping)

Metadata / data schema

The metadata file (.torrent) is the schema of the content. It is immutable and content-addressed:

info dictionary (the part hashed to produce info_hash):
  name          STRING     -- file or directory name
  piece_length  INT        -- e.g. 1,048,576 (1 MB)
  pieces        BYTES[]    -- concatenated SHA hashes, 20B (v1) or 32B (v2) per piece
  length        INT        -- total file size (single-file mode)
  files[]       LIST       -- {length, path[]} per file (multi-file mode)
outer:
  announce      STRING     -- primary tracker URL
  announce_list LIST       -- backup trackers / tiers
  info_hash     = HASH(bencode(info))   -- THE swarm id, also the content address

Tracker peer store (in-memory KV, sharded by info_hash):

Key:   info_hash (20 bytes)
Value: hash-set of peers -> { peer_id, ip, port, is_seeder, last_seen_ts }
  TTL per peer entry ~45 min; expire on missed re-announce.
  Shard by: info_hash  -> each swarm's peer list lives on one tracker shard.

Why NoSQL / in-memory, not SQL: the tracker workload is a pure key-value lookup (info_hash -> peer set) with high churn, TTL expiry, no joins, and no cross-key transactions - a perfect fit for a sharded in-memory store (Redis) partitioned by info_hash. The file content is never in any database - it is content-addressed and lives in the peers, verified by hash. There is no mutable relational state anywhere in the design that would justify an ACID SQL store; the only “consistency” requirement (piece integrity) is enforced cryptographically at the edge, not by a database.

Bottlenecks & Scaling

Where it breaks first, in order, and the fix for each:

1. Swarm bootstrap - the first few peers and the lone origin seeder (breaks first). At launch there is exactly one seeder and everyone wants everything from it. Until enough distinct pieces are spread across peers, the origin is a temporary bottleneck. Fix: the origin super-seeds - it hands each connecting peer a different rare piece rather than the same early pieces, maximizing distinct-piece spread so peers can immediately trade with each other. Once every piece exists on at least a few peers (a “warm” swarm), the origin’s share of upload drops toward its steady-state near-constant contribution. Seed a handful of full copies (from BoE, ~50GB total) to be safe.

2. Piece rarity / swarm death from churn. A piece held by too few peers can go extinct if those peers leave, permanently breaking completion for everyone. Fix: rarest-first selection actively replicates the scarcest pieces, and the publisher keeps a permanent origin seeder guaranteeing >= 1 full copy always exists. Monitoring watches the minimum piece availability across the swarm; if it approaches 1, the origin re-injects the endangered pieces.

3. Freeloaders draining upload capacity. Leeches that download without uploading shrink the swarm’s effective serving capacity. Fix: tit-for-tat choking - each peer serves its upload slots to whoever reciprocates fastest, choking non-contributors down to optimistic-unchoke crumbs. Uploading becomes the only reliable way to get good download speed, so selfish peers cooperate.

4. Tracker as a coordination single point of failure / hot key. A hot file’s tracker shard sees all its announces; if it dies, newcomers cannot join. Fix: shard trackers by info_hash (each shard owns a slice of files), replicate each shard, and back the whole thing with the DHT + PEX so discovery survives any tracker outage - the peer list is also spread across the peers themselves. A hot info_hash is a hot key, but the payload (address lists) is kilobytes, so replicas absorb it trivially.

5. Peer connection / NAT limits. A peer behind NAT cannot accept inbound connections, halving the connectable pairs; connection counts also cap how many peers you can trade with. Fix: NAT traversal (UPnP port mapping, hole-punching, and relay fallback) so NATed peers can still be dialed; keep each peer’s active set small (~20-80 connections) and rely on PEX to keep discovering fresh peers as ones churn out. The swarm does not need any peer connected to all others - a sparse, well-mixed graph suffices for pieces to flow.

6. Corrupt / malicious pieces poisoning the file. An attacker floods bad bytes. Fix: per-piece (v1) or per-block Merkle-proof (v2) hash verification against the metadata rooted in info_hash. Bad pieces fail verification, are discarded, the sender is penalized/disconnected, and nothing unverified is ever forwarded - poison cannot spread.

7. Metadata distribution at launch. If millions grab a .torrent from one web server, that server is a mini version of the original problem. Fix: magnet links carry only the tiny info_hash; the full metadata (piece hashes) is fetched from peers via a wire-protocol extension and verified against the info_hash. The metadata itself becomes P2P-distributed, so no central metadata host is stressed.

8. Shard key, stated plainly. Tracker peer store and DHT keyspace both shard on info_hash - the swarm is the unit of coordination, so co-locating one file’s entire peer set on one shard (tracker) or on the DHT nodes closest to its info_hash (DHT) is exactly right for the only query that exists (“who is in this swarm”). Sharding by peer IP or by geography would scatter a single swarm’s peer list and turn one lookup into a scatter-gather - info_hash is the only correct key because the file is the unit of discovery.

9. Global distribution / cross-region locality. Peers on opposite sides of the planet trade slowly over high-RTT links. Fix: this is best-effort and self-optimizing - choking naturally prefers faster (usually nearer) peers because they deliver higher rates, so the tit-for-tat loop clusters trading among low-latency partners without any explicit geo-routing. Some clients add IP-locality hints so trackers return nearby peers first, cutting cross-region transit.

10. Single points of failure, overall. Trackers are sharded and replicated and are backstopped by the DHT; the DHT is inherently decentralized and self-healing; the origin seeder is the only deliberately-permanent node and even it can leave once the swarm collectively holds every piece. There is no box whose loss loses the file as long as the union of live peers’ pieces covers all 10,240 - durability is a property of the swarm, not of any server.

Wrap-Up

The trade-offs that define this design:

  • Peers are the infrastructure, not servers. We invert the cost curve so serving capacity grows with demand - the origin’s egress collapses from ~10 PB to ~50 GB (a 200,000x reduction) because every downloaded piece instantly becomes an upload source. The cost is that we give up central control and pay for it with a coordination layer (tracker + DHT) and edge-enforced integrity.
  • Decentralized discovery over a central coordinator. We run a fast tracker for warm-start but backstop it with a Kademlia DHT and PEX so no single box can isolate a swarm whose bytes are already fully distributed - trading O(log N) lookup latency for censorship resistance and self-healing.
  • Rarest-first over sequential/random piece order. We actively equalize piece availability so the scarcest pieces never drift to extinction under churn, making swarm durability statistical-but-robust - at the cost of never having a usable prefix mid-download (which is why streaming needs a different, sequential bias).
  • Tit-for-tat choking over altruistic uploading. We make selfish peers cooperate by tying received bandwidth to given bandwidth, with optimistic unchokes to onboard newcomers - trading a globally-optimal fairness model for a simple, local, incentive-compatible one that actually survives real leeches.
  • Trust the hash, not the peer. Content-addressing (per-piece or Merkle-proof-per-block hashes rooted in info_hash) lets peers verify bytes from total strangers, so corruption and poisoning are caught on arrival and never propagate - integrity depends on cryptography, not on trusting anyone in the swarm.

One-line summary: a decentralized swarm where a file is content-addressed and split into hash-verified pieces, peers discover each other through a sharded tracker plus a Kademlia DHT (never a byte-serving server), every peer simultaneously downloads rarest-pieces-first and uploads to whoever reciprocates fastest under tit-for-tat choking, and integrity is enforced by hashes rooted in the info_hash - so the origin injects a few copies once and the swarm’s own upload capacity multiplies to serve millions, with durability guaranteed by the collective set of pieces across all live peers rather than by any central server.