Instagram sounds like Twitter with pictures, and half of it is. The follow graph, the feed of people you follow, the celebrity who breaks fan-out - all of that is the same problem the news feed poses, and I will not pretend otherwise. But the other half is the part people wave away in interviews and then drown in: a photo is not 280 bytes of text, it is a 3MB blob that has to be uploaded over a flaky phone connection, resized into five renditions, stored durably forever, and then served from an edge cache to millions of people in under 100ms. At 100M uploads a day that media pipeline is its own distributed system, and it is where this design actually gets hard.

So I will spend most of the depth on the parts that are genuinely different from a text feed: the upload path, the transcoding pipeline, blob storage, and CDN delivery. The feed fan-out I will design properly but faster, because it is a known shape. The one decision that colors everything: photo bytes never touch the application servers or the database - they go straight to object storage and are served straight from a CDN, and the app tier only ever moves small metadata around. Get that boundary right and the system scales; get it wrong and your app servers become a 172 GB/sec bottleneck.

Functional Requirements (FR)

In scope:

  • Upload a photo. A user uploads an image with an optional caption. It gets stored durably and becomes visible on their profile and in their followers’ feeds.
  • Home feed. Given a user, return a reverse-chronological-ish feed of recent photos from the accounts they follow, paginated.
  • Follow / unfollow. A user follows or unfollows another account; this changes whose photos appear in their feed.
  • User profile / photo grid. The list of a single user’s own photos, newest first - a simpler secondary read.
  • View a photo. Serve the right-sized image fast (thumbnail in the grid, medium in the feed, full on tap).

Explicitly out of scope, said out loud so I control the scope:

  • Stories, Reels, video, live. Video transcoding is a much bigger animal; I am designing photos.
  • Likes, comments, DMs, search, explore, ads, notifications. Each is its own system. I will mention like-counts only where they affect ranking.
  • Content moderation, spam, and copyright detection. They run as async consumers off the upload pipeline; I am not designing the classifiers.
  • The ranking model internals. I treat feed ranking as a thin read-time layer over a candidate set, same as any feed.

The single decision that drives the architecture: this is read-heavy and byte-heavy. Reads (feed loads and image fetches) dominate writes (uploads) by ~100:1 on request count and by an enormous factor on bytes, and the bytes are large and immutable. Immutable large objects want object storage and a CDN; small mutable metadata wants a database and a cache. Keeping those two worlds separate is the whole game.

Non-Functional Requirements (NFR)

  • Scale: 500M DAU. 100M photo uploads/day. Average user follows a few hundred accounts with a long, heavy tail (celebrities with 100M+ followers). Photos are stored forever.
  • Latency: feed metadata read p99 under 200ms. Image fetch (from CDN edge) p99 under 100ms. Upload acknowledgement to the user under ~1s once the bytes land, with transcoding finishing asynchronously within a few seconds.
  • Availability: 99.99% on the read paths (feed and image serving). A blank feed or a broken image is the most visible failure the product has. Upload can tolerate slightly lower availability since it is retryable from the client.
  • Consistency: eventual for the feed and follower counts. It is fine if a photo reaches a follower’s feed a few seconds late. Within a single user’s own view we want read-your-writes: after I upload, I must see my photo on my own profile immediately.
  • Durability: a stored photo must never be lost - this is the product’s memory. Blobs live in replicated/erasure-coded object storage with 11-nines durability. Metadata is replicated. Materialized feeds are a cache and can be rebuilt.
  • Freshness: a new photo should reach most followers’ feeds within a few seconds under normal load.

Back-of-the-Envelope Estimation (BoE)

Real numbers with the arithmetic, because the storage and bandwidth figures dictate the design.

Writes (uploads):

100M uploads / day
= 100,000,000 / 86,400 sec
≈ 1,160 uploads/sec  (call it ~1.2K WPS average)
Peak (3x average) ≈ 3,500 uploads/sec

Reads (feed loads):

500M DAU, each opening the app and refreshing several times.
Assume ~20 feed fetches/user/day (open, pull-to-refresh, scroll pages).
500M * 20 = 10B feed reads/day
= 10B / 86,400 ≈ 115,000 feed reads/sec average
Peak (3x) ≈ 350,000 feed reads/sec

Image fetches (the real read volume):

Each feed page shows ~10 photos; profile grids and taps add more.
Say ~30 image GETs per active session, ~3 sessions/user/day.
500M * 90 = 45B image fetches/day
≈ 520,000 image fetches/sec average, peak ~1.5M/sec.

Feed reads (metadata) are ~100x uploads. Image fetches are ~5x feed reads again and are all served by the CDN, not the origin. Two very different read tiers.

Storage - the number that shapes everything. We never store one image; we store a set of renditions plus the compressed original:

thumbnail (150px)     ~10 KB
small (320px)         ~40 KB
medium (640px)        ~120 KB
large (1080px)        ~350 KB
compressed original   ~1.2 MB
------------------------------
total per photo       ≈ 1.7 MB   (round to ~2 MB with metadata + overhead)

100M uploads/day * 2 MB ≈ 200 TB/day of new media
* 365 ≈ 73 PB/year

73 petabytes a year. That is the headline. It is why the blobs go into a purpose-built object store with erasure coding (not 3x replication - that would be 219 PB/year of raw disk) and why nothing about photo bytes ever goes near the database.

Metadata storage is tiny by comparison:

Per photo row: photo_id (8) + owner_id (8) + caption (~150)
             + blob key / URLs (~200) + dims/exif/ts (~50) ≈ ~450 bytes
100M/day * 450 B ≈ 45 GB/day ≈ 16 TB/year

16 TB/year of metadata versus 73 PB/year of pixels. This asymmetry is why we split the storage tiers.

Bandwidth (egress) - the CDN’s job:

Feed image serving: avg image served ~150 KB (mostly medium size).
520K fetches/sec * 150 KB ≈ 78 GB/sec average.
Peak ≈ 1.5M/sec * 150 KB ≈ 225 GB/sec.

Hundreds of GB/sec of egress. No origin fleet serves that; the CDN does, with an origin offload of well over 95%. Upload ingress is comparatively trivial: 3,500 uploads/sec * 3MB raw ≈ 10 GB/sec, and it lands directly in the object store.

Feed cache size: the feed is a list of photo IDs (8 bytes), not images. Cache ~500 recent IDs per active user.

500 IDs * 8 B ≈ 4 KB/user (round to ~8 KB with overhead)
Only active users (500M DAU): 500M * 8 KB ≈ 4 TB of feed cache RAM.

~4 TB across a Redis cluster - large but standard. Store IDs, not photos, and only for active users.

High-Level Design (HLD)

Three paths that could not be more different in cost: the upload path (low QPS but huge bytes and heavy async work), the feed metadata path (high QPS, latency-critical, tiny bytes), and the image delivery path (enormous QPS and bytes, served entirely from the edge). The trick is that they share almost nothing on the hot path.

                          +---------------------------+
                          |          Clients          |
                          |   (mobile, web, API)      |
                          +-------------+-------------+
                                        |
                          +-------------v-------------+
                          |       Load Balancer        |
                          +--+--------------+--------+-+
              UPLOAD path  |                |        | FEED path
          +----------------v-+          +---v--------v----+
          |  Upload Service   |          |  Feed Service    |
          | (presign, confirm)|          |  (stateless)     |
          +--+-------------+--+          +----+-------------+
             |             |                  | read precomputed
   1. presign|      3.confirm|                 | feed (photo IDs)
             |             | (metadata)   +----v-----------+
   +---------v---+   +-----v--------+     | Feed Cache      |
   | Object Store |   | Metadata DB  |     | (Redis: user    |
   | (blobs, EC,  |   | (photos,     |     |  -> [photo_id]) |
   |  presigned   |   |  sharded)    |     +----+-----------+
   |  PUT direct) |   +-----+--------+          | hydrate IDs
   +------+-------+         |                    |
          | 2. object       | emit         +-----v----------+
          |   created event | fanout job   | Metadata Cache  |
   +------v---------+  +-----v--------+     | (Redis: id ->   |
   | Transcode Queue|  | Fanout Queue |     |  photo meta)    |
   | (Kafka)        |  | (Kafka)      |     +-----------------+
   +------+---------+  +-----+--------+
          |                  | consumed
   +------v---------+  +-----v---------+     +----------------+
   | Transcode      |  | Fanout Workers |     | Social Graph   |
   | Workers        |  | (push id into  |---->| Service        |
   | (make          |  |  follower feeds)|     | (follow lists) |
   |  renditions)   |  +----------------+     +----------------+
   +------+---------+
          | write renditions back to Object Store
          v
   +----------------+        image GETs        +----------------+
   |  Object Store   | <--- origin fill ------- |      CDN        | <--- clients
   +----------------+                          +----------------+

Upload flow (post a photo):

  1. Client calls POST /photos/upload-url. Upload Service authenticates, assigns a photo_id (Snowflake, time-sortable), and returns a presigned PUT URL into the object store plus the photo_id.
  2. Client uploads the raw bytes directly to the object store using the presigned URL. The bytes never pass through the app tier. The client can resume/retry on a flaky network without involving our servers.
  3. Client calls POST /photos/{id}/confirm with the caption once the PUT succeeds. The object store also emits an “object created” event.
  4. Upload Service (or a consumer of that event) enqueues a transcode job on Kafka. It does not create the visible metadata row as “ready” yet - the photo is in PENDING state.
  5. Transcode Workers pull the raw blob, generate the five renditions, write them back to the object store under deterministic keys, and flip the metadata row to READY. Only then do they emit a fan-out job.
  6. Fan-out Workers read the author’s followers from the Social Graph Service and push photo_id into each follower’s cached feed - for non-celebrity authors only (the hybrid, below).

Feed read flow (the hot metadata path):

  1. Client GET /feed. Feed Service reads the user’s precomputed feed (a list of photo IDs) from the Feed Cache - one Redis read.
  2. Merge in recent photos from any celebrities the user follows (pulled at read time from those celebrities’ own cached photo lists). This is the hybrid.
  3. Rank the merged candidate IDs (thin read-time layer).
  4. Hydrate the top N IDs into photo metadata (owner, caption, rendition URLs) with a single batched MGET against the Metadata Cache.
  5. Return the page: JSON with, per photo, a set of CDN URLs for each rendition. The client picks the size it needs.

Image delivery flow (the byte-heavy path):

  1. Client requests an image URL (a CDN URL embedded in the feed response).
  2. CDN edge serves it from cache if present (>95% of the time). Images are immutable, so they cache forever.
  3. On a miss, the CDN fills from the object store origin, caches at the edge, and serves. The app tier is never involved.

The whole point of the HLD: the three paths are decoupled. Uploads go straight to blob storage. Feed metadata is small and precomputed. Images are served from the edge. The application servers only ever shuffle kilobytes of metadata, which is why a modest app fleet can front a petabyte-scale product.

Component Deep Dive

The hard parts, naive-first then evolved: (1) the upload and transcoding pipeline, (2) blob storage and the object store, (3) CDN delivery, renditions, and hot content, (4) feed fan-out and the celebrity split.

1. The upload and transcoding pipeline

This is the part that is genuinely different from a text feed, so it gets the most room.

Naive approach: upload through the app server, store in the DB

The obvious first cut: client POSTs the multipart image to an Upload Service, the service reads the bytes into memory, maybe stores them as a BLOB column in the database or writes them to local disk, synchronously resizes them, and returns 200.

Where it breaks, on every axis:

  • App servers become byte pipes. At peak, 3,500 uploads/sec * 3MB = ~10 GB/sec flowing through the app tier’s memory and NICs. You are now scaling stateless app servers for bandwidth, which is the most expensive possible reason to scale them, and a slow phone upload ties up a server thread for seconds.
  • BLOBs in the database is a catastrophe. A relational DB storing 73 PB of pixels destroys its buffer pool, makes every backup take days, and turns row reads into disk-thrashing. Databases are for small mutable rows, not immutable megabytes.
  • Synchronous transcoding blocks the request. Making five renditions of a 3MB image takes hundreds of ms to seconds of CPU. Doing it inline means the user’s upload hangs on your CPU, and a burst of uploads saturates the app CPU and stalls unrelated requests.

Everything about the naive path conflates three jobs - moving bytes, storing bytes, and processing bytes - that must be separated.

Evolved approach: direct-to-blob upload + async transcoding

Split the three jobs cleanly.

Move bytes: presigned direct upload. The client asks for a presigned URL and PUTs the raw bytes straight to the object store. Our servers issue a short-lived signed URL (a cheap metadata operation) and are otherwise uninvolved in the byte transfer. This removes the 10 GB/sec from our app tier entirely and lets the client use the object store’s own resumable/multipart upload for flaky networks.

Store bytes: object store, never the DB. The raw upload lands under a key like raw/{photo_id}. The DB only ever holds the key and derived URLs, never pixels.

Process bytes: asynchronous transcoding off a queue.

on object-created(raw/{photo_id}):
    enqueue transcode job {photo_id}

transcode worker:
    img = objectstore.get(raw/{photo_id})
    validate(img)                       # real image? size limits? strip EXIF GPS
    for size in [150, 320, 640, 1080]:
        rendition = resize(img, size)
        objectstore.put(f"{photo_id}/{size}.jpg", rendition)
    objectstore.put(f"{photo_id}/orig.jpg", recompress(img))
    metadata_db.update(photo_id, state=READY, dims=..., urls=...)
    enqueue fanout job {photo_id, owner_id}   # only now is it feed-visible

Transcoding is a horizontally scaled, CPU-bound worker pool that scales independently of the app tier. A backlog delays feed visibility by seconds; it never affects upload acknowledgement or the read path.

The consistency problem this creates: orphans and dangling rows. Splitting upload into “PUT the blob” and “confirm the metadata” over an unreliable network guarantees partial failures:

  • Orphan blob: client PUTs the bytes, then dies before calling confirm. A blob exists with no metadata row. Fix: a background GC sweeper deletes raw/{photo_id} objects older than N hours that have no READY/PENDING metadata row. Cheap and safe because the object store lifecycle policy can do most of it.
  • Dangling metadata: we create a PENDING row, but transcoding never completes (worker crash). Fix: transcode jobs are idempotent and retried; a row stuck in PENDING past a timeout is either retried or reaped. The row is not feed-visible until READY, so a stuck upload is invisible, not broken.
  • Read-your-writes for the author: the uploader must see their own photo immediately even before fan-out. We insert the photo into the author’s own photo-list cache synchronously at confirm time (or optimistically render client-side from the local file) so the author’s profile shows it instantly, while followers get it via async fan-out.

The state machine - PENDING -> READY -> (optionally) FAILED - plus idempotent retriable workers and a GC sweeper is what makes an unreliable two-step upload reliable overall.

2. Blob storage and the object store

Naive approach: a big NFS / local-disk file server

Store renditions as files on a fleet of file servers behind a path like /photos/2026/07/12/{id}.jpg.

Where it breaks: you now own replication, rebalancing, failure recovery, and capacity planning for 73 PB/year of files. A single directory with millions of entries thrashes the filesystem. When a disk dies you scramble to re-replicate by hand. Hot files overload whichever server holds them. This is re-implementing an object store, badly.

Evolved approach: a managed/dedicated object store with erasure coding

Use an S3-class object store (managed, or an internal Haystack/Ceph-style system at this scale). Properties that matter:

  • Erasure coding, not 3x replication. Storing 73 PB at 3x is 219 PB of disk. Erasure coding (e.g. 10+4) gives comparable durability at ~1.4x overhead, so ~102 PB. At petabyte scale the storage-efficiency difference is millions of dollars, so this is a real decision, not a detail.
  • Flat keyspace, deterministic keys. Keys are {photo_id}/{size}.jpg. photo_id is a Snowflake, so keys are effectively uniformly distributed across the store’s internal shards - no hot directory. Deterministic keys mean we do not even need to store every URL in the DB; we can derive them from photo_id and size.
  • Immutability. A rendition, once written, never changes. This is the property that makes CDN caching and edit/delete handling trivial: to “edit,” you write a new object under a new id; to delete, you tombstone the metadata and let a sweeper reclaim the blobs.
  • Tiered storage for cost. Photos have a brutal recency curve: the vast majority of fetches hit content from the last few days. Move renditions older than, say, 90 days to a colder, cheaper storage tier (still durable, slightly higher latency), keeping hot recent content on fast storage. This alone saves a fortune across 73 PB/year.

Small-file overhead is worth a note: an object store optimized for many small objects (Haystack’s original motivation at Facebook) avoids the per-file inode/metadata overhead that a naive filesystem pays on billions of thumbnails. At this scale you either pick a store designed for small immutable objects or you batch small renditions into larger container files with an offset index.

3. CDN delivery, renditions, and hot content

The 78-225 GB/sec of egress is the single largest resource in the system, and it must never touch the origin more than it has to.

Naive approach: serve images from the origin app/object tier

Client hits GET /image/{id} on our servers, we stream the blob back.

Where it breaks: 520K image fetches/sec at 150KB each is ~78 GB/sec average, ~225 GB/sec at peak, and it is globally distributed. Serving that from a central origin means impossible egress cost, terrible latency for far-away users, and an origin fleet melting under a viral photo where one image gets millions of requests in minutes.

Evolved approach: CDN everything, exploit immutability, right-size at the edge

  • CDN in front of the object store. Image URLs point at the CDN. Edges cache renditions and serve them locally. Because renditions are immutable, cache TTLs are effectively infinite, giving a >95% edge hit rate and cutting origin egress to under 5% of total.
  • Serve the right rendition, never the original. The feed uses 640.jpg, the grid uses 150.jpg, a tap loads 1080.jpg. Precomputing sizes means we never ship a 1.2MB original where a 120KB medium will do - this is where most of the bandwidth savings live. Modern formats (WebP/AVIF) shave another 30-50% and can be negotiated via Accept headers at the edge.
  • Hot content / viral photo (the read hot key). A single celebrity photo can draw millions of fetches in minutes, all for the same object. The CDN already absorbs most of this because it is one cache key replicated across every edge worldwide - the hotter the object, the more edges hold it. For the sliver that reaches the origin, the object store shards by photo_id so different photos land on different shards; a single mega-hot object is protected by an origin shield layer (a mid-tier cache) so at most a handful of origin reads happen per object globally. Immutability is what makes all of this safe: no invalidation, no staleness, cache forever.
  • Upload-side edge acceleration. The presigned PUT can target the nearest object-store region / upload edge so a user in Jakarta is not PUTting 3MB across an ocean before the app acknowledges.

The combination - CDN with infinite TTL on immutable renditions, right-sized images, and an origin shield for viral objects - is what turns a 225 GB/sec problem into a manageable one where the origin barely notices.

4. Feed fan-out and the celebrity split

This is the same class of problem as any social feed, so I will move fast, but it is core and must be designed.

Naive approach: fan-out on read (query at load time)

On feed load, read the user’s follow list, fetch recent photo IDs from every followee, merge-sort by time, return the top page.

Where it breaks: this runs on every one of 350K feed reads/sec, each a scatter-gather across hundreds of followees on different shards, recomputed every refresh even when nothing changed. p99 is uncontrollable and depends on how many accounts you follow. Cheap writes, ruinous reads - backwards for a read-heavy system.

Evolved approach: fan-out on write, hybridized for celebrities

Precompute each user’s feed. When a photo goes READY, push its photo_id into every follower’s cached feed list. A read becomes “fetch my list,” which is a single Redis LRANGE.

on photo READY(photo_id, owner_id):
    if follower_count(owner_id) < CELEB_THRESHOLD:
        for f in graph.get_followers(owner_id):
            feed_cache.lpush(f, photo_id)
            feed_cache.ltrim(f, 0, 500)     # cap length
    else:
        pass    # celebrity: merged at read time

Where pure fan-out-on-write breaks: cost is O(followers) per photo, and the follower distribution has a monstrous tail. A celebrity with 100M followers posting one photo is 100M feed writes. You cannot do that synchronously and should not do it at all.

The hybrid (the answer): fan out normal accounts on write, and merge celebrities in on read.

on load_feed(user):
    base = feed_cache.lrange(user, 0, 500)          # fanned-out photos
    celebs = graph.celebrities_followed_by(user)     # small set
    celeb_photos = [author_photos_cache.recent(c) for c in celebs]
    candidates = merge(base, flatten(celeb_photos))
    return rank_and_hydrate(candidates)
StrategyWrite costRead costBreaks onVerdict
Fan-out on readO(1) storeO(followees) scatter-gather per readHigh-follow users, read QPSToo slow to read
Fan-out on writeO(followers) per photoO(1) list fetchCelebrities (100M writes/photo)Great except the tail
Hybrid (write + celeb read)O(followers) normal, O(1) celebO(1) + small celeb mergeNothing at scaleThe answer

Store photo IDs only in feeds, never metadata or pixels; hydrate the top N with one batched MGET from the Metadata Cache at read time, so the same photo referenced by a million feeds is one metadata entry, not a million copies. Feeds are a cache: an evicted feed (inactive user returns) is rebuilt on demand via the fan-out-on-read query, then kept warm on write - which is why we only materialize the 4 TB for DAU, not all accounts. Ranking is a thin read-time layer over the candidate set with a reverse-chronological fallback if the ranker is slow, so the feed degrades but never blanks.

API Design & Data Schema

API

POST /api/v1/photos/upload-url
Response 200:
{
  "photo_id": "1789324411223300",
  "upload_url": "https://blob.cdn.example/raw/1789...?X-Signature=...",  // presigned PUT, short TTL
  "expires_in": 300
}

PUT  <upload_url>            # client uploads raw bytes DIRECTLY to object store
Body: <binary image>        # never passes through the app tier
Response 200 (from object store)

POST /api/v1/photos/{id}/confirm
Body: { "caption": "sunset over the bay" }
Response 202:
{ "photo_id": "1789324411223300", "state": "PENDING" }   // becomes READY after transcode
Errors:
  400  blob missing / not a valid image
  401  not authenticated
  409  already confirmed

GET /api/v1/feed?limit=20&cursor=<opaque>
Response 200:
{
  "photos": [
    {
      "photo_id": "...", "owner": { "id": "...", "handle": "..." },
      "caption": "...", "created_at": "...",
      "renditions": {
        "thumb":  "https://img.cdn.example/1789.../150.jpg",
        "small":  "https://img.cdn.example/1789.../320.jpg",
        "medium": "https://img.cdn.example/1789.../640.jpg",
        "large":  "https://img.cdn.example/1789.../1080.jpg"
      }
    }
  ],
  "next_cursor": "<opaque>"
}
Latency target: p99 < 200ms (metadata only; images fetched separately from CDN)

POST   /api/v1/follow    { "target_id": "999" }   -> 202 Accepted (async)
DELETE /api/v1/follow    { "target_id": "999" }   -> 202 Accepted
GET    /api/v1/users/{id}/photos?limit=30&cursor=..  -> profile grid

Pagination is cursor-based, not offset-based. Offsets duplicate or skip rows when new photos arrive between pages. The cursor encodes the last-seen photo_id; since IDs are Snowflake and time-sortable, it is a stable position in the stream.

Data store choices

The access patterns genuinely differ across four stores; be explicit about each and its shard key.

1. Object Store (blobs) - not a database. Renditions and originals, keyed {photo_id}/{size}.jpg, erasure-coded, immutable, tiered by age, fronted by the CDN. This holds the 73 PB/year. Covered above.

2. Photo Metadata - NoSQL wide-column (Cassandra / DynamoDB). Access is point-lookup by photo_id and range-scan of one owner’s recent photos. No joins, no cross-row transactions, needs horizontal write scale. That is the wide-column sweet spot.

Table: photos
  photo_id     BIGINT   PARTITION KEY   -- Snowflake, time-sortable
  owner_id     BIGINT
  caption      STRING
  state        ENUM     -- PENDING | READY | FAILED
  width        INT
  height       INT
  created_at   TIMESTAMP                -- also embedded in photo_id
  -- rendition URLs derivable from photo_id + size; store base key only

  Shard by: photo_id  (uniform, single-key reads land on one shard)

Table: user_photos          -- an owner's own photos, for grid + celeb read-merge
  owner_id    BIGINT   PARTITION KEY
  photo_id    BIGINT   CLUSTERING KEY DESC   -- newest first
  Shard by: owner_id
  Read pattern: "recent K photos by user X" = single-partition range scan

3. Social Graph - two adjacency lists. We need “who do I follow” (read-time celeb merge, feed rebuild) and “who follows X” (fan-out on write). One table cannot answer both directions efficiently, so store both.

Table: following            Table: followers
  user_id   PARTITION KEY     user_id   PARTITION KEY   -- the followed account
  followee  CLUSTERING KEY     follower  CLUSTERING KEY
  Shard by: user_id           Shard by: user_id

"who do I follow"  -> following[me]   (single partition)
"who follows X"    -> followers[X]    (single partition; huge for celebs)
follower_count[X]  -> maintained as an approximate async counter

A follow does two writes (both tables). Follows are rare relative to reads, so doubled write cost is fine.

4. Feed Cache and Metadata Cache - Redis. user_id -> [photo_id] capped lists (the precomputed feed, sharded by user_id), and photo_id -> photo_meta shared entries (sharded by photo_id, hydrated with one MGET per feed page).

Why NoSQL over SQL across the board: the workload is high-volume, partition-scoped reads and writes at hundreds of TB of metadata (and PB of blobs), with no need for cross-entity ACID transactions. SQL’s joins and transactions are exactly the features we do not need, and a single relational instance cannot hold or serve this. The one place strong consistency could matter (a payments ledger) does not exist in this problem. User accounts/auth, if you want strict guarantees there, can live in a separate small relational store - it is low-volume and unrelated to the media hot path.

Bottlenecks & Scaling

Where it breaks first, in order, and the fix for each:

1. Image egress bandwidth (breaks first and biggest - 225 GB/sec peak). No origin serves this. Fix: CDN in front of immutable renditions with effectively infinite TTL, >95% edge hit rate, right-sized images (medium in feed, thumb in grid), WebP/AVIF at the edge, and an origin shield for viral objects. This is the single most important scaling decision on the read side.

2. Fan-out write amplification on celebrities. One photo = up to 100M feed writes. Fix: the hybrid model. Do not fan out accounts above the follower threshold; merge them at read time. Tune the threshold by watching fan-out queue lag. This is the most important decision on the write side.

3. Upload bytes through the app tier (~10 GB/sec). Fix: presigned direct-to-blob upload. The app tier issues a signed URL and never touches pixels. Transcoding is an independent async worker pool off Kafka.

4. Storage cost and growth (73 PB/year). Fix: erasure coding instead of 3x replication (~1.4x vs 3x overhead), age-tiered storage (hot recent content on fast disk, older content on cold cheap tier), and a small-object-optimized store to avoid per-file overhead on billions of thumbnails.

5. Transcode queue backlog / thundering herd. A burst of uploads or a slow worker pool delays feed visibility. Fix: partition the transcode queue, scale workers horizontally on CPU, idempotent + retriable jobs (re-transcoding is a safe overwrite of deterministic keys), and a dead-letter queue for poison images. Backlog delays visibility by seconds; it never blocks upload ack or reads.

6. Feed read QPS (350K/sec). Fix: precomputed, cached feeds so a read is one Redis list fetch plus a small bounded celeb merge. Store IDs, hydrate metadata with one batched MGET.

7. Metadata / celeb read hot key. Millions hydrating the same celebrity photo’s metadata at once. Fix: replicate hot metadata keys across cache nodes (read a random replica), per-instance local LRU with a few-second TTL, and note the actual pixels are already served by the CDN, not this cache.

8. Shard keys, stated plainly: Object store -> photo_id (uniform, no hot directory). Photo metadata -> photo_id. User photos -> owner_id (range scan of one grid). Social graph -> user_id on both adjacency tables. Feed cache -> user_id. Sharding on created_at anywhere would be the classic mistake: it puts all current writes on the “now” shard and creates a permanent hot spot.

9. Single points of failure. Upload Service and Feed Service are stateless behind load balancers. Kafka, the caches, the metadata store, and the object store are all replicated/erasure-coded. Transcode and fan-out workers are horizontally scaled, idempotent consumer groups. No single box whose loss stops uploads or feeds.

10. Multi-region for a global user base. Fix: CDN edges are already global. Deploy Feed Service, caches, and object-store regions in multiple geographies routed by GeoDNS; presigned uploads target the nearest region. Because feeds are eventually consistent and rebuildable, cross-region metadata replication is forgiving. The author reads their own just-posted photo from their home region (read-your-writes on the user_photos path only).

Wrap-Up

The trade-offs that define this design:

  • Bytes and metadata live in separate worlds. Photo bytes go straight from client to object store to CDN and are never touched by the app tier or the database; the app tier only moves kilobytes of metadata. This one boundary is what lets a modest fleet front a petabyte-scale product.
  • Direct-to-blob upload + async transcoding. We accept the added complexity of a two-step upload (presign then confirm) and its orphan/dangling edge cases - handled by a state machine, idempotent workers, and a GC sweeper - to get the byte transfer and CPU-heavy resizing off the request path entirely.
  • CDN and immutability do the heavy lifting on reads. Immutable, right-sized renditions cached forever at the edge turn 225 GB/sec of egress into a >95% offloaded, low-latency problem, and make edits/deletes trivial (write a new object, tombstone the old).
  • Hybrid fan-out over either pure model. Normal accounts fan out on write for instant reads; celebrities merge in on read so one photo never triggers 100M writes. Store IDs, not photos, and dedup metadata behind a shared cache.
  • Eventual consistency, embraced, with read-your-writes where it matters. A photo reaching followers a few seconds late is free latency and availability; the only strict guarantee we buy is that the author sees their own upload immediately.

One-line summary: a read-heavy, byte-heavy photo service where uploads go directly to erasure-coded object storage and are transcoded asynchronously into right-sized immutable renditions served from a CDN, while small metadata is fanned out to precomputed ID-only feeds on write (celebrities merged on read) and hydrated in under 200ms at 350K feed reads per second.