Design S3 (Object Storage) - System Design
An object store looks like a hash map with two methods: PUT(key, bytes) and GET(key) -> bytes. That framing is exactly why it is hard. A hash map lives in one process’s memory; this one holds trillions of objects and exabytes of bytes across hundreds of thousands of disks in dozens of datacenters, and promises that once a PUT returns 200 OK the bytes will still be there in a thousand years - eleven nines of durability, meaning if you store ten million objects you expect to lose one every ten thousand years. The naive version is a directory on a big disk behind an HTTP server, and it dies the first time a disk fails, the first time one key gets a billion reads, and the first time the index of keys outgrows a single machine. Everything real about S3 is the machinery that keeps a dumb key-value interface alive when the disk will fail, the datacenter will burn, and the keyspace will not fit anywhere. ...