Design S3-Like Object Storage

Hard

System design · Storage

Design an S3-like object storage service. Customers PUT, GET, DELETE and LIST objects (1KB to 5TB) in buckets, upload large objects in parts, read byte ranges, and apply lifecycle rules. Focus on separating metadata from data, placement, erasure coding versus replication, durability (eleven nines) and repair, and consistency of overwrites and listings. Out of scope: billing, IAM policy language, and the CDN in front.

Requirements

Separate object data from metadata and shard both
Support multipart upload, range reads, versioning and lifecycle transitions
Reach very high durability with failure-domain-aware placement, checksums, scrubbing and repair
Define overwrite, delete and listing consistency

Workspace