Design a Cloud Data Warehouse
HardSystem design · Data Warehouse · Snowflake
Design a Snowflake-style cloud data warehouse for ~1PB of customer data. Users load data, run SQL, scale compute independently from storage (multiple isolated warehouses over the same tables), and share data securely with other accounts without copying it. Focus on the storage/compute split, the metadata catalog and pruning, caching, transactions, and cost controls. The internals of a single distributed query (joins, shuffle, skew) are covered by the query engine question.
Requirements
- Separate storage metadata from compute clusters
- Support columnar files, pruning, partition metadata, and result caching
- Isolate customer workloads and support elastic warehouses
- Discuss query planning, catalog metadata, transactions, and cost controls