Design a Lakehouse Table Storage Layer
HardSystem design · Lakehouse Storage · Databricks
Design a lakehouse table storage layer like Delta Lake or Iceberg. Data lives as immutable files in object storage, but users need ACID table transactions, schema evolution, time travel, compaction, and fast scans for analytics jobs.
In scope: the transaction log and snapshots, atomic commits and optimistic concurrency, checkpoints, file statistics and pruning, small-file compaction, vacuum and retention. Out of scope: the query engine and SQL planner.
Requirements
- Maintain an append-only, ordered transaction log and derive table snapshots from it
- Commit atomically with optimistic concurrency and file-level conflict detection; support rollback
- Schema enforcement and evolution, and per-file statistics for partition and data skipping
- Compact small files and vacuum unreferenced files only after the time travel window