Design a Distributed Query Engine
HardSystem design · Query Execution · Snowflake
Design the execution engine of a distributed analytical SQL system (Presto/Trino/Spark SQL style) running on a cluster of ~100 workers over columnar files in object storage. Cover parsing and optimisation, turning the logical plan into distributed stages, scheduling tasks, shuffling intermediate data, join strategy and skew, handling worker failure and cancellation, memory limits, and debuggability. Table format and warehouse concerns (catalog, sharing, billing) are out of scope.
Requirements
- Support scans, filters, joins, aggregations, and result materialization
- Handle worker failures, retries, cancellation, and resource limits
- Mitigate skew and avoid excessive shuffle cost
- Discuss query coordinator, worker protocol, metadata, and observability