Design a Distributed Job Scheduler
HardSystem design · Scheduling · Databricks
Design a distributed job scheduler and workflow executor, in the style of Airflow or an internal batch platform. Users submit graphs of tasks with dependencies, resource requirements and priorities; the system places ready tasks on a heterogeneous worker fleet, retries failures, shares capacity fairly between tenants, and tracks state. Recurring and delayed triggers are inputs to this system and can be treated as given (see Distributed Cron and Delayed Job Queue); the focus is dependency resolution, placement, leases, and fairness.
Requirements
- A task runs only after all dependencies succeed; reject cyclic graphs at submission
- Handle retries, idempotency, worker crashes and preemption without corrupting task state
- Track job state, logs, priorities and per-tenant quotas
- Match resource requirements (memory, GPU) to workers and apply backpressure when capacity is short