Design a Delayed Job Queue
MediumSystem design · Async Processing
Design a delayed job queue: producers enqueue one-off jobs to run immediately or at a specific future time (5 seconds to 6 months out), and a fleet of workers executes them. Jobs have priorities, are retried with backoff on failure, and move to a dead-letter queue when retries run out. Focus on how due jobs are found and claimed efficiently, not on recurring schedules (see Distributed Cron) or dependency graphs (see Distributed Job Scheduler).
Requirements
- Workers can crash mid-job; a job must run again, and duplicate execution must be safe (at-least-once, not exactly-once)
- Support visibility timeouts with lease extension, retries with backoff, priorities, and dead-letter queues
- Finding due jobs must not scan all pending jobs
- Jobs can be cancelled before they run