Design a Distributed Job Scheduler with Cron, Pause and Resume

Design a job scheduler service. Users register jobs that run once at a given time or repeatedly on a cron schedule, and they can pause a job and later resume it. Assume an unlimited pool of workers is available to execute jobs, so the design is about deciding when each job runs, dispatching it exactly when it is due, and keeping track of every run. ```hint Find the due jobs cheaply Think about how the scheduler finds the jobs due in the next second without scanning every registered job, and what it stores to make that lookup fast. ``` ```hint One run, one owner Several scheduler instances and retrying workers can all touch the same run. Decide what identifies a run uniquely and how ownership of it is claimed and released. ``` ### Constraints and Clarifications - Given: cron schedules, one-time jobs, pause and resume, and unlimited worker capacity. - Not given: the number of jobs, how precisely a job must start on time, how long jobs run, or what guarantee a run must have (at least once or exactly once). Ask, or state your assumptions. ### Clarifying Questions - How many jobs are registered, and how many fire at the busiest moment, such as the top of the hour? - How close to its scheduled time must a job start: within a second, or within a minute? - Is it acceptable for a run to happen twice after a failure, or must each scheduled occurrence run exactly once? - If a run is still going when the next occurrence is due, should the next one start, wait or be skipped? - What does pausing do to a run that is already executing, and on resume, should occurrences missed during the pause run? - In which time zone are cron expressions evaluated, and how should daylight-saving changes behave? ### What a Strong Answer Covers - Settles volume, start-time precision, run guarantees, overlap and misfire policy before designing - An API and data model for jobs, schedules, pause state and individual runs - A due-job lookup that does not scan all jobs, and a dispatch path from scheduler to queue to worker - Correct pause and resume semantics, including runs in progress and missed occurrences - Exactly-once or at-least-once behavior made explicit: claims, leases, idempotency keys, retries - Scaling the scheduler itself across instances, failure handling, and metrics such as dispatch lag ### Follow-up Questions - Ten million jobs are scheduled for midnight. How do you keep that moment from overwhelming the scheduler and the downstream systems? - A worker dies halfway through a run. How is that detected, and what happens to the run? - A job runs at 02:30 local time every day. What happens on the days the clocks change? - Users want job B to start only after job A succeeds. How do you add dependencies?