Design a Job Scheduler with Durable State and Failure Recovery

Design a job scheduler. Explain how clients create work, how the system decides when work is eligible to run, how workers execute it, and how the scheduler handles failures. ### Constraints Job volume, duration, scheduling precision, recurrence, dependencies, and execution guarantees are not specified. Clarify them before choosing components. Any concrete workload or guarantee you use is an explicit design assumption. Treat the scheduler as a system-design problem rather than assuming a particular algorithmic scheduling objective. ### Clarifying Questions - Are jobs immediate, scheduled for a future time, recurring, or a mixture? - What lateness is acceptable, and what should happen to missed recurring runs? - Can jobs run more than once, and are their external effects idempotent? - Do jobs have dependencies, resource requirements, priorities, or tenant limits? ```hint Follow a worker failure Consider the interval after a job's side effect succeeds but before the worker records completion. ``` ### What a Strong Answer Covers - A job API and state model tied to stated scheduling requirements. - Durable eligibility tracking, dispatch, worker claims, retries, and observability. - Duplicate execution, ownership expiry, and scaling choices with explicit guarantees. ### Follow-up Questions - How would cancellation interact with a job that is already running? - How would you avoid a burst of due jobs overwhelming one tenant's dependency?