Design a Cron Job Scheduler for 100K Jobs With At-Least-Once Execution

Design a job scheduler service. Users register recurring jobs, each with a cron expression that says when it runs, and the service must start a run of the job every time its schedule fires. ### Constraints and Clarifications - About 100,000 job definitions. - Up to 10,000 job runs executing at the same time. - Every job is a cron job. There is no "run now" request: a job runs only when its schedule fires. - Execution is at least once: every scheduled run must eventually execute. A run may occasionally execute more than once, but it must never be silently skipped. - Not given: what a run does or how long it takes, how close to its scheduled time a run must start, the finest schedule granularity, and what happens to fire times missed during an outage. Ask, or state your assumptions. ### Clarifying Questions - What does running a job mean (calling an HTTP endpoint, starting a container, publishing a message), and how long does a typical run take? - How soon after its scheduled time must a run start: within a second, or within a minute? - Do schedules use minute granularity, like standard cron, or can a job fire every few seconds? - Is 10,000 the peak number of runs the system must support at once, or a cap it must enforce? - If a job's previous run is still executing when its next fire time arrives, should the new run start anyway, wait, or be skipped? ### Part 1 — Data model and API Define the entities and how they are stored, and the API for creating, updating, pausing and deleting jobs and for viewing past runs. Be ready to defend each entity. In particular, if your model keeps the schedule in its own entity, explain why it should not simply be stored with the job definition. ```hint Let the requirements size the model Under these requirements, how many schedules can one job have, and is there any run that does not come from a job's own schedule? Compare that with what you need to record each time a schedule fires. ``` #### What This Part Should Cover - The job definition, the per-run record, their keys, and where each job's next fire time is kept - A justified decision on keeping the schedule with the job or in its own entity, and when the other choice would be right - API endpoints for job management and run history - What happens to runs already planned when a job is edited, paused or deleted ### Part 2 — Finding due jobs and running them With 100,000 definitions and up to 10,000 runs in flight, how does the system find the jobs that are due, hand them to workers, and keep concurrency within that bound? What happens at a moment when thousands of jobs share the same fire time, such as the start of an hour? ```hint Make "what is due?" a cheap question Decide what the scheduler asks the database on every tick, and what lets that query avoid touching all 100,000 definitions. Then decide how several scheduler instances split the work without both starting the same run. ``` #### What This Part Should Cover - Detecting due jobs and computing each job's next fire time - Dispatching runs to a worker fleet and enforcing the concurrency bound - Running more than one scheduler instance for availability without double dispatch - Behavior and start delay when a large burst of jobs fires together ### Part 3 — Guaranteeing at-least-once execution Trace every point where a scheduled run could be lost: the scheduler crashing while it plans a run, a dispatch message that never arrives, a worker dying mid-run. Show how your design makes sure each scheduled run eventually executes, and where duplicate executions can then appear. ```hint Find the commit point Identify the single write after which a run is guaranteed to happen, and check that a crash immediately before or immediately after it is harmless. ``` ```hint Slow is not the same as dead A worker that stops responding may still be running the job. Decide how the system detects it, retries the run, and keeps the old worker from overwriting the new attempt's result. ``` #### What This Part Should Cover - Creating the run record atomically with advancing the job's schedule - Leases or heartbeats, timeouts, retries with backoff, and a final failed state with alerting - Where duplicates come from and how job handlers are protected from them - Fire times missed while the scheduler was down ### What a Strong Answer Covers - Clarifies run duration, start-time precision, and the overlap and missed-run policies before designing - A data model justified by these requirements rather than copied from a generic scheduler design - Rough numbers for fire rate, burst size, concurrency and run-history storage - An end-to-end argument that no scheduled run is lost, with duplicates acknowledged and contained - Observability: start delay, queue depth, runs in flight, retries and expired leases ### Follow-up Questions - Product later asks for a "run now" button. What changes in your data model and dispatch path? - A job's runs start taking longer than the interval between its fire times. What should happen, and who decides? - How do daylight-saving transitions affect jobs scheduled in a local time zone? - One job's side effect must never happen twice. What can the platform provide, and what must the job itself do?