Design a Distributed Job Scheduler with Run Logs and Exactly-Once Execution

Design a distributed job scheduler. Clients submit jobs to run at a specified time or on a recurring schedule; the system dispatches each due job to a pool of workers and lets clients check the status of every run. Two requirements go beyond a basic scheduler: the system must also store each job run's execution logs so they can be retrieved later, and the interviewer requires exactly-once execution. Be ready to discuss what exactly-once can mean when machines fail, and how your design provides it. ### Constraints and Clarifications - Job counts, schedule granularity, job durations, log sizes, and latency targets are not given. Ask for them, and keep the design parameterized instead of assuming numbers. - Treat a job's payload as opaque work that a worker runs. Jobs may have side effects outside the scheduler, such as calling another service or writing to a database. - Scheduler nodes and workers can crash, pause, or lose network connectivity at any moment, including in the middle of a job. ### Clarifying Questions - Are jobs one-time, recurring (cron-like), or both? Can a client pause, update, or cancel a job after submitting it? - When a run fails, should the system retry automatically, and up to what limit? - If the scheduler is unavailable when a run is due, should the missed run execute late, execute once for all missed occurrences, or be skipped? - What exactly must happen once: creating one run per scheduled occurrence, starting the job's code once, or producing its external effect once? Can the systems a job writes to accept an idempotency key or take part in a transaction? - Who reads the logs, how soon after a run must they be available (live tail or after completion), and how long must they be kept? ### Part 1 — Core Scheduler Design Design the APIs, data model, and components that accept jobs, determine when each run is due, dispatch runs to workers, track run state, and recover from worker failure. ```hint Separate the definition from its executions A recurring job produces many executions, and each execution may need several tries. Decide which record represents one scheduled occurrence. ``` #### What This Part Should Cover - APIs and a data model that distinguish job definitions, individual runs, and attempts. - How due runs are found and dispatched without a single-node bottleneck, and how two scheduler nodes are prevented from firing the same occurrence. - Worker claiming, liveness detection, timeouts, and retries of failed or abandoned runs. ### Part 2 — Storing Job Logs Extend the design so that the logs produced by every run are stored durably and can be fetched per run. ```hint Size the log path separately Log volume can far exceed the scheduler's metadata writes. Consider where large, append-heavy data should live and what small record links a run to it. ``` #### What This Part Should Cover - Where log content lives versus log metadata, and how they are linked. - How workers ship logs, including when a worker crashes mid-run and when a run has several attempts. - Retrieval, retention, and access control. ### Part 3 — Exactly-Once Execution The interviewer insists on exactly-once execution. Explain what the guarantee can and cannot mean when workers may fail at any point, then show how your design comes as close as possible and what a job must do to cooperate. If you propose deduplication, say precisely where and when it happens. ```hint Find the ambiguous crash Picture a worker that stops responding after it may or may not have performed the job's side effect. What can the scheduler know at that moment? ``` #### What This Part Should Cover - A precise definition of the guarantee and the failure case that prevents a scheduler alone from providing it for arbitrary side effects. - Mechanisms at each layer: one run per occurrence, one valid owner per run, and deduplicated or transactional effects. - What happens to jobs whose effects cannot be made idempotent, and how the position is explained to the interviewer. ### What a Strong Answer Covers - A coherent end-to-end flow from job submission to run completion, with explicit run states and transitions. - Failure handling for scheduler nodes, workers, the dispatch path, and log uploads, without silently lost or duplicated runs. - A precise, defensible position on exactly-once semantics that is neither a flat refusal nor an unsupported promise. - Scaling and observability: partitioning of due-time lookups, burst handling, lag and retry metrics, and stuck-run detection. ### Follow-up Questions 1. A worker's lease expires during a long garbage-collection pause, the run is reassigned, and the original worker then resumes. What prevents both from recording results or applying the effect twice? 2. How would you guarantee that a run's logs are complete whenever its status shows success? 3. How does the design change if a job's side effect is sending an email through a provider that offers no idempotency key? 4. How would you handle a large share of recurring jobs all being due at the top of the hour?