# Design a Distributed Cron Job Scheduler Design a distributed scheduler for recurring jobs, similar in scope to a workflow-oriented cron service. Users register schedules and job definitions. The system determines when runs are due, dispatches work to a fleet of workers, tracks attempts, retries eligible failures, and exposes run status. Discuss both the control plane and execution path. ### Constraints & Assumptions - Scheduler and worker processes can crash or become partitioned. - A job run may take longer than its scheduling interval. - Delivery can be at least once; the system must make duplicate execution visible and manageable. - Schedule updates and cancellations race with run creation and dispatch. - Timezones, daylight-saving changes, backfills, and missed schedules require explicit policy. ### Clarifying Questions to Ask - Are jobs arbitrary containers, registered task names, or remote callbacks? - What delivery guarantee is promised, and must user code be idempotent? - What concurrency, dependency, priority, and retry controls are required? - How long must run history and logs be retained? ### Part 1: Data Model and APIs Define schedules, immutable run instances, attempts, leases, and state transitions, plus the APIs users and workers need. #### What This Part Should Cover - Versioned schedule definitions and deterministic run identities. - Separate run and attempt records with a valid transition model. - Idempotent create, claim, heartbeat, completion, retry, and cancellation operations. ### Part 2: Scheduling and Dispatch Explain how multiple scheduler replicas find due work without creating conflicting runs, and how workers claim and execute tasks. #### What This Part Should Cover - Ownership through partitioning, leases, or database locking. - Durable run creation before queue publication, with repair for partial failure. - Worker leases, heartbeats, visibility timeouts, and stale-attempt handling. ### Part 3: Reliability and Operations Address overload, retries, poison jobs, clock behavior, deployment, testing, and observability. #### What This Part Should Cover - Backoff, retry budgets, dead-letter handling, quotas, and fairness. - UTC storage plus explicit timezone evaluation and daylight-saving policy. - Reconciliation loops and invariant-based fault tests. - Metrics for scheduling delay, queue age, lease expiry, retries, and terminal outcomes. ### What a Strong Answer Covers - Durable, deterministic run creation and clear at-least-once semantics. - No claim of general exactly-once execution across arbitrary side effects. - Recovery paths for crashes between state changes and message publication. - Thoughtful handling of overlapping runs, updates, cancellation, time, and load. ### Follow-up Questions - How would you support dependencies between jobs without scanning all run records? - How would you backfill a month of missed runs without overwhelming normal traffic? - How would you isolate a tenant whose jobs repeatedly exhaust worker capacity?