Design an internal **job scheduling platform** for a large company. The platform lets internal services and engineers submit jobs that run at a specified time (one-time) or on a recurring schedule (cron-like). Your design must scale to roughly **10 million scheduled jobs**, execute jobs reliably, and expose operational visibility to both job owners and on-call engineers. Produce an end-to-end design that addresses: - **Functional requirements:** create, update, pause/resume, cancel, and inspect jobs; one-time and recurring jobs; dispatch ready jobs to workers; retry failed jobs with backoff; expose run history and logs. - **Non-functional requirements:** high availability, horizontal scalability, low scheduling latency, durability, idempotency, and observability. - **Components:** API design, data model, scheduler architecture, worker/executor design, failure handling, and monitoring. - **The crux:** explain precisely how the system avoids double execution while still recovering from crashes — scheduler crash after claim, worker crash mid-execution, queue redelivery, and network partitions. ```hint Where to start At 10M jobs you cannot table-scan to find what is due. What property would make "which jobs fire in the next few seconds?" a cheap lookup whose cost is independent of the total job count — and, separately, what happens to that lookup when one instant is far hotter than the average? Pin down both before you design tables. ``` ```hint One winner per run Once due-job lookup is cheap, two schedulers can still race for the same due run. Ask: what makes "I claimed this run" a decision the *database* arbitrates, so exactly one scheduler proceeds — and how does a *crashed* claimant's run become eligible again instead of being stuck forever? Those two requirements (single winner + recoverable) point at the shape of the claim. ``` ```hint Mind the gap between "decided" and "enqueued" There is a window between marking a run as taken in the database and actually publishing it to the queue. If the process dies inside that window, reason through both failure modes: what does a naïve "write, then publish" *lose*, and what does "publish, then write" *duplicate*? Once you see that neither ordering is safe on its own, ask what would let the *decision* and the *intent to dispatch* become durable as a single fact, with delivery handled separately and tolerant of replay. ``` ```hint Two different crashes, two different cures A scheduler crashing *before* the run reaches a worker is not the same failure as a worker crashing *mid-execution* — they leave the run in different states and need different recovery signals. Tease the two apart: what tells you a claimant is dead versus a live-but-slow worker, and what stops the side effect from happening twice if you do re-dispatch? Keep "recover from a crash" and "exactly-once" as separate goals. ``` ### Constraints & Assumptions - **Scale:** ~10 million registered jobs, a mix of one-time and recurring. Some minutes are "hot" — many jobs due at `:00` (midnight UTC, top-of-hour cron), so the design must absorb bursts, not just the average rate. - **Schedule types:** one-time (fire once at a timestamp) and recurring (cron-like expression with a timezone). - **Scheduling latency:** most due jobs should dispatch within a few seconds of their scheduled time (low lag, not hard real-time). - **Execution semantics:** treat **at-least-once dispatch with idempotent execution** as the practical target; true exactly-once across arbitrary side effects is out of scope to *guarantee* but should be approximated. - **Availability:** the control plane and scheduling plane must tolerate single-node failures without losing or duplicating runs. - **Job payloads:** a job may invoke a container, a service callback (HTTP/RPC), or a workflow. Treat the executor as pluggable; the scheduler runs no business logic itself. - **Multi-tenancy:** many internal teams share the platform; per-team quotas and isolation matter. ### Clarifying Questions to Ask - What is the smallest scheduling granularity (per-minute, per-second) and the acceptable dispatch lag? - Are recurring jobs allowed to overlap (a run still executing when the next fires), or must they be serialized per job? - On recovery from a long outage, should missed runs be backfilled, skipped, or capped at the most recent occurrence? - What execution targets must be supported (containers, HTTP callbacks, workflows), and who owns the worker fleet? - What are the durability and retention requirements for job definitions versus run history/logs? - Are there per-tenant limits on job count, submission rate, or concurrent executions? ### What a Strong Answer Covers - A clear split between the **control plane** (CRUD/API, metadata) and the **scheduling/execution plane** (due-job lookup, claim, dispatch, execute). - A **due-job lookup** that stays cheap regardless of total job count, with an explicit plan for hot minutes rather than the average case. - A **claim protocol** naming a concrete mechanism that guarantees a single winner per run and explains why losers back off — plus how a crashed claimant's run is recovered. - A direct answer to the **dispatch gap** between deciding and enqueuing, not hand-waving "then publish to the queue" — including which failure mode the chosen approach eliminates. - **Double-execution defenses** that hold up under each crash/redelivery case the crux lists, with a clear statement of what each layer buys. - **Recurring-job** correctness: computing the next occurrence, DST/timezone handling, overlap policy, and missed-run / catch-up policy. - **Retry, timeout, and poison-job** handling (backoff, timeout detection, dead-letter). - **Scaling & availability** of each tier (stateless API, shard rebalancing, partitioned queue, autoscaled workers). - **Observability**: scheduling lag, queue depth/age, failure and retry rates, lease expirations, orphaned shards, and the alerts that matter. - Sensible **tradeoff** discussion (DB polling vs delay queue; at-least-once vs exactly-once; bucket granularity). ### Follow-up Questions - A single minute has 2 million jobs due (midnight UTC cron storm). Walk through what happens and how your sharding/queue absorbs it without breaking the latency SLA. - A scheduler crashes in the window between recording its claim and the run actually reaching the queue. What state is the run left in, how does the system detect and recover it, and why does the recovery not produce a second execution? - A worker is executing a 30-minute job; its lease expires at 60s and the run is reclaimed. How do you prevent two concurrent executions, and what role do heartbeats and idempotency keys play? - How would you let a team safely cancel a recurring job mid-flight, ensuring no already-claimed run still fires after cancellation?