Design a job scheduler that initially serves a small workload but must later handle 100 times as many jobs. Clients submit one-time jobs with execution time, payload reference, retry policy, and idempotency key. Workers claim due jobs, execute them, and record outcomes. Discuss the simple initial design, the bottlenecks that appear at higher scale, and a migration path that does not lose or duplicate accepted jobs. ### Constraints & Assumptions - Execution is at least once; job handlers must be idempotent. - A worker can crash after performing work but before acknowledging completion. - Delayed, retried, canceled, and dead-lettered jobs need visible states. ### Clarifying Questions to Ask - What scheduling precision and maximum delay are acceptable? - How long do jobs run, and do they require different resource types? - Is strict ordering required for any job key? ```hint Separate finding due work from running it A scheduler can move due job IDs to partitioned ready queues, while workers consume those queues independently. ``` ### What a Strong Answer Covers - A durable job state machine, lease or visibility timeout, idempotent submission, and retry scheduling. - Partitioning by time and tenant, ready queues, worker autoscaling, backpressure, and fairness. - A staged migration with dual-read or backfill evidence, reconciliation, and rollback. ### Follow-up Questions - How would you avoid a thundering herd at the top of an hour? - How would you support long-running jobs whose lease expires? - What metrics prove the 100-times migration is safe?