Design a Simulation Job Scheduling Platform That Matches Jobs to Hardware Requirements

Design a platform that runs simulation jobs and schedules each job onto hardware that meets its requirements. Users submit jobs that declare what they need, for example a number of CPU cores, an amount of memory, or a particular type and number of GPUs. The platform queues the jobs, places each one on a suitable machine, runs it, and reports status and results. The report gives only the problem name, so settle the scale, the hardware mix and the scheduling policy with the interviewer before designing. ```hint Matching jobs to machines Think about how the scheduler knows what each machine has free right now, and what it should do when a job needs hardware that is currently all in use. ``` ```hint When a machine disappears Ask what happens to a job if the machine running it stops responding halfway through, and how you make sure it is neither lost nor run twice at once. ``` ### Clarifying Questions - How many jobs per day, how long does a typical job run, and how many machines with what hardware mix? - Is the fleet fixed, or can capacity be added on demand, for example from a cloud provider? - Are jobs independent, or are they submitted as large batches (many scenarios of one simulation) that users track together? - Do teams have priorities or quotas, and may a higher-priority job preempt a running one? - Do some jobs need several machines at once? - What should happen to a job that fails, and must results be reproducible? ### What a Strong Answer Covers - Requirements, scale estimates, and a job specification that captures hardware requirements - Scheduling: matching requirements to free capacity, packing, priorities, fair sharing, and keeping large jobs from starving - Worker agents, heartbeats and leases so that every job finishes or is retried according to its policy, never lost or run twice concurrently - Storage of job state, logs and results, and APIs for submitting, tracking and cancelling jobs and batches - Scaling the scheduler and the fleet, capacity per hardware type, and failure handling - Monitoring of queue wait per hardware type, utilization and failure rates ### Follow-up Questions - GPU machines are scarce and a long queue of GPU jobs builds up. How do you decide what runs next while keeping CPU-only jobs flowing? - A job needs four GPU machines at once. How do you avoid holding three idle machines while waiting for the fourth? - The scheduler itself crashes. How do you avoid losing jobs or running one twice? - How would you decide whether to build this or adopt an existing cluster scheduler?