## Design a Scheduler for ML Training and Batch Inference Jobs Design a scheduler for machine-learning compute jobs. Focus on training and batch inference, while allowing lower-priority evaluation jobs. Explain workload-specific requirements, resource-allocation policy, compute-resource monitoring, and crash handling across every component. Include multi-tenancy after the first five concerns are sound. ### Part 1 — Model Jobs and Resources Define job, task, attempt, queue, tenant, worker, and resource identities plus submission, status, cancellation, and retry APIs. #### What This Part Should Cover - Resource vectors such as CPU, memory, accelerator count and type, storage, and network needs. - Different execution shapes for training, batch inference, and occasional evaluation. - Priority, deadline, checkpoint, retry, and data-locality fields only where the workload requires them. - Immutable job specifications separated from mutable attempts and observed state. ```hint Schedule an attempt, not an abstract job Retries and partial failures are easier to reason about when each execution attempt has its own identity and lease. ``` ### Part 2 — Allocate Resources and Order Work Design admission control, queues, placement, and allocation policy for long-running training and throughput-oriented batch inference. #### What This Part Should Cover - Hard feasibility filters before scoring eligible workers. - Fairness, priority, quotas, deadlines, and starvation prevention. - Gang placement or topology constraints when a training job needs several workers together. - Preemption, backfilling, and fragmentation trade-offs with explicit checkpoint costs. ```hint Separate feasibility from preference A worker that lacks the required accelerator is not a low-scoring choice; it is not a valid placement candidate. ``` ### Part 3 — Monitor Compute Resources Explain how the scheduler learns capacity, reservations, health, utilization, and topology without double-allocating stale resources. #### What This Part Should Cover - Worker-agent registration, heartbeats, inventory versions, and health state. - Allocatable capacity distinguished from measured utilization. - Leased reservations and fencing tokens for accepted placements. - Reconciliation between scheduler state, worker observations, and running attempts. ```hint Usage is not ownership A low utilization sample does not mean reserved capacity is free for another job. ``` ### Part 4 — Handle Crashes and Ambiguous Outcomes Walk through scheduler, queue, worker, network, and storage failures, including a timeout after a launch request whose result is unknown. #### What This Part Should Cover - Durable job and attempt state machines with idempotent transitions. - Scheduler failover, expiring leases, and stale-worker fencing. - Checkpoint and restart policy for training versus replayable inference partitions. - Output commit or manifest semantics that prevent duplicate visible results. ```hint Reconcile before retrying elsewhere A launch timeout is an unknown outcome; immediately starting a second attempt can spend the same allocation twice. ``` ### Part 5 — Isolate Tenants and Operate the System Add quotas, workload isolation, observability, and capacity planning without weakening scheduling correctness. #### What This Part Should Cover - Tenant-scoped quotas, fair shares, identities, secrets, and network boundaries. - Queue delay, placement failures, utilization, preemption, checkpoint, and retry metrics. - Audit trails for policy decisions and administrative overrides. - Tests for contention, topology scarcity, stale agents, failover, and noisy neighbors. ```hint Make policy decisions explainable Operators should be able to tell why a runnable job waited while another job received the scarce resource. ``` ### What a Strong Answer Covers - Distinguishes training, batch inference, and evaluation requirements. - Defines a concrete allocation policy after feasibility and admission checks. - Treats resource state as leased, versioned, and reconciled. - Handles component crashes and duplicate attempts without duplicate visible work. - Adds multi-tenant fairness, isolation, and operational evidence. ### Follow-up Questions 1. When should training jobs be preemptible, and how does checkpoint cost affect that choice? 2. How would you place a distributed training job that needs several connected accelerators at once? 3. What happens when a worker finishes an inference partition after its lease expired? 4. Which metric shows resource fragmentation rather than true capacity shortage?