## Design a Fair GPU Training Job Platform Design a platform where machine-learning engineers submit one-off GPU training jobs. The platform must allocate a limited, heterogeneous GPU fleet fairly and prevent one user from monopolizing capacity by flooding the system with jobs. ### Constraints & Assumptions - Jobs are submitted once; recurring and workflow-DAG scheduling are out of scope. - A job declares its container or executable, arguments, data and output references, GPU type and count, CPU, memory, expected duration, and priority class. - Some jobs may require several GPUs to start together. - The fleet spans multiple hosts and may contain different GPU models, memory capacities, health states, and locality constraints. - The design must distinguish submission rate limiting from fair allocation of admitted work. - Required fleet size, tenant model, queueing delay, job duration, failure policy, and preemption rules must be clarified. ### Clarifying Questions to Ask - Is fairness per user, team, tenant, or paid quota, and may weights or reserved capacity differ? - Are jobs interruptible, checkpointable, or preemptible? What happens to partially completed work? - Must multi-GPU jobs fit on one host, one rack, or any connected set of hosts? - Which GPU types may substitute for one another, and how important are data locality and topology? - What submission burst, fleet utilization, scheduling latency, and queue-time objectives are required? ### Part 1 — Define Submission and Job State Specify the submit, cancel, inspect, and list APIs and the durable job record. Explain validation, idempotency, ownership, quotas, and the lifecycle of a one-off job. #### What This Part Should Cover - A caller-scoped idempotency key and immutable normalized job specification. - Admission checks for identity, image and data access, resource bounds, and account limits. - States such as accepted, queued, reserved, running, succeeded, failed, canceled, and expired. - Separate submission-rate limits, outstanding-job limits, and resource quotas. ```hint Limit requests and resources separately A request token bucket can stop API abuse, but fair GPU allocation must account for the admitted jobs and the resources they consume. ``` ### Part 2 — Schedule Work Fairly Design the pending queues and scheduling policy. Prevent a user with many queued jobs from starving other users while still keeping the GPU fleet useful. #### What This Part Should Cover - A fairness unit and entitlement model, such as weighted tenant or user shares. - Resource-aware fair selection using dominant-resource share, virtual time, or another declared policy. - Priority behavior, aging or starvation prevention, and limits on priority abuse. - Backfilling only when it does not indefinitely delay an earlier eligible job. ```hint Measure service, not queue length Fairness is better expressed in GPU time or dominant resource share than by giving every queued job the same position. ``` ### Part 3 — Place and Launch GPU Jobs Explain how the scheduler discovers available capacity, reserves GPUs, places single- and multi-GPU jobs, and hands work to execution agents without double allocation. #### What This Part Should Cover - Versioned resource inventory with GPU model, memory, topology, locality, and health. - Atomic or leased reservations and gang placement for jobs that need several GPUs together. - A claim-and-launch protocol with fencing tokens or conditional state transitions. - Handling fragmentation, incompatible requests, and safe resource release. ```hint Make a reservation expire safely If a scheduler dies after selecting GPUs, another scheduler must eventually reclaim them without allowing two jobs to use the same devices. ``` ### Part 4 — Handle Failures and Operate the Platform Address scheduler failover, worker loss, launch ambiguity, cancellation, retry policy, checkpoint-aware preemption, observability, and capacity planning. #### What This Part Should Cover - Idempotent control-plane transitions and reconciliation of desired versus observed state. - Explicit retry and preemption policies that do not silently reset fairness accounting. - Isolation of untrusted jobs, credentials, datasets, and output artifacts. - Metrics for queue delay by cohort, allocation share, utilization, fragmentation, failures, and starvation. ```hint Reconcile uncertain outcomes A timed-out launch request does not prove that no container started; compare durable intent with node-agent observations before retrying elsewhere. ``` ### What a Strong Answer Covers - Durable, idempotent submission and a precise one-off job lifecycle. - Independent controls for API flooding, admitted backlog, and resource-share fairness. - Resource- and topology-aware placement with leased reservations and safe multi-GPU startup. - Failure reconciliation, isolation, auditable scheduling decisions, and fairness-specific observability. ### Follow-up Questions 1. How would you keep small jobs moving without starving a large eight-GPU job? 2. How should a team that is below its guaranteed share borrow idle capacity from another team? 3. What happens when a node agent reports success after the scheduler's reservation lease expired? 4. How would you explain to a user why their job has waited longer than a later submission?