Design a Sora-Style Video Generation Service with Fair Queueing and Cancellation

Design the backend of a text-to-video generation product in the style of Sora. A user submits a text prompt with generation settings, the request becomes a long-running generation task that runs on a GPU, and when it finishes the user retrieves the video. The design has one simplifying constraint: **each task runs on exactly one GPU**. A single task is never split across several GPUs, so the scheduler never has to find several free GPUs at once for the same task. GPU capacity is the scarce resource, and many users compete for it. The discussion should concentrate on two areas: how tasks from different users are queued fairly for the GPU fleet, and how a user cancels a task that is waiting or already running. ### Constraints and Clarifications - One task uses exactly one GPU for its whole run. - Demand can exceed GPU capacity, so tasks wait in a queue before they run. - Cancellation requests are expected to be infrequent compared with submissions. - No latency, throughput or fleet-size targets are given; state the assumptions your design depends on. ### Clarifying Questions - Who must be treated fairly: individual users, organizations, or subscription tiers with different shares? - Can a task's GPU time be estimated from its settings (for example resolution and clip length) before it runs? - Does a GPU run one task at a time, and are all GPUs in the fleet interchangeable? - Can a running task be paused and resumed later, or can it only be stopped? - How do users learn that a video is ready: by polling, by a push notification, or both? - Is a user charged for the GPU time a task used before it was canceled? ### Part 1 — Architecture and task lifecycle Describe the end-to-end design: the user-facing API, the states a task passes through from submission to a finished video, how queued tasks are handed to GPU workers, and where prompts, task state and generated videos are stored. ```hint Let the constraint shape dispatch With one task per GPU, work out what the system actually has to decide at the moment a GPU becomes free, and which component should start that hand-off. ``` #### What This Part Should Cover - A task state machine that includes the states a cancellation can interrupt. - The dispatch mechanism between the queue and the GPU workers, and how a worker that dies mid-task is detected and its task recovered. - Separate storage for task metadata and for large video outputs, and how the finished video reaches the user. ### Part 2 — Fair queueing Some users submit far more tasks than others, and tasks differ widely in how much GPU time they need. Propose several ways to share the GPU fleet fairly among users, compare their trade-offs, and choose one. ```hint Pick the unit of fairness Decide whether fairness is counted in tasks or in GPU time, then test each option against one user who submits many long tasks while another submits a few short ones. ``` #### Clarifying Questions for this Part - When only one user has queued tasks, may that user take the whole fleet? - Should a user who has been idle for a while get extra priority when they return? #### What This Part Should Cover - At least two distinct fair-queueing approaches, with the concrete weakness of each. - Handling of variable task cost, of bursts, and of users returning after being idle. - Where the policy runs, and what happens to its state if the scheduler restarts. ### Part 3 — Cancellation A user can cancel a task while it is queued, while it is being handed to a GPU, or while it is running. Design cancellation for each of these cases and compare approaches. In particular, evaluate this proposal: because each task holds at most one GPU and cancellations are infrequent, the cancel API can simply make a direct RPC or HTTP call to the GPU worker running the task. Say when that is sufficient and where it stops being extensible. ```hint Follow the race Trace a cancel request that arrives at the same moment the scheduler hands the task to a GPU, and decide which component is the source of truth for whether the task may run. ``` #### What This Part Should Cover - Cancellation behavior in each task state, including the dispatch race and a task that has just finished. - The direct worker call weighed against at least one alternative, argued from the one-GPU-per-task constraint and the low cancel volume. - Delivery guarantees: idempotent cancel requests, an unreachable worker, and the point at which the GPU returns to the pool. ### What a Strong Answer Covers - Explicit use of the one-GPU-per-task constraint to keep scheduling simple, rather than designing general multi-GPU placement. - A single source of truth for task state that the scheduler, the workers and the cancel path all respect. - Trade-off reasoning for both fairness and cancellation instead of a single asserted answer. - Failure handling and observability focused on per-user queue wait, GPU utilization and cancel latency. ### Follow-up Questions - If a future model needed several GPUs per task, what in your scheduling and cancellation design would break first? - How would you add a paid priority tier without starving free users? - How would you preempt a running task for a more urgent one, and what would that require from the worker? - How would you show a user a realistic estimated start time under your fair-queueing policy?