Design a Text-to-Video Generation Service: GPU Scheduling, Failures, Traffic Bursts

Design the backend of a text-to-video generation service in the style of Sora: users submit a prompt (and options such as duration and resolution), and a large generative model running on GPUs produces a video clip that the user can then view and download. Generating one clip can occupy expensive GPUs for minutes. Focus on three areas: **scheduling generation jobs onto GPU workers**, **recovering from failures** during long-running jobs, and **handling traffic bursts** when demand far exceeds GPU capacity. ```hint Minutes, not milliseconds Because each job holds scarce GPUs for minutes, think about what the user should see between submitting and receiving a video, and what the system should do with work that cannot start right away. ``` ```hint Losing a worker halfway Consider how much progress a job can lose when its worker dies, and what you could save along the way so a restart does not begin from zero. ``` ### Clarifying Questions - What request volume is expected, and how long does one clip take to generate on how many GPUs? - Are there user tiers with different priorities or quotas (free versus paid)? - Which durations and resolutions are offered, and do different options need different GPU configurations? - What latency expectation is set for users: an estimated wait time, or a notification when the clip is ready? - Must prompts and outputs pass content-safety checks before and after generation? ### What a Strong Answer Covers - Capacity reasoning: GPU-seconds per clip compared with fleet size, and what the resulting queueing means for users - An asynchronous job lifecycle with durable job state, status updates and delivery of results - A scheduler that places jobs according to GPU requirements, priorities and fairness, and keeps GPUs highly utilized - Failure handling: worker loss, retries, checkpointing of partial progress and idempotent job execution - Burst handling: admission control, quotas, queue limits, wait-time estimates and elastic capacity - Storage and delivery of outputs, safety checks, and monitoring of cost and queue health ### Follow-up Questions - A new model version needs twice as much GPU memory. How do you roll it out alongside the old one? - How would you give paying users a guaranteed maximum wait while still using idle capacity for free users? - How would you let users preview a low-resolution version before the full clip is ready? - GPU capacity spans several regions. How do you place jobs across them?