Design a Sora-like text-to-video generation platform. Users submit a text prompt, optional generation settings (duration, resolution, fps, seed, model variant), and possibly optional conditioning media such as an init image or reference clip. The system generates a short video and returns a downloadable result once the job is complete. Because a single clip takes seconds-to-minutes of GPU time, the system is inherently **asynchronous**: a submit returns a job handle immediately, and the user polls or receives a webhook when the result is ready. Your design should cover the user-facing API and job lifecycle, the high-level service architecture, how GPU inference workers are scheduled, how the system handles unstable workers / crashes / retries / partial failures, how intermediate and final artifacts are stored, how safety / rate limits / quotas are enforced, and how quality / latency / reliability are monitored. The hardest, most-probed areas are (a) the **job lifecycle / state machine** and (b) **failure handling** when workers are unstable — go deep on both. ```hint Where to start Anchor the whole design on the **inference profile**: GPU-bound, long-running, expensive, failure-prone. That single observation forces an **async, queue-backed** architecture where the database is the source of truth for job state and the GPU is the scarce resource you schedule for fairness and utilization, not request throughput. ``` ```hint Job lifecycle Draw an explicit **state machine** with terminal states (e.g. `COMPLETE`, `FAILED_PERMANENT`, `CANCELED`, `TIMED_OUT`) that never transition out, and give every `RUNNING` job a deadline so a wedged worker can't hang forever. Make `POST /videos` **idempotent** via an `Idempotency-Key` so a client retry on timeout never double-creates (and double-bills) a GPU job. ``` ```hint Failure model Pick a single mechanism that lets the database decide who owns a job and when it has died, and that survives a worker that disappears and later comes back. Think about what damage a partitioned worker that reconnects could do to a job already retried elsewhere — and what property each result write would need to carry so that a stale, returning attempt can never win over the one the system already trusts. ``` ```hint Classify failures Don't model "a job failed" as one thing. Sort failures by what the *right reaction* is — would retrying as-is have any chance of succeeding, would it only succeed under different conditions, or is it guaranteed to fail again? Map each class to retry / retry-differently / give-up, and ask what intermediate state a long job could persist so that a recovered attempt resumes cheaply instead of restarting from zero. ``` ```hint Scheduling & batching The GPU is what you're optimizing for, so reason about what makes it idle or unfair: cold model-weight loads, one user monopolizing the pool, and small kernels. Think about which jobs you'd want to land on the *same* worker (weight affinity), how you'd stop a flood of free-tier work from starving paid traffic (priority + fair queuing), and the latency cost you pay when you micro-batch jobs together to fill the GPU. ``` ### Constraints & Assumptions State your own numbers in the interview — the figures below are illustrative, not benchmarks. - A single job is a **short clip** (e.g. 5–10 s at up to 720p, 24 fps), taking on the order of **tens of seconds to a few minutes** of GPU wall-time on one accelerator — 3–6 orders of magnitude slower than a typical web request. - The scarce, expensive resource is the **GPU accelerator**. API, queue, and database compute are cheap by comparison; the design optimizes for **GPU utilization and fairness**, not request throughput. - Jobs are **long-running and stateful mid-flight**. Worker crashes, OOMs, and spot-instance preemptions are **routine**, not exceptional — fault tolerance is a first-class requirement. - Output is **regulated content**: both the prompt and the generated frames must pass safety/moderation, and outputs typically require watermarking / provenance. - Assume a target sustained throughput (e.g. ~1 job/s) and multiple user tiers (free / paid / interactive preview) with different priority. - The system is fully **async**: every job must reach a terminal state and never silently hang. ### Clarifying Questions to Ask - What clip lengths, resolutions, and frame rates must we support, and which of those are user-tunable vs fixed per tier? - What is the target throughput and the latency SLA (queue wait vs total time), and how does it differ across free / paid / preview tiers? - What conditioning inputs are in scope (text only, init image, mask, reference clip, audio)? - What are the safety/policy requirements (banned categories, real-person likeness, watermarking, provenance, audit retention)? - What is the retention policy for generated videos and intermediates, and are there data-residency or erasure (GDPR) obligations? - Is billing per-GPU-second, per-job, or subscription — and must we guarantee no double-billing on retries? ### What a Strong Answer Covers - **Async-first framing**: justifies the queue-backed, GPU-scheduled architecture from the inference profile rather than defaulting to a synchronous request/response design. - **Idempotent API + explicit job state machine** with well-defined terminal states and a guard against silent hangs (per-job deadline / timeout). - **Concrete failure-detection and ownership scheme**: a real mechanism for noticing a dead worker and resolving who owns a job, with a defense against a returning/partitioned worker corrupting state — not a hand-wave at "we retry." - **Failure taxonomy**: distinguishes retryable vs permanent failures and reasons about the right action per class, including resource-exhaustion (OOM) failures and a cap on runaway / repeatedly-failing jobs. - **Single-winner result publish**: only one attempt's output can ever become the result, so partial or duplicate artifacts never leak to the user. - **Scheduling for utilization *and* fairness**: priority queues, weight affinity, batching tradeoffs, admission control / backpressure, and spot+on-demand autoscaling driven by wait-time. - **Storage split**: transactional metadata in a DB vs large artifacts in object storage, with lifecycle expiry and signed-URL delivery. - **Two-stage safety** (pre-prompt and post-frame) plus watermarking/provenance, and **layered quotas** (rate, concurrency, budget) enforced atomically. - **Observability that separates queue wait from compute time** and tracks model quality per version. - **Sensible cost/latency reasoning**: a defensible estimate of steady-state accelerator count from throughput and per-job time, and an argument for why duration/resolution scale super-linearly and are therefore gated parameters. ### Follow-up Questions - A worker is network-partitioned, declared dead, its job retried and completed elsewhere — then the original worker reconnects and tries to publish its result. Walk through exactly how your design prevents a double publish or double bill. - 80% of a 90 s job completes and then the spot instance is preempted. How do you avoid redoing the expensive work, and what are the limits of resuming a non-deterministic GPU computation? - Inference finished successfully but the post-processing (encode / upscale / watermark) stage crashed. What do you re-run, what must you *not* re-run, and why? - A single prompt deterministically crashes whatever worker picks it up (a "poison pill"). How does the system keep this from taking down the fleet, and how do you bill it? - Your GPU utilization drops to 60% while queues are full and wait times climb. What are the likely causes, and how does your monitoring let you localize the bug to scheduling/placement vs the model vs hardware?