## Design an Asynchronous Text-to-Video Generation Service Design a service in which a user submits a text prompt, the system performs a long-running video-generation job, and the user later retrieves the generated asset. Explain whether AWS SQS fits the work queue and how the design behaves when any component fails. ### Constraints & Assumptions - Generation is too slow and resource-intensive to keep an HTTP request open. - A submitted job must have a stable identity and an observable lifecycle. - Queue delivery may be duplicated, delayed, or reordered. - Generated assets are much larger than queue messages. - Do not assume traffic, generation time, model size, retention, or freshness targets; identify the values that would change the design. ### Part 1 — Define the Product and Job Contract Specify the submission, status, cancellation, and result APIs. Define the job states and the information needed to reproduce or investigate one generation. #### What This Part Should Cover - Idempotent submission and a stable job identifier. - Prompt and input validation, authorization, and policy checks. - Queued, running, succeeded, failed, and cancelled states with legal transitions. - Model version, parameters, ownership, timestamps, and output references. ```hint Separate acceptance from completion The submission response should confirm durable ownership of a job, not imply that a video has already been generated. ``` ### Part 2 — Choose the Queue and Worker Protocol Trace a job from durable acceptance through SQS to an eligible generation worker. Compare SQS with at least one alternative and explain how visibility timeouts, acknowledgement, and duplicate delivery interact with a long-running task. #### What This Part Should Cover - A database transaction or outbox that prevents accepted jobs from being lost before enqueue. - Small queue messages that reference durable job records and input objects. - Worker leases or heartbeats when generation can exceed a visibility timeout. - Conditional state updates and idempotency under at-least-once delivery. - The trade-off between a managed work queue and a replayable ordered log. ```hint A queue message is not the job record Keep the authoritative state in durable storage so a redelivery can decide whether work is still needed. ``` ### Part 3 — Handle Failure and Cancellation Explain recovery for enqueue failure, worker crash, timeout, duplicate delivery, malformed input, model failure, output-upload failure, poison messages, and cancellation racing with completion. #### What This Part Should Cover - Retryable versus terminal errors and bounded retry policy. - Dead-letter handling with an operator-visible reason and replay path. - Atomic publication of an output reference only after the asset is durable. - Cancellation checks before expensive phases and a conditional final commit. - Reconciliation for jobs stuck in transitional states. ```hint Make every transition conditional A late worker should not overwrite a cancellation or a success already committed by another attempt. ``` ### Part 4 — Scale and Operate the Service Discuss resource scheduling, backpressure, storage, delivery, fairness, observability, and deployment of new model versions. #### What This Part Should Cover - Batching and routing by model version, hardware need, or job priority. - Admission control, per-tenant quotas, and protection against one queue monopolizing workers. - Object storage for inputs and outputs plus controlled download access. - Queue age, state-transition latency, retry, failure, cancellation, and resource-utilization signals. - Versioned rollout and rollback without silently changing an accepted job. ```hint Measure age as well as depth A short queue can still contain a job that has been stranded for too long. ``` ### What a Strong Answer Covers - A durable asynchronous contract from submission to result retrieval. - A reasoned SQS choice with correct at-least-once and visibility-timeout semantics. - Explicit behavior for duplicates, partial failures, retries, cancellation, and dead letters. - Separation of metadata, queue messages, large assets, and model execution. - Capacity and operational decisions tied to stated workload assumptions. ### Follow-up Questions 1. When would a FIFO queue materially improve the design, and what would it cost? 2. How would you prevent a retried job from publishing two different results? 3. What happens when a worker remains healthy but generation exceeds the current lease? 4. How would you drain jobs accepted for an old model version during a rollback?