## Scenario Design an asynchronous service that accepts a text prompt, generates a video, and lets a user observe progress and retrieve the final artifact. Generation is expensive and may take minutes. Workers can fail, users may retry requests, and some jobs should be cancelled. Discuss the API, job state machine, queue choice, worker orchestration, artifact storage, progress events, retries, admission control, security, observability, and cost isolation. Compare a durable message queue with using Redis as the primary work queue. ### Constraints & Assumptions - The baseline accepts one prompt and generation configuration and returns a job ID immediately. - Generated artifacts are immutable and may be large. - A user must not read another user's prompt, progress, or video. - Generation is at-least-once at the worker boundary; visible job completion must remain idempotent. - `CANCELLED` is terminal. Define atomic cancel outcomes for queued, claimed, running, cancel-pending, and already-terminal jobs. ### Clarifying Questions to Ask - What job volume, completion-time target, artifact size, and retention are expected? - Are progress updates exact or best-effort? - Can running model execution be preempted on cancellation? - Which database transition is the completion-versus-cancellation linearization point, and what does each losing caller receive? - Are prompt moderation, output review, quotas, or priority classes required? - Must identical requests be deduplicated? ```hint Keep durable job state outside the queue The queue says work is available; a transactional job record defines ownership, attempts, cancellation, and the one committed result. ``` ### What a Strong Answer Covers - Submit, status, cancel, and artifact APIs with authentication and idempotency. - Durable state transitions, work leases, heartbeats, retry classification, and stale-worker fencing. - A complete cancellation transition table, deterministic API race outcomes, and fencing that prevents a cancelled attempt from publishing an artifact. - Queue durability and delivery semantics, with a reasoned SQS-versus-Redis comparison. - Object storage, signed retrieval, lifecycle policy, and atomic publication of the winning result. - Admission control, quotas, priority fairness, safety controls, and detailed stage metrics. ### Follow-up Questions 1. How does the system prevent two retried workers from publishing different final videos? 2. When is Redis appropriate in this design even if it is not the source-of-truth queue? 3. How would you support resumable multi-stage generation?