# Design a Large-File Upload and Analysis Flow Design a service that accepts large user files, validates them, analyzes them asynchronously, and makes a report available when processing finishes. The client needs a synchronous API to initiate uploads and retrieve status, while completion may be delivered through an offline notification. Address batched requests as well as individual uploads. ### Constraints & Assumptions - Files may be too large to proxy safely through an application server. - Uploads and analysis jobs may be retried. - Validation includes file size, declared type, checksum, and authorization. - Analysis can take minutes and may fail transiently or permanently. - A user must not read another user's file or report. ### Clarifying Questions to Ask - What is the maximum file and batch size? - Does a report require every file in a batch, or may partial results be exposed? - Which completion channels are required, and what delivery guarantee is expected? - Must duplicate bytes or duplicate client requests share analysis work? ### Part 1 - API and upload path Define initiation, multipart transfer, completion, status, and report APIs. Explain where authentication, metadata, checksums, and validation occur. #### What This Part Should Cover - Direct object-storage upload with scoped credentials - Explicit upload and batch state machines - Idempotency keys and checksum verification - Authorization on every metadata and object transition ### Part 2 - Analysis and notification path Design the queue, workers, retry policy, result storage, and completion notifications. Explain how duplicate events and worker crashes are handled. #### What This Part Should Cover - Durable handoff after verified upload completion - At-least-once delivery with idempotent processing - Bounded retries and a terminal failure state - Notification outbox or equivalent reliable publication ### Part 3 - Scale and failure recovery Discuss backpressure, large batches, progress reporting, cleanup, regional failure, and observability. #### What This Part Should Cover - Capacity isolation and workload-aware queues - Resumable transfer and retry-safe state transitions - Orphaned-upload and retention cleanup - Metrics tied to each state and failure boundary ```hint Treat upload and analysis as separate state machines An object can be fully uploaded but not yet accepted for analysis. Persist those transitions explicitly instead of inferring them from queue messages. ``` ### What a Strong Answer Covers - A secure direct-upload path that keeps large bytes away from API servers - Durable, idempotent asynchronous analysis with precise states - Batch semantics, progress, notification, and report authorization - Backpressure, retries, cleanup, and observability grounded in failure cases ### Follow-up Questions 1. How would you prevent an object-created event from starting analysis before multipart completion is verified? 2. How would you let a client retry a batch request without duplicating work? 3. What data would you retain to diagnose a report that never generated?