## Design Direct File Uploads for a Chat Application Design file uploads for an existing chat application. Clients should upload file bytes directly to a cloud object store through a short-lived upload capability so the chat backend does not proxy the payload. The backend must track upload state, operate under heavy request volume, surface useful progress or time estimates, and parse files after upload. ### Constraints & Assumptions - Users may attach files only to conversations they are authorized to access. - File size, allowed types, retention, regional placement, and whether a message can be sent before processing completes must be clarified. - Object-store callbacks and client completion requests may be duplicated, delayed, reordered, or missing. - Upload time depends on client throughput; parsing time also depends on queue delay and file characteristics. ### Part 1 — Define the Upload and Message Contract Specify APIs and durable records for initiating, completing, canceling, and reading upload status. Relate an upload to a conversation, sender, object key, and eventual chat attachment. #### What This Part Should Cover - Stable upload and message or attachment IDs. - Authorization and validation before issuing a presigned upload capability. - States such as initiated, uploading, uploaded, verifying, parsing, ready, failed, canceled, and expired. - Caller-scoped idempotency and a policy for abandoned uploads. ```hint Separate upload intent from object bytes The backend can own authorization and lifecycle state while the client transfers the large payload directly to storage. ``` ### Part 2 — Upload Directly and Surface Progress Design the direct client-to-storage path, including large files, retries, integrity, progress, and an estimate of remaining time. #### What This Part Should Cover - A capability scoped to one object, operation, size or policy, content expectations, and short expiry. - Multipart or resumable behavior when required and idempotent part handling. - Client-observed bytes and rolling throughput for upload progress. - Honest separation of upload ETA from queued verification and parsing time. ```hint Estimate each stage from its own evidence Bytes transferred can estimate network progress, while parse completion depends on queue position and workload history. ``` ### Part 3 — Confirm, Verify, and Parse the File Explain how a client completion call or storage event becomes a trusted backend record and launches asynchronous parsing. #### What This Part Should Cover - Verification that the expected object exists and matches allowed metadata before trusting a client acknowledgement. - Idempotent convergence of client completion and provider events. - Durable enqueueing of scanning or parsing after upload verification. - Publication rules that keep an unsafe or incomplete attachment out of normal chat reads. ```hint Treat completion as a claim A client saying “upload finished” is a signal to verify durable storage state, not proof by itself. ``` ### Part 4 — Control Load and Recover from Failure Address request bursts, queueing, fairness, retries, provider failures, orphan cleanup, observability, and reconciliation. #### What This Part Should Cover - Admission and rate limits before issuing unlimited upload work. - Durable priority or fair queues for bounded post-upload workers. - Retry classification and dead-letter or quarantine handling. - Reconciliation among backend records, storage objects, and parser state. - Metrics for initiation, completion, throughput, stage latency, failures, backlog, and orphaned bytes. ```hint Queue bounded work, not every byte Direct upload protects backend bandwidth; queues are most useful for admission and the asynchronous work that follows storage completion. ``` ### What a Strong Answer Covers - Direct, scoped, authorized upload without proxying file bytes through the chat backend. - Durable, idempotent lifecycle state across client and storage events. - Stage-specific progress and estimates that do not promise false precision. - Bounded parsing queues, safe publication, reconciliation, and cleanup. ### Follow-up Questions 1. What happens if the object arrives but the client never calls the completion endpoint? 2. How would a resumed multipart upload preserve integrity and authorization? 3. When should a pending attachment become visible to other chat participants? 4. How would you estimate parsing delay when file types have very different costs?