File ingestion system that never drops or double-processes a file

You are a product manager partnering with engineering on a service that takes in files and processes them. Design the processing pipeline so that it is complete and reliable: walk through how a file moves from arrival to the point where its results are ready for downstream use, how the system makes sure nothing is lost or handled twice, and how you would know when something has gone wrong. ```hint Make "complete" checkable Completeness can only be proven if the pipeline knows what should have arrived and can account for every record in what did arrive. Think about what the sender and the pipeline would each need to record so that a gap becomes detectable. ``` ### Clarifying Questions - Where do the files come from (customers uploading through an app, partner systems dropping batches, internal services), and does the sender supply anything we can check against, such as a manifest, checksums or record counts? - What does "complete" mean to the people who use the output: every expected file arrived, every record inside each file was loaded, or both? Is a partially loaded file ever acceptable? - How fresh must the results be, and is it acceptable to hold back a batch while waiting for a late file? - Must files be applied in a particular order, for example incremental updates that each build on the previous one? ### What a Strong Answer Covers - A precise definition of completeness and reliability for this pipeline, turned into guarantees that can be measured. - An end-to-end flow from arrival to downstream availability, with a durable record of each file's state. - How missing, partial, corrupt and duplicate files are detected, and how every record is accounted for as loaded or set aside with a reason. - Failure handling: retries, idempotent processing, crash recovery, files that keep failing, and replaying data after a bug. - Monitoring and alerting, plus the trade-offs a PM owns, such as freshness versus completeness, strict versus lenient validation, and building versus using a managed service. ### Follow-up Questions - A sender re-sends yesterday's file with corrections after the original was already loaded. How does the pipeline apply the fix without double counting? - One file from a nightly batch of related files never arrives. Do you publish the rest or hold everything, and who should make that call? - How would you report the pipeline's reliability to downstream teams, and what targets would you commit to?