# Design a 200-Terabyte Media Migration and Training Input Pipeline Design a one-time transfer of 200 TB of images and videos from an on-premises environment to a cloud GPU training cluster. The data may consist of many small files. The transfer may use a 1, 10, or 100 Gbps path, and the deadline is measured in days. First choose between network transfer and physical transfer using quantitative estimates. Then design the cloud landing and GPU input path. ### Constraints & Assumptions - Use 200 TB = 1.6 million gigabits for idealized bandwidth arithmetic. - Real transfer throughput should include a stated efficiency factor rather than assume perfect line rate. - The compute cluster must not use one machine's local disk as the durable source of truth. - Integrity, encryption, resumability, and bounded impact on production networking are required. ### Clarifying Questions to Ask - Is the transfer one-time or recurring, and what is the exact completion deadline? - Where are the source and cloud region, and is a private link already available? - How many files exist, what are their size distribution and codecs, and which preprocessing is required? ### Part 1 — Choose and Execute the Migration Estimate ideal and realistic transfer times at the supplied link speeds. Select a network or physical-transfer plan, including partitioning, parallelism, resumability, checksums, and security. #### What This Part Should Cover - Correct unit conversion and a decision tied to the measured deadline and available bandwidth. - Object storage as a landing zone, multipart transfer, manifests, checksums, retries, and final reconciliation. ### Part 2 — Feed the GPU Training Cluster Design how data moves from object storage through preprocessing into repeated training reads without starving GPUs. #### What This Part Should Cover - Offline validation and preprocessing, large training shards, streaming reads, prefetch, and multiple data-loader workers. - Bottleneck measurement across network, object-store request rate, CPU decode, and GPU compute. ```hint Calculate before choosing Divide total bits by link bits per second, then account for efficiency and compare with physical-device logistics. ``` ```hint Avoid tiny-file training reads Package validated examples into bounded sequential shards so workers amortize metadata and request overhead. ``` ### What a Strong Answer Covers - A quantitative migration choice, durable decoupling through object storage, end-to-end integrity, and an input pipeline whose throughput is measured rather than assumed. - Clear separation between the one-time migration path and any future recurring ingestion path. ### Follow-up Questions - How would the design change for a daily incremental feed after the initial migration? - How would you recover if checksums show that one percent of uploaded objects are missing or corrupt?