Keep GPUs Fed with an Efficient Data-Preparation Pipeline

A model-training job frequently leaves its GPUs idle while waiting for input batches. Explain how you would diagnose and improve data preparation and loading. ### Constraints & Assumptions The source asks about CPU, I/O, preprocessing, and batch preparation. No framework or storage system is prescribed. Preserve data semantics and reproducibility while improving throughput. ### Clarifying Questions Where are the samples stored? Which transformations run per example? Are batch shapes variable? Is the job distributed? Are GPU gaps caused by input starvation or synchronization elsewhere? ### What a Strong Answer Covers Measure the pipeline, separate storage and CPU bottlenecks, overlap bounded stages, and validate sharding, ordering, and restart behavior after changes. ### Follow-up Questions When can adding loader workers make performance worse? What benefits and costs come from caching, prefetching, pinned memory, and sequence bucketing? How would you prevent duplicate samples across workers?