A distributed neural-network training job shows GPU utilization repeatedly swinging from about 90% down to 20%. Explain how you would diagnose the throughput problem and isolate whether the bottleneck is input, CPU preprocessing, communication, synchronization, memory, or GPU kernels. ### Constraints & Assumptions - You can collect host, accelerator, data-loader, and network metrics and run controlled experiments. - The diagnosis should preserve training correctness. - Do not assume low average utilization has a single cause. ### Clarifying Questions to Ask - Is the pattern synchronized across workers and aligned with step boundaries or checkpoints? - Did it begin after a change in model, data, hardware, or worker count? - Are step time and samples per second oscillating with utilization? ```hint Build a timeline Correlate GPU gaps with data wait, CPU saturation, collective operations, allocator activity, and storage reads on the same time axis. ``` ### What a Strong Answer Covers - A measurement-first workflow with per-step traces and queue-depth metrics. - Controlled experiments that vary workers, data, preprocessing, batch size, and communication independently. - Correct mitigations for starvation, stragglers, collectives, checkpoint pauses, and inefficient kernels. - Before-and-after throughput and correctness validation. ### Follow-up Questions - What would synchronized dips across every worker suggest? - How would you distinguish data starvation from an all-reduce bottleneck? - Why can increasing data-loader workers make performance worse?