Explain Performance Engineering and How to Build a Trustworthy GPU Benchmark

This round, for a GPU performance engineering team, had no coding: it was a conversation about your understanding of performance engineering and of benchmarking. The specific questions were not reported, so the two parts below cover the two reported themes. Use concrete GPU examples where you can, such as a kernel, a data transfer, or a training or inference step. ### Clarifying Questions - Which kind of work does the team measure: individual GPU kernels, communication between GPUs and machines, or whole applications? - Is the benchmarking mainly for catching regressions across software and driver versions, for comparing hardware, or for optimizing specific workloads? ### Part 1 — What performance engineering is and how you practice it Explain what performance engineering means to you, and walk through how you would take a GPU workload that is slower than expected to a verified improvement. ```hint Know the ceiling first Before profiling, work out what the hardware could deliver for this particular workload, so that every measurement can be stated as a distance from a limit. ``` #### What This Part Should Cover - A repeatable loop: measure, form a hypothesis about the bottleneck, change one thing, verify - Bounding the workload by hardware limits and deciding which limit applies - The tools and evidence used at each step, and when to stop optimizing ### Part 2 — What makes a benchmark trustworthy Explain how micro-benchmarks differ from end-to-end benchmarks, what can make a benchmark result wrong or unrepeatable, and how you would run and report a benchmark so that others can trust and reproduce it. ```hint What does the timer bracket On a GPU, the host call that starts work returns before the work finishes, and the hardware's state changes from one run to the next. Think about what your timer really measures, and what differs between runs. ``` #### What This Part Should Cover - Micro versus end-to-end benchmarks, and what each can and cannot show - Sources of bias and noise in GPU measurements, and how to control them - Repetitions, statistics and the context reported with a result ### What a Strong Answer Covers - Concrete, GPU-specific examples rather than generic statements - Clear distinctions between latency and throughput, and between noise and systematic error - Every optimization claim backed by a measurement and compared with a hardware bound - Results communicated with their uncertainty and enough detail to reproduce ### Follow-up Questions - After a driver update, a benchmark is slightly slower. How do you decide whether the regression is real? - A kernel reaches a small fraction of peak arithmetic throughput, while the profiler shows memory bandwidth near its peak. What do you do next? - A micro-benchmark shows a large speedup that does not appear end to end. What are the likely reasons? - How would you run a benchmark suite in continuous integration so that it catches regressions without frequent false alarms?