Design a Simulation System to Evaluate a Self-Driving Model on Limited Compute

This system design round is open-ended. The interviewer asks: **"How would you evaluate a self-driving model?"** and then steers the conversation toward infrastructure: how to run thousands of simulations when there is not enough compute, how to make simulation faster, and what to do when a bug turns up. The reported round was conversational and no diagram was drawn, so the structure has to come from how you talk through it. Design the system that takes a new version of the driving model, evaluates it in simulation against a large set of scenarios, and produces a result the team can act on. ### Clarifying Questions - Which "model" is under evaluation: the full driving software stack, or one component such as planning or perception? - What is the result used for: a pass/fail release gate, a comparison with the current production version, or both? - Where do scenarios come from: replayed recordings of real drives, hand-authored or generated scenarios, or both? - How much compute is actually available, and how fast must results come back (for every code change, nightly, or before each release)? - When the interviewer says "a bug", do they mean a bug in the model under test, or a bug in the simulator or evaluation pipeline itself? ### Part 1 — What to evaluate and how to decide Define what a simulation run measures, how results across many scenarios are combined, and how the system decides whether the new model is better, worse, or unsafe compared with the current one. ```hint Averages hide the failures that matter Think about whether one number over thousands of scenarios can tell you that the model now fails a rare but dangerous situation. ``` #### What This Part Should Cover - Scenario coverage organized by situation type, including rare and safety-critical cases - Concrete per-run metrics for safety, rule compliance, progress and comfort - Comparison against a baseline version with per-category results and hard safety gates, not a single average ### Part 2 — Running thousands of simulations without enough compute The compute budget cannot run every scenario at full fidelity for every model version. Explain how you still get a trustworthy answer. ```hint Not every scenario earns a full run Ask which runs carry the most information about this particular change, and what a cheaper run can and cannot tell you. ``` #### What This Part Should Cover - Prioritizing and sampling scenarios by risk, coverage and relevance to the change - Tiered fidelity: cheap checks first, expensive runs only where needed - Scheduling, preemption and reuse of results across versions ### Part 3 — Making simulation faster Explain how to make each simulation, and the whole batch, finish sooner. ```hint Find where the time goes Break one simulation run into its stages and ask which of them the model under test actually needs recomputed. ``` #### What This Part Should Cover - Running faster than real time and in parallel, with deterministic results - Skipping or caching expensive stages (for example sensor rendering) when the component under test does not need them - Early termination and batching of runs ### Part 4 — When a bug turns up A run exposes a bug. Explain how the system and the team handle it. ```hint Two suspects Before blaming the model, decide how you would show the failure is not an artifact of the simulator. ``` #### What This Part Should Cover - Exact reproducibility of any failing run - Separating model bugs from simulator or pipeline bugs - Turning each confirmed failure into a permanent regression scenario and blocking release on it ### What a Strong Answer Covers - Turning a vague prompt into explicit goals (release gate versus comparison) before designing - A clear pipeline from scenario store to scheduler to simulation workers to results and reports - Explicit trade-offs between coverage, fidelity, cost and turnaround time - Reproducibility and observability throughout, so every number can be traced to exact inputs - Keeping the design explainable without a diagram ### Follow-up Questions - How do you know your simulated scenarios are realistic enough that passing them predicts real-road behavior? - The new model is better on average but worse in one rare scenario category. Do you ship? How does the system surface this? - How would you detect that the scenario set itself has gone stale as the model improves? - Two teams change different components on the same day. How do you attribute a regression to the right change?