How validate a driving simulation is realistic?

You work on evaluating Waymo’s driving simulation.

You have:

• Real-world (logged) driving data collected on-road.
• Simulated driving data generated by the simulator for similar scenarios.

The simulator will be used to evaluate autonomy performance (e.g., collision risk, comfort, rule compliance), so you must determine whether the simulation is realistic enough to be trusted for performance evaluation.

Task

Design a data-driven validation framework to answer:

1. Is the simulated data distribution close to real-world data?
2. Does realism hold across important scenario slices (e.g., intersections, merges, pedestrians, weather, rare/long-tail events)?
3. What metrics and statistical tests would you use to quantify realism?
4. How would you decide pass/fail thresholds and handle the fact that the real world contains rare but critical events?
5. If simulation is not realistic, how do you diagnose and prioritize fixes?

Assumptions (you may make reasonable ones)

• Both datasets include time series for ego + nearby agents (positions, velocities, headings), map context, and event labels (e.g., near-miss, collision) where available.
• Real and simulated runs can be matched by scenario type but are not necessarily one-to-one identical.