How would you validate a driving simulator’s realism?

You work on autonomous driving evaluation. You have two datasets for the same set of driving scenarios: - **Real-world logs** collected from vehicles (ground-truth observations). - **Simulator outputs** generated by running the same scenarios in a simulator (may include simulated agent behaviors, trajectories, interactions, and outcomes). Your goal is to decide whether the simulator is **realistic enough** to be used to evaluate or regress vehicle performance. **Task:** Propose an end-to-end approach to evaluate simulator realism and identify where it diverges from reality. Include in your answer: 1. What “realistic” means operationally (multiple candidate definitions and tradeoffs). 2. Which metrics you would compute (trajectory-level, interaction-level, safety/event-level, distributional). 3. How you would design comparisons between real vs simulated data (matching, conditioning on context, scenario stratification). 4. How you would test whether simulator improvements actually lead to better correlation with real-world performance (and avoid being misled by confounding). 5. Key failure modes (Simpson’s paradox across scenario types, selection bias in logged data, rare-event evaluation, sensor/label differences). Assume you can compute arbitrary features from both datasets and you can sample many simulator rollouts per scenario.