How would you validate a driving simulator’s realism?

Quick Overview

This Analytics & Experimentation prompt for a Data Scientist evaluates system-level experimental design and statistical-analysis skills for simulator validation, covering concepts such as distributional comparison, scenario stratification, interaction- and event-level evaluation, and causal reasoning at a high-level (system and statistical abstraction). It is commonly asked because autonomous-driving teams must quantify simulator realism and divergence from real-world logs to ensure scalable, reliable offline evaluation of vehicle performance while managing tradeoffs between fidelity, coverage, and rare-event and confounding failure modes.

How would you validate a driving simulator’s realism?

Company: Meta

Role: Data Scientist

Category: Analytics & Experimentation

Difficulty: medium

Interview Round: Onsite

You work on autonomous driving evaluation. You have two datasets for the same set of driving scenarios: - **Real-world logs** collected from vehicles (ground-truth observations). - **Simulator outputs** generated by running the same scenarios in a simulator (may include simulated agent behaviors, trajectories, interactions, and outcomes). Your goal is to decide whether the simulator is **realistic enough** to be used to evaluate or regress vehicle performance. **Task:** Propose an end-to-end approach to evaluate simulator realism and identify where it diverges from reality. Include in your answer: 1. What “realistic” means operationally (multiple candidate definitions and tradeoffs). 2. Which metrics you would compute (trajectory-level, interaction-level, safety/event-level, distributional). 3. How you would design comparisons between real vs simulated data (matching, conditioning on context, scenario stratification). 4. How you would test whether simulator improvements actually lead to better correlation with real-world performance (and avoid being misled by confounding). 5. Key failure modes (Simpson’s paradox across scenario types, selection bias in logged data, rare-event evaluation, sensor/label differences). Assume you can compute arbitrary features from both datasets and you can sample many simulator rollouts per scenario.

Quick Answer: This Analytics & Experimentation prompt for a Data Scientist evaluates system-level experimental design and statistical-analysis skills for simulator validation, covering concepts such as distributional comparison, scenario stratification, interaction- and event-level evaluation, and causal reasoning at a high-level (system and statistical abstraction). It is commonly asked because autonomous-driving teams must quantify simulator realism and divergence from real-world logs to ensure scalable, reliable offline evaluation of vehicle performance while managing tradeoffs between fidelity, coverage, and rare-event and confounding failure modes.

|Home/Analytics & Experimentation/Meta
Meta logo
Meta
Nov 24, 2025, 12:00 AM
mediumData ScientistOnsiteAnalytics & Experimentation
5
0

You work on autonomous driving evaluation.

You have two datasets for the same set of driving scenarios:

  • Real-world logs collected from vehicles (ground-truth observations).
  • Simulator outputs generated by running the same scenarios in a simulator (may include simulated agent behaviors, trajectories, interactions, and outcomes).

Your goal is to decide whether the simulator is realistic enough to be used to evaluate or regress vehicle performance.

Task: Propose an end-to-end approach to evaluate simulator realism and identify where it diverges from reality.

Include in your answer:

  1. What “realistic” means operationally (multiple candidate definitions and tradeoffs).
  2. Which metrics you would compute (trajectory-level, interaction-level, safety/event-level, distributional).
  3. How you would design comparisons between real vs simulated data (matching, conditioning on context, scenario stratification).
  4. How you would test whether simulator improvements actually lead to better correlation with real-world performance (and avoid being misled by confounding).
  5. Key failure modes (Simpson’s paradox across scenario types, selection bias in logged data, rare-event evaluation, sensor/label differences).

Assume you can compute arbitrary features from both datasets and you can sample many simulator rollouts per scenario.

Loading comments...