Autonomous Vehicle Safety Metrics

What's being tested

Interviewers probe your ability to operationalize safety: define measurable, statistically sound metrics for rare adverse events, design experiments/analyses that can detect meaningful changes, and reason about uncertainty and tradeoffs when data are sparse. For Waymo, this maps to showing you can turn telemetry and human-intervention logs into defensible statements about change in risk, choose appropriate statistical models for low-count events, and communicate uncertainty to engineering and product partners.

Core knowledge

Unit of analysis (exposure) — choose between trip, mile, hour, or scenario-exposure; denominators drive interpretability and power. Always state the exposure used and why (e.g., miles normalizes for routing).
Primary safety metrics — common choices: collision_rate (events per million miles), intervention_rate (operator or fallback interventions per 100k miles), and near-miss proxies (time-to-collision distributions, hard-brake events). Define event, dedup rules, and attribution windows.
Rare-event modelling — treat discrete counts with Poisson (mean≈variance) or negative binomial for overdispersion; use zero-inflated models if many zeros. For count k over exposure T: rate r = k/T and Var(r)≈k/T².
Exact inference for small counts — use exact Poisson or exact binomial CI (chi-square inversion) rather than normal approximations when k is small (k<10). For testing, use exact rate ratio tests or conditional Poisson tests.
Power and sample-size for rates — approximate exposure T required to detect an absolute rate difference Δ: $T \approx \frac{(z_{1-\alpha/2}+z_{1-\beta})^2(\lambda_1+\lambda_2)}{\Delta^2}$ where λ are rates per unit exposure. For tiny baseline rates, required T often scales to millions/billions of miles.
Sequential / online testing — use alpha-spending or group-sequential methods (e.g., O’Brien–Fleming) for repeated looks; for safety-critical launches prefer pre-specified stopping rules and conservative corrections.
Confounding & causal inference — control for exposure mix (urban vs. highway), weather, time-of-day, and driver takeover propensity via stratification, matching, or weighted regression; randomized A/B is ideal for software stacks but often impractical.
Composite & leading indicators — combine rare crash metrics with higher-rate proxies (e.g., hard_brake_rate) using composite scores or hierarchical models to increase sensitivity while preserving interpretability.
Hierarchical / Bayesian models — borrow strength across segments (vehicle version, route type) with random effects to stabilize low-count estimates and get better posterior intervals for rates.
Multiple comparisons & FDR — when evaluating many routes/scenarios, correct using Benjamini–Hochberg for false-discovery control or adjust family-wise error for conservative safety statements.
Evaluation windows & latency — decide on event attribution lookback windows, handle late-arriving reconciliations as measurement error, and quantify how delayed labeling affects power and bias.

Tip: Always report both absolute and relative changes (e.g., drop of 0.0002 events/mile = 20% relative) and include exposure units so stakeholders can reason about operational impact.

Worked example — "Design a safety metric for AV interventions"

Frame: In the first 30 seconds ask: what counts as an intervention (human take-over vs autopilot fallback), what is the exposure unit (miles vs trips), and which populations/ODDs (operational design domains) to include. Skeleton answer pillars: (1) precise event definition and dedup rules, (2) denominator choice and stratification plan, (3) statistical model and expected baseline rate, (4) power/sample-size and rollout testing plan, (5) monitoring and alerting thresholds. Tradeoff: choosing miles normalizes exposure but can hide scenario-specific risk (e.g., intersections per mile); call out a plan to report both intervention_rate_per_mile and intervention_rate_per-intersection. Close by stating validation steps (manual label audits, synthetic injection tests) and what you'd do with more time: estimate required miles to detect a 20% reduction using historical intervention_rate and fit a hierarchical Poisson model to borrow strength across cities.

A second angle — "Estimate miles required to detect a 10% reduction in collision rate"

Here the same primitives apply but constraints change: this is a power calculation for very rare events. Start by estimating baseline rate λ (e.g., collisions per million miles) and variance model (Poisson vs overdispersed). Use the exposure formula (see Core knowledge) to solve for required T; communicate results as orders of magnitude (e.g., “you need O(10⁸–10⁹) miles for small relative reductions”), and present alternatives: use composite endpoints including near-misses, extend study by synthetic scenario testing in simulation, or run paired comparisons within controlled ODDs to increase signal. Emphasize cost–benefit: if required miles are infeasible, propose higher-signal proxies or staged evaluation.

Common pitfalls

Pitfall: Misdefining denominator — reporting change in absolute count without normalizing for exposure (fewer miles driven leads to fewer events but not safer behavior). Always pair counts with exposure and normalized rates.

Pitfall: Normal-approximation mistakes — using z-tests/CIs for Poisson counts when k is very small leads to anti-conservative p-values. Use exact or Poisson-based intervals for small k.

Pitfall: Overclaiming causality — presenting uncontrolled before/after comparisons as evidence of safety improvement; state confounders, show stratified analyses, or use randomized/controlled designs when possible.

Connections

This topic naturally pivots to simulation validation (how simulation-derived risk correlates with field metrics), anomaly detection for incident triage, and causal inference (instrumental variables, difference-in-differences) when randomized experiments aren’t possible.