Compare two rare-event detection models statistically

Read the full interview experience this question came from →

Quick Overview

This question evaluates a candidate's competency in statistical model evaluation for rare-event detection, covering selection of appropriate metrics for imbalanced data, estimation of confidence intervals and hypothesis testing with small samples, and considerations such as calibration, thresholding, paired versus unpaired comparisons, and cost/alert-budget trade-offs. Commonly asked in Machine Learning and data-science interviews, it tests both conceptual understanding of statistical assumptions and practical application of limited aggregated results to quantify uncertainty and support decision-making, and is categorized under Machine Learning and statistical inference with emphasis on both conceptual and practical levels of abstraction.

Compare two rare-event detection models statistically

Company: Waymo

Role: Data Scientist

Category: Machine Learning

Difficulty: easy

Interview Round: Onsite

You are evaluating two models (Model A and Model B) for **rare-event detection** (e.g., fraud, abuse, medical adverse event). Positives are extremely rare. You are given only limited evaluation results (e.g., a small number of aggregated counts such as TP/FP/FN/TN, or precision/recall at a chosen threshold) for each model—assume you have enough information to derive confusion-matrix counts, but the number of positives is small. ### Questions 1. Which metrics are most appropriate for comparing the models in a rare-event setting (and why not accuracy/ROC-AUC alone)? 2. How would you compare Model A vs Model B **with statistical uncertainty**? - Write down the relevant distributions/assumptions (e.g., binomial) and how you’d compute confidence intervals. - How would you test whether one model is significantly better? 3. If you only have “a few numbers” (small sample), what would you do to make a decision responsibly? Include thresholds, calibration, and cost/alert-budget considerations. State your assumptions (paired vs unpaired evaluation, fixed threshold vs full curve, etc.).

Overview: This question evaluates a candidate's competency in statistical model evaluation for rare-event detection, covering selection of appropriate metrics for imbalanced data, estimation of confidence intervals and hypothesis testing with small samples, and considerations such as calibration, thresholding, paired versus unpaired comparisons, and cost/alert-budget trade-offs. Commonly asked in Machine Learning and data-science interviews, it tests both conceptual understanding of statistical assumptions and practical application of limited aggregated results to quantify uncertainty and support decision-making, and is categorized under Machine Learning and statistical inference with emphasis on both conceptual and practical levels of abstraction.

Read the full Waymo Data Scientist interview experience this question came from

Community answers

Answer by shivam_bhatt

Precision and Recall are better. No accuracy as data is skewed For ROC-AUC. As there are lot of negatives, a small false positive rate can lead to lot of alerts Why need statistics ; Let say model A catches 10 cases and model B catches 8 cases, but we have to see if model A wasn't just lucky. Statistical uncertainty means acknowledging that a result from a limited sample may differ from the model’s underlying performance. Use Binomial distribution - To predict the success of an event is fixed number of attemps. To compute confidence interval( A 95% confidence interval means that if we repeatedly collected samples and constructed intervals this way, about 95% of those intervals would contain the true population.) For small sample use Wilson and exact binomial intervals To test if model A is better or not - Create Null hypotheses(If the models were equally good, how surprising would a difference like this be just from sampling randomness Calculate p-value McNemar’s test : Fraud caught by A but missed by B. and vice-versa Fisher exact test : used to assess the association between two binary variables If both models were tested on the same cases, I would use their disagreements to compare recall with an exact McNemar test. For independent samples, I could use Fisher’s exact test.
|Home/Waymo/Machine Learning
Waymo logo
Waymo
Jan 17, 2026
easyData ScientistOnsiteMachine Learning
81
0

You are evaluating two models (Model A and Model B) for rare-event detection (e.g., fraud, abuse, medical adverse event). Positives are extremely rare.

You are given only limited evaluation results (e.g., a small number of aggregated counts such as TP/FP/FN/TN, or precision/recall at a chosen threshold) for each model—assume you have enough information to derive confusion-matrix counts, but the number of positives is small.

Questions

  1. Which metrics are most appropriate for comparing the models in a rare-event setting (and why not accuracy/ROC-AUC alone)?
  2. How would you compare Model A vs Model B with statistical uncertainty ?
    • Write down the relevant distributions/assumptions (e.g., binomial) and how you’d compute confidence intervals.
    • How would you test whether one model is significantly better?
  3. If you only have “a few numbers” (small sample), what would you do to make a decision responsibly? Include thresholds, calibration, and cost/alert-budget considerations.

State your assumptions (paired vs unpaired evaluation, fixed threshold vs full curve, etc.).

Browse More Questions

More Machine Learning•More Waymo•More Data Scientist•Waymo Data Scientist•Waymo Machine Learning•Data Scientist Machine Learning