Sensor Data Quality And Label Analytics

What's being tested

Interviewers are probing your ability to treat sensor telemetry and annotated labels as noisy measurement systems: quantify how measurement error affects key metrics, design sampling/adjudication to diagnose root cause, and recommend actionable mitigations that trade cost for statistical confidence. For Waymo, this matters because model decisions and safety metrics are computed from multi-sensor fused signals and human labels; the candidate must show rigorous statistical reasoning, experiment design, and metric-driven prioritization — not systems plumbing or labeler management.

Core knowledge

Label noise: random vs systematic noise; random adds variance and attenuates effect sizes, systematic creates bias. Model evaluation must separate the two to avoid misleading conclusions.
Confusion matrix: use per-class sensitivity (TPR), specificity (TNR), precision (PPV), and negative predictive value (NPV) to reason about observed vs true rates; label error changes both numerator and denominator.
Prevalence correction: if annotator sensitivity Se and specificity Sp are estimated, true prevalence p relates to observed prevalence p_obs as
$p=\frac{p_{obs} + Sp -1}{Se + Sp -1}$
Use when gold adjudication yields Se/Sp for noisy labels.
Annotator models: Dawid–Skene / EM estimates per-annotator confusion matrices and latent true labels; works for thousands of items but may overfit with few repeats — regularize or group annotators.
Agreement metrics: Cohen's kappa (pairwise), Krippendorff's alpha (multi-annotator, missing labels); interpret values contextually (0.6 may be fine for hard perception tasks).
Sampling & stratification: design stratified review by model confidence, edge-case flags, object size, time-of-day; prioritized sampling uncovers rare but important failure modes efficiently.
Adjudication design: use a gold set (expert-reviewed) vs majority-vote; for safety-critical labels prefer expert adjudication with documented rules, and measure inter-adjudicator agreement.
Impact on experiments: misclassification attenuates treatment effects in A/B tests; required sample size increases by factor roughly $1/(1 - 2\epsilon)^2$ for symmetric misclassification rate $\epsilon$ . Always propagate label uncertainty into power calculations.
Covariate vs label shift: distinguish covariate shift (p(x) changes) from label shift (p(y) changes) and conditional shift (p(y|x) changes); detection methods differ (density-ratio estimation vs confusion-matrix calibration).
Metric instrumentation: compute per-segment metrics (by sensor, lighting, object distance) and time-windowed metrics (p95, p99) to localize degradation; global aggregates often hide critical slices.
Noise-aware model evaluation: consider training with noise-robust losses, label smoothing, or reweighting by annotator reliability; evaluate on a reserved adjudicated test set, not noisy training labels.
Cost-effectiveness: quantify marginal value of more adjudication by modeling uncertainty reduction per adjudicated label and compare to expected safety or metric improvement.

Worked example — "Quantify labeler agreement and its impact on model performance"

First 30s: ask whether labels are multi-class or binary, whether multiple annotators per item exist, what adjudication standard (expert) is available, and the prevalence of positive cases. Skeleton: (1) estimate per-annotator confusion matrices from a doubly-labeled subset; (2) use Dawid–Skene or simple majority to infer latent truth and compute Se/Sp; (3) propagate Se/Sp into corrected precision/recall using the prevalence correction and adjusted confusion formulas; (4) design targeted adjudication on slices where annotator disagreement or model uncertainty is high. Key tradeoff: full adjudication gives unbiased metrics but is expensive — prioritize items that most affect safety metrics (e.g., low-confidence pedestrian detections at night). Close by noting you'd run a power analysis to decide adjudication budget and, if time allows, simulate the effect of different adjudication rates on downstream metric variance.

A second angle — "Detect and diagnose sensor-quality regressions that change validation metrics"

Here the framing shifts from labeler error to upstream sensor degradation manifesting as changed label distributions. Approach: (1) compare per-sensor and per-hardware-version slices over time, adjusting for traffic/context; (2) use unlabeled-signal drift detectors (e.g., feature-distribution divergence like KL or MMD) and correlate detected drift with rises in annotation disagreement or drops in per-slice precision; (3) run a targeted labeling push on periods flagged by drift detectors to confirm if label distribution or annotator confusion changed. The same statistical tools (confusion matrices, stratified sampling, adjudication) apply, but constraints emphasize temporal causality and faster triage.

Common pitfalls

Pitfall: Treating labels as ground truth.
Analytical mistake: reporting model precision/recall on noisy labels without quantifying annotator error leads to overconfidence or false regressions. Always estimate and report label uncertainty and correct metrics when possible.

Pitfall: Presenting only global aggregates.
Communication mistake: saying "precision dropped 3%" without per-sensor, lighting, or object-size slices hides whether degradation is safety-critical. Show segmentation early.

Pitfall: Overfitting annotator models on tiny repeats.
Depth mistake: running Dawid–Skene or per-annotator confusion estimation with too few doubly-labeled items yields unstable estimates; prefer pooling annotators by skill-level or using Bayesian priors.

Connections

This area often pivots into experiment design (how label noise affects A/B testing power), model calibration and reliability (calibration curves, expected calibration error), and active learning/data curation (prioritizing which frames to label or adjudicate). Interviewers may also ask about integrating these analyses into monitoring dashboards.