Missing Data, Imputation, And Selection Bias

What's being tested

Interviewers probe your ability to reason about how missing data and selection bias affect analysis and model evaluation, and to pick defensible mitigation strategies for A/B tests, causal estimates, or production models. Meta cares because user opt-outs, instrumentation gaps, and differential dropout routinely distort `DAU`/retention metrics and treatment effect estimates; the interviewer wants to see clear assumptions, practical methods (imputation, weighting, censoring), and appropriate sensitivity checks.

Core knowledge

Missingness taxonomy: know the three mechanisms — MCAR (Missing Completely At Random), MAR (Missing At Random conditional on observed X), MNAR (Missing Not At Random depends on unobserved values) — and that only MCAR guarantees unbiased complete-case estimates without adjustment.
Complete-case analysis: dropping rows with missing values is unbiased under MCAR but reduces power; if missingness correlates with outcome or treatment, expect bias and report reduced effective N.
Single vs multiple imputation: single imputation (mean, median) underestimates variance; Multiple Imputation (MI) creates m datasets, fits model each, then pools with Rubin’s rules to capture imputation uncertainty.
Inverse Probability Weighting (IPW): weight observed rows by 1 / P(R=1 | X) where R is observed indicator; corrects for MAR if model for P(R=1|X) is well-specified, but increases variance when probabilities are small.
Model-based imputation / joint modeling: methods like multivariate normal EM or chained equations (`mice`) assume MAR conditional model; suitable when predictive relationships between features and missingness are strong.
Practical imputers and libs: `sklearn`'s `IterativeImputer` (model-based), `KNNImputer`, `SimpleImputer` for pipelines, and `mice`/`pan` in R for MI; remember production vs offline differences (latency, feature-store defaults).
Bias vs variance tradeoff: mean imputation reduces variance artificially, IPW inflates variance via large weights; MI balances bias correction and realistic uncertainty when m≥5 is typical.
Sensitivity analysis for MNAR: perform tipping-point analysis, selection-models or pattern-mixture models to quantify how large MNAR effects must be to change conclusions; report bounds rather than singlepoint corrections.
Missingness indicators: adding a binary indicator for "was missing" can capture informative missingness for prediction but can induce bias in causal estimation unless interpreted carefully.
Time-to-event / censoring: treat dropout as censoring — use survival analysis (Kaplan–Meier, Cox) if missingness is due to event timing; ensure censoring is independent conditional on covariates, or perform IPCW (inverse probability of censoring weights).
Diagnostics and tests: compare distributions of observed covariates by R, run logistic regression predicting R to detect MAR patterns, and use standardized mean differences to quantify imbalance after weighting or imputation.
Reporting and reproducibility: always state missingness rates by cohort/time, list assumptions (MCAR/MAR/MNAR), show pre/post diagnostics, and include sensitivity ranges so stakeholders see robustness.

Worked example — “Estimate treatment effect when retention is missing for a subset of users”

Frame: ask what fraction and pattern of missing retention, whether missingness differs by treatment, and what auxiliary features are available. Pillars: (1) quantify missingness by arm and covariates; (2) choose strategy under plausible mechanism (IPW or MI for MAR; bounds/sensitivity for MNAR); (3) compute primary ATE with uncertainty that includes missingness correction; (4) diagnostic and sensitivity checks. A strong candidate would first show a simple table: missing rate by arm and baseline `DAU` quintile. If missingness differs by arm, you cannot ignore it; consider IPW where weight $w_i = 1 / P(R_i=1 | X_i)$ estimated via logistic regression. Flag the tradeoff: IPW corrects bias but amplifies variance when some P(R=1|X)≈0. If time permits, run Multiple Imputation (m=20) using retention ~ treatment + X, pool estimates, and perform a tipping-point MNAR sensitivity to show how large unobserved bias must be to overturn result. Close by recommending collection of an auxiliary signal (e.g., server-side logs) to reduce MNAR risk.

A second angle — “Missing covariates in a ranking model used live”

Here the constraint is production: imputation must be fast and stable at inference. Steps: (1) audit which features are missing at train vs serving; (2) prefer deterministic, cheap imputations (global median with `missing` indicator) or learned embeddings for "missing" category; (3) validate that model calibration and ranking AUC do not degrade when applying the same imputation in offline-to-online. Emphasize difference from causal setting: predictive performance may improve by exploiting informative missingness via indicators, while causal estimates require accounting for confounding missingness. Also consider catastrophic drift if missing patterns change over time; add monitoring for feature missingness rates and population shift.

Common pitfalls

Pitfall: assuming MAR without checking — many examiners will trap you if you proceed to impute or weight without first showing diagnostics; always model P(R=1|X) and report imbalance diagnostics.

Pitfall: using mean imputation for inferential targets — it biases variance and correlations, leading to overconfident p-values; prefer MI or IPW for causal estimates.

Pitfall: failing to separate prediction vs causal goals — using missingness indicators helps prediction but can bias ATE unless interpreted and adjusted correctly; explicitly state your goal before choosing a method.

Connections

This topic naturally connects to causal inference (instrumental variables, unconfoundedness), survival analysis/censoring, and data quality/ETL issues (log-loss, late-arriving events) because upstream logging problems create systematic missingness that spills into analysis.