Choose tests under non‑normal, unequal variance
Overview
This question evaluates understanding of statistical inference for heavy‑tailed, heteroskedastic A/B test metrics, covering CLT conditions for t‑tests, variance‑robust tests, nonparametric and resampling approaches, log transformations and back‑transformation interpretation, and handling zero inflation within the Statistics & Math domain for Data Scientist roles. It is commonly asked because real‑world experimental metrics violate parametric assumptions, so interviewers probe both conceptual understanding of asymptotics and test assumptions and practical application of resampling, transformation, and two‑part modeling choices along with appropriate robustness checks.
Heavy-Tailed, Heteroskedastic Metrics in A/B Tests (AOV example)
Context: You are comparing two groups in an A/B test on a spend metric (e.g., Average Order Value per user over a period). The outcome is heavy‑tailed, non‑normal, and shows heteroskedasticity across groups.
Questions
a) Validity of t-tests
- Under what conditions is a two-sample t-test still valid via the Central Limit Theorem (CLT)?
- When does Welch’s t-test materially reduce Type I error inflation? Be specific about sample size, variance ratio, and tail behavior.
b) Nonparametric and resampling methods
- Evaluate Mann–Whitney U, permutation tests, and bootstrap confidence intervals (CIs) for mean vs median effects. When will each mislead decision‑making?
c) Log transformation of AOV
- If a teammate suggests log-transforming highly skewed AOV, what treatment effect does a log-scale comparison estimate?
- Show how to back-transform and interpret a mean difference on the log scale as a multiplicative effect on the original scale.
- When does “log‑then‑t‑test” bias estimates (e.g., zeros, log-normality violations, Duan smearing)?
d) Zero inflation (15% zeros)
- Compare delta‑lognormal / two‑part (hurdle) models versus trimmed means.
- Justify your choice and describe robustness checks you would run.
Solution
Login required