Estimate variance for ratio metrics
Overview
This question evaluates statistical inference and experimental-analysis skills for data scientists, including variance estimation for ratio metrics via the delta method, covariance estimation from sample moments, model-based comparisons (ANCOVA/log models with offsets), bootstrap and t-test inference considerations, and robustness techniques for heavy-tailed revenue such as winsorization and robust M-estimators. It is commonly asked to assess both theoretical derivation and practical estimation trade-offs in A/B or experimental settings; the category is Statistics & Math and the level of abstraction spans conceptual understanding and practical application.
KPI Variance via Delta Method and Inference Choices for ARPU
Context
You run experiments where each arm produces aggregate totals per analysis unit (e.g., day, country-day, or bucket):
- Revenue (R)
- ActiveUsers (A)
Your KPI is ARPU = R/A computed per unit, then compared across arms. You are told corr(R, A) = 0.6 at the unit level.
Tasks
- Using the delta method for the ratio g(R, A) = R/A, derive an approximate Var(ARPU) that includes the covariance term.
- State clearly how you would estimate all inputs (means, variances, covariance/correlation) from sample moments.
-
Compare two analysis strategies:
- Testing ARPU directly (ratio of totals per unit).
- Modeling log-Revenue with log-ActiveUsers as a covariate (ANCOVA-style), including the offset variant.
-
Guidance on inference choices:
- When is a t-test acceptable?
- When prefer a bootstrap (and which kind)?
- Handling heavy-tailed revenue: recommendations on winsorization and robust M-estimators.
Solution
Login required