Variance Estimation for Ratio Metrics
Asked of: Data Scientist
Last updated
-
What it is Variance estimation for metrics that are ratios of two aggregates, like clicks/impressions (CTR), revenue/users (ARPU), or watch time/sessions. It quantifies uncertainty for these ratios so you can form confidence intervals and hypothesis tests, typically using the delta method, Fieller intervals, or bootstrap.
-
Why interviewers ask about it Product teams rely on ratio metrics daily, and naïve variance formulas can flip a launch decision. Employers like Meta expect you to know when delta is safe, when Fieller or bootstrap is needed, and how to do this at the randomization unit with correlated numerator/denominator data.
-
Core ideas to know
- Estimate at the randomization unit using ratio-of-sums, not mean of per-user ratios.
- Delta method: Var(Y/X) needs Var(Y), Var(X), and Cov(Y, X); covariance matters.
- For A/B, difference of ratios uses a multivariate delta across groups; arms are independent.
- Fieller’s theorem gives exact CIs for a ratio; intervals can be unbounded if denominators near zero.
- Bootstrap works broadly but can be slow; heavy tails may require winsorization or robust bootstrap.
- Log-scale percent change: delta on log(Y/X) yields near-symmetric intervals and easy interpretation.
- Guardrails: diagnose sample-ratio mismatch (SRM) and cluster correlation before trusting intervals.
-
A common pitfall Candidates average per-user rates and compute a standard error on that mean, ignoring that users with larger denominators should carry more weight. Others drop the covariance term and treat numerator and denominator as independent, underestimating uncertainty. Two-proportion z-tests are misapplied when denominators vary across users or sessions. Finally, failing to notice near-zero denominators can produce nonsensical (even unbounded) confidence intervals.
-
Further reading
- Applying the Delta Method in Metric Analytics (Deng et al., arXiv). Practical, industry-focused guide with worked examples for large-scale A/B testing. [https://arxiv.org/abs/1803.06336] (arxiv.org)
- NIST: Ratio of Means Confidence Interval (Fieller). Clear explanation of Fieller intervals and when they become unbounded. [https://www.itl.nist.gov/div898/software/dataplot/refman1/auxillar/ratimean.htm] (itl.nist.gov)
- Methods for confidence interval estimation of a ratio parameter (BMC Med Res Methodology). Compares delta, Fieller, and likelihood-based approaches with guidance on use. [https://bmcmedresmethodol.biomedcentral.com/articles/10.1186/1471-2288-5-32] (bmcmedresmethodol.biomedcentral.com)