Choose robust metrics for skewed comments
Company: Meta
Role: Data Scientist
Category: Statistics & Math
Difficulty: hard
Interview Round: Onsite
A website’s per-user daily comment counts are extremely skewed and zero-inflated. You roll out a backend optimization expected to increase engagement.
(a) Explain when mean, median, trimmed mean (10%), winsorized mean (95/5), and geometric mean of (1+count)−1 are preferable estimators of central tendency for such data. Discuss bias/variance trade-offs under heavy tails (e.g., Pareto) and interpretability for product decisions.
(b) Suppose the control group’s per-user counts for a day are [0,0,0,1,1,2,2,3,20,50] and treatment’s are [0,0,1,1,1,2,2,3,5,10]. Compute mean, median, 10% trimmed mean, and winsorized mean for each, and determine which estimator most reliably detects a practically meaningful improvement here. Justify rigorously.
(c) Describe how you would form a 95% CI for your chosen estimator using nonparametric bootstrap with stratification by user activity buckets. State assumptions and how you’d check them.
(d) If you must report an effect size that’s robust but comparable across experiments, propose a transformation and effect metric (e.g., log1p-based percent change or quantile treatment effect at τ=0.8) and defend its choice.
Overview: This question evaluates understanding of robust estimation and inference for zero‑inflated, heavy‑tailed count data, including central tendency choices (mean, median, trimmed and winsorized means, geometric mean), nonparametric bootstrap confidence intervals, and robust effect‑size transformations.
Read the full Meta Data Scientist interview experience this question came from
Community answers
Answer by SS
(a)
(a) Which estimator to use?
Mean
Use when: You care about total impact (overall comments)
Problem: Gets heavily affected by a few very active users
👉 Can be noisy with heavy tails
Median
Use when: You care about the “typical user”
Problem: With many zeros, median may stay 0 even if things improve
👉 Too insensitive here
10% Trimmed Mean
Use when: You want to ignore extreme low & high values
Problem: Removes data → can distort results (especially with many zeros)
👉 Good balance, but loses data
Winsorized Mean (95/5)
Use when: You want robustness but keep all data
How: Cap extreme values instead of removing them
👉 Best practical choice in many cases
Geometric Mean of (1 + count) − 1
Use when: You want to reduce impact of big outliers smoothly
Problem: Harder to explain, not tied to total counts
👉 Useful but less intuitive
Answer by SS
Mean
Control: sum = 79 → Mean = 7.9
Treatment: sum = 25 → Mean = 2.5
👉 Looks like control > treatment (driven by 20 & 50)
Median
Control: middle = (1 + 2)/2 = 1.5
Treatment: middle = (1 + 2)/2 = 1.5
👉 No difference
10% Trimmed Mean
(remove 1 smallest & 1 largest)
Control:
Trim → [0, 0, 1, 1, 2, 2, 3, 20]
Sum = 29 → 3.625
Treatment:
Trim → [0, 1, 1, 1, 2, 2, 3, 5]
Sum = 15 → 1.875
👉 Control still higher, but gap is smaller
Winsorized Mean (95/5)
(replace min with 2nd smallest, max with 2nd largest)
Control:
Winsorized → [0, 0, 0, 1, 1, 2, 2, 3, 20, 20]
Sum = 49 → 4.9
Treatment:
Winsorized → [0, 0, 1, 1, 1, 2, 2, 3, 5, 5]
Sum = 20 → 2.0
👉 Control still higher, but extreme effect reduced