How do you diagnose a ratio metric change
Company: Google
Role: Data Scientist
Category: Analytics & Experimentation
Difficulty: medium
Interview Round: Onsite
In an A/B test, the treatment group shows a statistically significant increase in a ratio metric:
- **CTR = clicks / impressions** increased by +1.2% relative.
However, product stakeholders are unsure whether this is a real improvement or an artifact.
Explain how you would diagnose *why* CTR changed and whether the change is trustworthy. Your answer should address:
1) Decomposing the ratio into numerator/denominator movements.
2) Composition effects (e.g., traffic mix shifts) and Simpson’s paradox.
3) Guardrails / invariants to check for instrumentation or ranking changes.
4) Statistical considerations for ratio metrics (e.g., variance estimation, delta method vs. bootstrap).
5) What follow-up analyses or experiment iterations you would run.
Quick Answer: Evaluates diagnostic and statistical reasoning for experiment analysis—specifically decomposition of ratio metrics into numerator/denominator movements, detection of composition effects and Simpson’s paradox, instrumentation and guardrail checks, and appropriate variance estimation methods.