Decide and justify product metrics amid trade-offs
Overview
This question evaluates proficiency in product metric selection, statistical experiment design, causal inference, and post-launch monitoring for feed-ranking trade-offs, assessing competencies in metrics engineering, A/B testing, variance reduction techniques, and decision-framework reasoning within the Analytics & Experimentation domain for a Data Scientist role. It is commonly asked to gauge a candidate's ability to justify metric choices under competing business objectives, design defensible randomized experiments, and specify monitoring and alerting strategies, and it tests both practical application (hands-on experiment setup and monitoring) and conceptual understanding (statistical properties, bias, heterogeneity, and trade-off reasoning).
Smart Sort Feed Ranking: Metrics, Experiment, Decisions, and Monitoring
Context: You are introducing a new Smart Sort ranking for a content feed. It should improve relevance/engagement but might reduce short-term ad impressions. You must choose a primary success metric, define guardrail metrics, design a defensible A/B test, and specify a principled decision framework and post-launch monitoring.
Tasks
- Metric choice
- From candidates {7-day retention, session minutes per DAU, revenue per DAU, CTR, creator supply health}, choose one primary metric and 2–3 guardrails.
- For each: justify the choice using statistical properties (variance, sensitivity to bots/heavy tails, weekday stability, Simpson’s paradox risks).
- Provide precise formulas, aggregation levels (per-user vs. per-session), and whether to use winsorization/log transforms.
- Experiment design
- Specify A/B design: unit of randomization (user vs. session), interference risks for ranking changes, and mitigations (e.g., sticky bucketing, ghost ads).
- Propose test duration.
- Compute the required sample size for an MDE of 1.0% relative change in the primary metric at 90% power and alpha=0.05. State the inputs required and how CUPED and/or stratification will reduce variance. Assume baseline is measured 2025-08-18 to 2025-08-31 and the test ends by 2025-09-01.
- Decision framework
- If the primary metric is up +0.8% but a guardrail (e.g., creator payout per impression, used as a proxy for supply health) is down −2.5% with p<0.05, describe a principled trade-off method (e.g., LTV delta combining engagement and revenue, constrained optimization, or multi-metric decision rules).
- Include handling of novelty effects, ramp strategies, sequential monitoring, and heterogeneous treatment effects (e.g., across countries).
- Post-launch monitoring
- Define the on-call dashboards and alert thresholds for the first week after a 10% rollout.
Solution
Login required