Design A/B test and success metrics for new feature
Overview
This question evaluates a data scientist's competency in experimental design, metric selection and guardrails, power and sample-size calculations, logging and attribution specifications, and revenue impact estimation for a new feature on a social media platform.
Meta
Oct 13, 2025, 9:49 PM
Data Scientist
Onsite
Analytics & Experimentation
0
0
Instagram Collections 2.0 — Define Success, Experiment Design, and Measurement
Context: You are proposing a new Instagram feature, Shareable Collections 2.0, intended to increase creator activity and monetization. Design how you would define success, run the experiment, and decide whether to launch.
1) Define Success and Guardrails
- State the mission and choose primary North Star metrics (one engagement metric, one revenue metric).
- List at least three counter metrics to protect user experience and integrity.
2) Design the Experiment
- Choose user-level A/B test or geo-clustered A/B test and justify your choice considering network effects and contamination.
- Propose a stratification scheme using deciles of an engagement score. Outline randomization and a ramp plan.
- With baseline D30 retention = 30% and a target 1% relative lift, compute the required sample size per arm at 90% power, alpha = 0.05. State assumptions and the formula you use.
- Define stopping rules, pre-registration plan, and guardrail thresholds (e.g., crash rate, time spent).
3) Measurement and Decisioning
- Specify the exact logging spec (key events and fields) and how you would attribute revenue.
- Given: average 4 ad impressions per DAU, CTR = 1%, revenue per ad click = $0.50. If the feature increases ad clicks by 2% on 100M DAUs, estimate weekly incremental revenue. Include sensitivity and uncertainty bands.
- List reasons not to launch even if the test shows statistically significant wins (cover technical feasibility and long-term costs).
Solution
Login required
Show