Design an Effective A/B Test for Algorithm Launch
Design an A/B Test for a New Mobile App Recommendation Algorithm
Context
A mobile app team plans to ship a new recommendation algorithm that ranks content in the app. Assume:
-
We can randomize at the user level with sticky assignment.
-
Traffic is sufficient to run a 50/50 A/B split after a brief ramp.
-
The algorithm may change engagement and performance (e.g., latency).
Task
Describe, end-to-end, how you would design and run this A/B test:
-
Experiment design
-
Unit of randomization, assignment, segmentation, and ramp plan.
-
Exposure definition and logging/instrumentation.
-
Analysis plan and stopping rules.
-
Metrics and success criteria
-
Primary, secondary, and guard-rail metrics.
-
How you will define success and launch criteria.
-
Sample size and test duration
-
How to determine required sample size and duration.
-
Include formulas and a small numeric example.
-
Common pitfalls and mitigations
-
Discuss randomization, segmentation, statistical power, guard-rail metrics, stopping rules, and launch criteria.
Constraints & Assumptions
-
Preserve the scope, facts, inputs, and requested outputs from the prompt above.
-
If the prompt leaves a detail unspecified, state a reasonable assumption before relying on it.
-
Keep the answer interview-ready: concise enough to present, but concrete enough to implement or evaluate.
Clarifying Questions to Ask
-
Clarify the business objective, unit of analysis, time window, exposure definition, and primary metric.
-
State assumptions about instrumentation, randomization, sample size, and data quality.
-
Separate descriptive analysis from causal claims.
What a Strong Answer Covers
-
A metric framework with primary, guardrail, and diagnostic metrics.
-
A credible analysis or experiment design with clear assumptions and bias checks.
-
SQL/statistical logic for segmentation, variance, confidence, and data validation where relevant.
-
An actionable recommendation that explains trade-offs and next steps.
Follow-up Questions
-
What sanity checks would you run before trusting the result?
-
How would you handle novelty effects, seasonality, or selection bias?
-
What decision would you make if metrics disagree?