Design and Analyze A/B Test for Recommendation Widget
Overview
This question evaluates a candidate's skills in experimental design, randomization and exposure logic, sample-size and power calculations, metric selection and causal/statistical analysis, instrumentation and monitoring, and stakeholder communication for product experiments.
Chime
Aug 4, 2025, 10:55 AM
Data Scientist
Technical Screen
Analytics & Experimentation
55
0
Scenario
You are designing and analyzing an online A/B test for launching a new recommendation widget in a consumer-facing product (e.g., mobile and web app). The widget recommends relevant actions or products on a home/feed surface.
Task
Explain, end-to-end, how you would set up, run, and analyze an A/B test for this recommendation widget.
Requirements
-
Experiment design and randomization
- Define hypothesis, unit of assignment, eligibility/exposure, and rollout plan.
-
Sample size and power
- Determine minimum detectable effect (MDE), sample size, duration, and traffic ramp.
-
Metrics
- Choose a single primary metric, key secondary metrics, and guardrail metrics; define how each is computed.
-
Execution and debugging
- Instrumentation, logging, pre-checks (e.g., SRM), and live monitoring.
-
Analysis
- Statistical tests, variance reduction, handling triggered exposure vs ITT, and multiple comparisons.
-
Pitfalls
- List issues that could invalidate the experiment and how you would detect/mitigate them.
-
Communication
- How you would communicate results and a go/no-go recommendation to stakeholders.
Assume users can be exposed multiple times across sessions and platforms, and there is no cross-user network effect.
Solution
Login required
Show