Design and Analyze A/B Test for Recommendation Widget

Q: Design and Analyze A/B Test for Recommendation Widget

This question evaluates a candidate's skills in experimental design, randomization and exposure logic, sample-size and power calculations, metric selection and causal/statistical analysis, instrumentation and monitoring, and stakeholder communication for product experiments.

Q: How do I approach Analytics & Experimentation interview questions?

Analytics & Experimentation questions require understanding of core concepts and practice. PracHub provides solutions with explanations to help you master analytics & experimentation interviews.

Question

Scenario

You are designing and analyzing an online A/B test for launching a new recommendation widget in a consumer-facing product (e.g., mobile and web app). The widget recommends relevant actions or products on a home/feed surface.

Task

Explain, end-to-end, how you would set up, run, and analyze an A/B test for this recommendation widget.

Requirements

Experiment design and randomization
- Define hypothesis, unit of assignment, eligibility/exposure, and rollout plan.
Sample size and power
- Determine minimum detectable effect (MDE), sample size, duration, and traffic ramp.
Metrics
- Choose a single primary metric, key secondary metrics, and guardrail metrics; define how each is computed.
Execution and debugging
- Instrumentation, logging, pre-checks (e.g., SRM), and live monitoring.
Analysis
- Statistical tests, variance reduction, handling triggered exposure vs ITT, and multiple comparisons.
Pitfalls
- List issues that could invalidate the experiment and how you would detect/mitigate them.
Communication
- How you would communicate results and a go/no-go recommendation to stakeholders.

Assume users can be exposed multiple times across sessions and platforms, and there is no cross-user network effect.

Design and Analyze A/B Test for Recommendation Widget

Scenario

Task

Requirements

Solution

Comments (0)