Determine Metrics for Evaluating Homepage Recommendation Carousel

Q: Determine Metrics for Evaluating Homepage Recommendation Carousel

This question evaluates a data scientist's competency in designing and analyzing A/B experiments for a homepage recommendation carousel, including selection of primary and secondary metrics, assessment of statistical power and practical significance, and the setup of guardrails.

Q: How do I approach Analytics & Experimentation interview questions?

Analytics & Experimentation questions require understanding of core concepts and practice. PracHub provides solutions with explanations to help you master analytics & experimentation interviews.

Question

Experiment Evaluation: Homepage Recommendation Carousel

Scenario

A product team has shipped an A/B test for a new recommendation carousel placed on the app's homepage. Users are randomly assigned at the user level to Control (no carousel) or Treatment (homepage shows the carousel). The goal is to improve user engagement and downstream conversions without harming overall experience or performance.

Task

Propose the primary metric(s) and secondary metric(s) you would track to evaluate the experiment.
Describe a clear decision framework to determine whether to roll the feature out to all users, including how you will handle statistical power, practical significance, and guardrails.

Assume a standard 50/50 split, sticky assignment, and at least one weekly cycle of traffic.

Determine Metrics for Evaluating Homepage Recommendation Carousel

Experiment Evaluation: Homepage Recommendation Carousel

Scenario

Task

Solution

Comments (0)