Determine Metrics for Evaluating Homepage Recommendation Carousel
Company: TikTok
Role: Data Scientist
Category: Analytics & Experimentation
Difficulty: medium
Interview Round: Onsite
##### Scenario
Product team launches an A/B test introducing a new recommendation carousel on the homepage.
##### Question
What primary and secondary metrics would you track to evaluate the experiment? Describe the decision framework you would use to determine whether to roll the feature out to all users.
##### Hints
Discuss click-through rate, conversion, guardrails, power, practical significance.
Quick Answer: This question evaluates a data scientist's competency in designing and analyzing A/B experiments for a homepage recommendation carousel, including selection of primary and secondary metrics, assessment of statistical power and practical significance, and the setup of guardrails.