Evaluate Widget Impact on User Engagement with A/B Testing
Company: Yahoo
Role: Data Scientist
Category: Analytics & Experimentation
Difficulty: medium
Interview Round: Technical Screen
##### Scenario
Company is preparing to roll out a new in-app recommendation widget and needs evidence that it improves user engagement.
##### Question
Design an A/B experiment to evaluate the widget’s impact on daily active users and session length. Which primary and guardrail metrics would you track and why? How would you determine required sample size and runtime? What potential biases or implementation pitfalls must be addressed?
##### Hints
Think about unit of randomization, metric sensitivity, power calculation, and avoiding novelty or logging bias.
Quick Answer: This question evaluates a data scientist's experimental design and causal inference skills, covering A/B test setup, unit of randomization and exposure, metric selection and guardrails, sample size and power calculations, and identification of biases and logging or implementation pitfalls in the Analytics & Experimentation domain.