How to Design Effective A/B Tests for Onboarding
Company: Netflix
Role: Data Scientist
Category: Analytics & Experimentation
Difficulty: medium
Interview Round: Onsite
##### Scenario
Product team plans to launch a redesigned onboarding flow and needs evidence it increases activation.
##### Question
Design an A/B test for the new onboarding. State hypothesis, unit of randomization, key metrics, guardrail metrics, and runtime calculation. If early results show uplift but increased support tickets, how would you decide whether to launch?
##### Hints
Address sample size, power, sequential checks, and balancing primary vs. secondary metrics.
Quick Answer: This question evaluates a data scientist's competency in experimental design, causal inference, metric selection, sample size and power calculation, sequential monitoring, and decision-making under trade-offs for an onboarding A/B test within the Analytics & Experimentation domain, reflecting a practical application with conceptual understanding.