Design A/B Test for Streaming Feature Network Effects
Company: Netflix
Role: Data Scientist
Category: Analytics & Experimentation
Difficulty: hard
Interview Round: Other
##### Scenario
Designing and evaluating an A/B test for a new streaming feature that may create social-network spill-overs.
##### Question
How would you choose primary and guardrail metrics for this experiment? What threats do network effects pose and how would you address them? Explain SUTVA and why it matters here. Describe how you would run and interpret a multivariate test on title artwork. How would you measure heterogeneous treatment effects across user segments?
##### Hints
Cover metric definition, experiment unit, variance reduction, and validity assumptions.
Quick Answer: This question evaluates a candidate's understanding of online experiment design and causal inference under network spillovers, covering metric selection, experimental unit definition, SUTVA, variance-reduction techniques, multivariate testing, and heterogeneous treatment effects.