Design an A/B Test for Group Video Calls Impact
Overview
This question evaluates experimental design and statistical inference competencies for A/B testing at scale, focusing on handling network interference, choosing the unit of randomization, clustering and design-effect-aware sample sizing, selecting appropriate tests for continuous and proportion metrics, and tailoring results communication for different stakeholders. Commonly asked in analytics and experimentation interviews, it probes both conceptual understanding of interference and trade-offs and practical application of sample-size calculations, cluster-adjusted analysis, and result presentation, with the domain classified as analytics & experimentation and the level bridging conceptual understanding and hands-on practical application.
A/B Experiment Design: Group Video Calls on Instagram
Scenario
Instagram wants to evaluate the impact of launching group video calls.
Task
Design an end-to-end experiment that accounts for strong network effects:
- State a clear hypothesis, success metrics, and guardrails.
- Choose the randomization unit (user, cluster, geography), explaining trade-offs under network interference.
- Propose a sample size and duration plan (include formulas and a small numeric example). Account for clustering via design effects.
- If clustering is infeasible, describe alternative designs to mitigate interference (e.g., two-stage/saturation, geo/switchback, encouragement designs).
-
Specify which statistical tests you would use for:
- Continuous metrics (e.g., time spent, calls per user).
- Proportion metrics (e.g., % of users who made any group call). Include differences under user-level vs. cluster/geo designs.
- Outline how you would present results to non-technical PMs vs. data-science peers.
Hints
- Discuss cluster vs. user-level assignment, geography splits, and how to measure/limit cross-arm contamination.
- Contrast t-tests and z-tests; note cluster-robust methods and randomization inference for geo/cluster designs.
- Tailor the communication: decision and business impact for PMs; assumptions, diagnostics, and methodology for DS peers.
Solution
Login required