Design an A/B Test for Group Video Calls Impact

Q: Design an A/B Test for Group Video Calls Impact

This question evaluates experimental design and statistical inference competencies for A/B testing at scale, focusing on handling network interference, choosing the unit of randomization, clustering and design-effect-aware sample sizing, selecting appropriate tests for continuous and proportion metrics, and tailoring results communication for different stakeholders. Commonly asked in analytics and experimentation interviews, it probes both conceptual understanding of interference and trade-offs and practical application of sample-size calculations, cluster-adjusted analysis, and result presentation, with the domain classified as analytics & experimentation and the level bridging conceptual understanding and hands-on practical application.

Q: How do I approach Analytics & Experimentation interview questions?

Analytics & Experimentation questions require understanding of core concepts and practice. PracHub provides solutions with explanations to help you master analytics & experimentation interviews.

Question

A/B Experiment Design: Group Video Calls on Instagram

Scenario

Instagram wants to evaluate the impact of launching group video calls.

Task

Design an end-to-end experiment that accounts for strong network effects:

State a clear hypothesis, success metrics, and guardrails.
Choose the randomization unit (user, cluster, geography), explaining trade-offs under network interference.
Propose a sample size and duration plan (include formulas and a small numeric example). Account for clustering via design effects.
If clustering is infeasible, describe alternative designs to mitigate interference (e.g., two-stage/saturation, geo/switchback, encouragement designs).
Specify which statistical tests you would use for:
- Continuous metrics (e.g., time spent, calls per user).
- Proportion metrics (e.g., % of users who made any group call). Include differences under user-level vs. cluster/geo designs.
Outline how you would present results to non-technical PMs vs. data-science peers.

Hints

Discuss cluster vs. user-level assignment, geography splits, and how to measure/limit cross-arm contamination.
Contrast t-tests and z-tests; note cluster-robust methods and randomization inference for geo/cluster designs.
Tailor the communication: decision and business impact for PMs; assumptions, diagnostics, and methodology for DS peers.

Design an A/B Test for Group Video Calls Impact

A/B Experiment Design: Group Video Calls on Instagram

Scenario

Task

Hints

Solution

Comments (0)