Design experiment for Group Calls with interference
Overview
This question evaluates a data scientist's competency in experimental design and causal inference for networked features, covering metrics selection (primary and guardrail metrics), strategies for mitigating interference via randomization or quasi-experimental approaches, exposure and contamination controls, rollout planning, and pre-registration and power analysis. Commonly asked in Analytics & Experimentation interviews for Data Scientist roles, it assesses reasoning about bias–variance trade-offs, engineering constraints, and validity diagnostics in interconnected user environments, requiring both conceptual understanding of identification and practical application of experiment implementation.
Design an Experiment for Group Calls in a 1:1 Calling App (with Network Interference)
You are adding a Group Calls feature to an existing 1:1 calling app. Design a robust experiment to measure the feature's impact when users can influence each other (network interference is likely).
Assume a large, global user base and that group calls allow a user to create a group and invite multiple contacts. Some users might only join via invites. You may gate creation or visibility to mitigate contamination.
Address the following:
(a) Metrics
- Select exactly one primary metric and 3–5 guardrail metrics.
- Explain when "total call volume" is misleading (e.g., cannibalization, unequal exposure) and when it could be acceptable.
- Propose user-normalized or rate-based alternatives to total call volume.
(b) Randomization Unit and Rollout
- Choose a randomization unit to mitigate interference (e.g., graph clusters, ego-clusters, geo switchbacks).
- Propose a rollout plan.
- Compare bias, variance, and engineering complexity trade-offs across options.
(c) If Cluster Randomization Is Not Feasible
- Propose a quasi-experimental plan: e.g., invitation-gated treatment, time-based/switchback assignment, exposure-weighted estimators, instrumental variables using staggered access, or difference-in-differences with pre-periods.
- List identifying assumptions and diagnostics (placebo tests, balance checks, spillover detection).
(d) Exposure, Triggers, and Contamination Controls
- Define exposure and triggers (creators vs. joiners; first exposure vs. ever-exposed).
- Specify exclusion criteria.
- Detail contamination controls (e.g., invite tokens only visible to treated egos).
(e) Pre-registration and Power
- Pre-register the analysis: hypotheses, windows, estimators, and stopping rules.
- Specify a minimal detectable effect (MDE) with plausible baselines.
- Include seasonality controls and how you'll handle novelty/ramp-up effects.
Solution
Login required