Design experiment for Group Calls with interference

Q: Design experiment for Group Calls with interference

This question evaluates a data scientist's competency in experimental design and causal inference for networked features, covering metrics selection (primary and guardrail metrics), strategies for mitigating interference via randomization or quasi-experimental approaches, exposure and contamination controls, rollout planning, and pre-registration and power analysis. Commonly asked in Analytics & Experimentation interviews for Data Scientist roles, it assesses reasoning about bias–variance trade-offs, engineering constraints, and validity diagnostics in interconnected user environments, requiring both conceptual understanding of identification and practical application of experiment implementation.

Q: How do I approach Analytics & Experimentation interview questions?

Analytics & Experimentation questions require understanding of core concepts and practice. PracHub provides solutions with explanations to help you master analytics & experimentation interviews.

Q: What difficulty level is this interview question?

This is a hard difficulty Analytics & Experimentation question, commonly asked during Technical Screen rounds at Meta.

Q: What role is this question designed for?

This question is commonly asked for Data Scientist candidates at Meta during technical interviews.

Question

Design an Experiment for Group Calls in a 1:1 Calling App (with Network Interference)

You are adding a Group Calls feature to an existing 1:1 calling app. Design a robust experiment to measure the feature's impact when users can influence each other (network interference is likely).

Assume a large, global user base and that group calls allow a user to create a group and invite multiple contacts. Some users might only join via invites. You may gate creation or visibility to mitigate contamination.

Address the following:

(a) Metrics

Select exactly one primary metric and 3–5 guardrail metrics.
Explain when "total call volume" is misleading (e.g., cannibalization, unequal exposure) and when it could be acceptable.
Propose user-normalized or rate-based alternatives to total call volume.

(b) Randomization Unit and Rollout

Choose a randomization unit to mitigate interference (e.g., graph clusters, ego-clusters, geo switchbacks).
Propose a rollout plan.
Compare bias, variance, and engineering complexity trade-offs across options.

(c) If Cluster Randomization Is Not Feasible

Propose a quasi-experimental plan: e.g., invitation-gated treatment, time-based/switchback assignment, exposure-weighted estimators, instrumental variables using staggered access, or difference-in-differences with pre-periods.
List identifying assumptions and diagnostics (placebo tests, balance checks, spillover detection).

(d) Exposure, Triggers, and Contamination Controls

Define exposure and triggers (creators vs. joiners; first exposure vs. ever-exposed).
Specify exclusion criteria.
Detail contamination controls (e.g., invite tokens only visible to treated egos).

(e) Pre-registration and Power

Pre-register the analysis: hypotheses, windows, estimators, and stopping rules.
Specify a minimal detectable effect (MDE) with plausible baselines.
Include seasonality controls and how you'll handle novelty/ramp-up effects.

Design experiment for Group Calls with interference

Quick Overview

Design an Experiment for Group Calls in a 1:1 Calling App (with Network Interference)

Solution

Comments (0)