Design and evaluate a new group call feature
Overview
This question evaluates a candidate's competency in product analytics, experimentation design, causal inference under network interference, metric definition, and tradeoff analysis for launching a group call feature in a messaging product, and is targeted to the Analytics & Experimentation domain for a Data Scientist role.
Meta
Dec 8, 2025, 12:00 AM
Data Scientist
Technical Screen
Analytics & Experimentation
0
0
Product / DS Case: Group Calls for Messenger Groups
Messenger has Groups but does not currently support group calls. You are evaluating whether to build and launch a group call feature.
Answer the following in a product-sense + data-science way:
- User need / problem validation
- How would you determine whether users actually need group calls (vs. alternatives like group chat, voice notes, 1:1 calls, or external apps)?
- What data and qualitative signals would you use?
- Choosing “group size” for the feature
- How would you decide what group sizes to support (e.g., max participants: 4, 8, 16, …)?
- What are the tradeoffs (user value, technical cost, quality/reliability, abuse/spam, discoverability)?
- Measuring success Propose:
- a primary success metric (or a small set),
- diagnostic metrics,
- guardrail metrics. Explain why.
- Experiment design under interference / network effects Because users are connected in groups, outcomes may spill over between treated and control users.
-
If you randomize at a
cluster level
(e.g., groups or user clusters), how do you:
- define clusters,
- avoid clusters that are still connected (spillover), or clusters that are too large,
- analyze the experiment correctly?
- A/B test and tradeoffs
- Propose an A/B test plan (unit of randomization, duration, eligibility, rollout).
- Discuss key tradeoffs and what could go wrong (e.g., novelty effects, seasonality, infra constraints, measurement gaps).
Solution
Login required
Show