Determine Metrics for Group-Video Calling Experiment Success
Quick Overview
Evaluates metrics and experiment design for launching group video calling in a messaging app. Strong answers review demand and quality data, choose an incremental group-communication metric, set participant limits, and design an A/B test with guardrails, power, and network-spillover handling.
Determine Metrics for Group-Video Calling Experiment Success
Company: Meta
Role: Data Scientist
Category: Analytics & Experimentation
Difficulty: medium
Interview Round: Technical Screen
##### Scenario
Product team debating whether to launch group-video calling; you must plan an experiment and choose metrics.
##### Question
a) Which additional data sources or reports would you review before deciding to add group calls?
b) What primary success metric would you track and why?
c) How would you decide the maximum participant limit?
d) Outline an A/B-test design (control vs treatment, metric definition, statistical considerations).
##### Hints
Address user need, baseline CTR, capacity, guardrail metrics, power, and comparing treatment to control—not to itself.
Quick Answer: Evaluates metrics and experiment design for launching group video calling in a messaging app. Strong answers review demand and quality data, choose an incremental group-communication metric, set participant limits, and design an A/B test with guardrails, power, and network-spillover handling.
Determine Metrics for Group-Video Calling Experiment Success
You are the data scientist for a large consumer messaging app that currently supports one-to-one video calls. The product team is considering group video calling with three or more participants, and you need to evaluate the decision with data and experiments.
Constraints & Assumptions
Treat this as a product analytics and experiment-design question.
Group calling may have network effects because multiple users participate in the same call.
Success should include user value, call quality, retention, and cost guardrails.
Avoid comparing treatment users to themselves; define a valid control.
Clarifying Questions to Ask Guidance
Are group audio calls or group chats already available?
What user problem is group video intended to solve?
Are there engineering constraints around call quality, participants, and cost?
Is the launch market-specific or global?
Part 1 - Pre-Launch Data
Which additional data sources or reports would you review before deciding to add group calls?
What This Part Should Cover Guidance
Group-chat behavior, one-to-one call funnels, sequential calls, survey and UX research, competitor benchmarks, support tickets, and infrastructure reports.
Demand, participant-size distribution, device/network readiness, and safety or abuse risk.
Part 2 - Primary Success Metric
What primary success metric would you track and why?
What This Part Should Cover Guidance
A metric tied to incremental high-quality group communication, such as successful group-call sessions per eligible user or repeat group callers.
Supporting metrics for invite, join, completion, duration, retention, and satisfaction.
Part 3 - Participant Limit
How would you determine an initial maximum participant limit?
What This Part Should Cover Guidance
Intended group-size demand, call quality by size, latency, drop rate, device constraints, abuse risk, and infrastructure cost.
Staged rollout or experiments with caps.
Part 4 - A/B Test Design
Outline an A/B test design, including treatment, control, metrics, guardrails, power, and comparison plan.
What This Part Should Cover Guidance
Eligibility, randomization unit, treatment and control definitions, network spillover handling, MDE, duration, and guardrails.
Direct comparison of treatment versus control with pre-specified metrics.
What a Strong Answer Covers Guidance
A strong answer estimates demand, chooses an incremental user-value metric, accounts for quality and cost, and designs an experiment that handles group-level spillovers and valid comparisons.
Follow-up Questions Guidance
How would you randomize if users invite friends across treatment and control?
What if group calls increase minutes but reduce message activity?