Evaluates demand estimation, metrics, and experiment design for WhatsApp group video calls. Strong answers infer latent demand from call and group-chat behavior, request additional research data, define success and guardrail metrics, and design a clustered test that handles network effects.
##### Scenario
WhatsApp is considering launching a group video-call feature; currently it does not exist.
##### Question
a) Using the available call-level data, outline how you would determine whether there is demand for group calls.
b) What additional quantitative and qualitative data would you request, and how would you analyze it?
c) After launch, list success and guard-rail metrics; if you could keep only one success metric, which one and why?
d) Before launch, engineering asks for an experiment design—describe step-by-step how you would run and analyze the test, including randomization unit, clustering to reduce spill-over, novelty effects, and criteria for statistical & business significance.
##### Hints
Frame analysis: opportunity sizing → metric hierarchy → clustered A/B test design → interpretation of lift vs. risk.
Quick Answer: Evaluates demand estimation, metrics, and experiment design for WhatsApp group video calls. Strong answers infer latent demand from call and group-chat behavior, request additional research data, define success and guardrail metrics, and design a clustered test that handles network effects.
WhatsApp is considering launching group video calls. Assume the feature does not currently exist, but call-level and messaging data are available for existing one-to-one communication.
Constraints & Assumptions
The task is to estimate demand, define success, and design a pre-launch or staged-launch experiment.
Existing one-to-one call data can provide demand proxies but cannot directly observe true group-call demand.
WhatsApp products have strong network effects, so experiment design should consider spillovers within groups and social clusters.
Include success metrics, guardrails, and a clear recommendation framework.
Clarifying Questions to Ask Guidance
Are group audio calls or group chats already available?
Which markets, devices, and network conditions are in scope for launch?
What is the primary goal: user retention, more meaningful communication, competitive parity, or monetization?
Are there engineering constraints on participant count, call quality, or rollout speed?
Part 1 - Estimate Demand from Existing Data
Using available call-level data, how would you determine whether there is demand for group video calls?
What This Part Should Cover Guidance
Proxies such as rapid sequential one-to-one calls, overlapping calls among group-chat members, repeated missed calls, and group-chat coordination before calls.
Opportunity sizing by eligible users, active callers, group-chat density, device/network readiness, and likely use cases.
Segmentation by market, cohort, relationship type, device quality, and current call intensity.
Part 2 - Request Additional Data
What additional quantitative and qualitative data would you request, and how would you analyze it?
What This Part Should Cover Guidance
Surveys, user interviews, beta waitlists, search/support signals, app-store feedback, and competitor or market benchmarks where appropriate.
Group chat context, invite flows, failed coordination, call intent, call quality, and device/network constraints.
Analyses that estimate frequency, group size, willingness to use, and current substitutes.
Part 3 - Define Metrics
After launch, list success and guardrail metrics. If you could keep only one success metric, which one and why?
What This Part Should Cover Guidance
Primary metrics such as incremental weekly active group callers, successful group-call sessions, or completed group-call minutes.
Funnel metrics for call creation, invites, joins, connection success, completion, repeat use, and retention.
Guardrails for one-to-one call cannibalization, messaging cannibalization, dropped calls, latency, crashes, abuse, privacy, and cost.
A defensible single metric tied to incremental, high-quality group communication.
Part 4 - Design an Experiment
Before launch, engineering asks for an experiment design. Describe step by step how you would run and analyze the test, including randomization unit, clustering, novelty effects, and launch criteria.
What This Part Should Cover Guidance
Cluster or social-graph-aware randomization to reduce spillover.
Analysis of direct and network effects, novelty decay, statistical significance, business significance, and guardrail thresholds.
What a Strong Answer Covers Guidance
A strong answer estimates latent demand from behavioral proxies, supplements it with research, defines incremental success metrics, and proposes an experiment that handles network effects and quality guardrails.
Follow-up Questions Guidance
How would you handle users whose friends are split between treatment and control?
What would you recommend if survey demand is high but behavioral proxies are weak?
How long would you run the experiment to account for novelty effects?