Determine Success Metrics for New Group Video-Call Feature
Quick Overview
Evaluates how to estimate demand and define success for a new Meta group video-call feature. Strong answers combine behavioral proxies, survey or research data, launch metrics, participant-cap analysis, call-quality guardrails, and network-effect-aware experimentation.
Determine Success Metrics for New Group Video-Call Feature
Company: Meta
Role: Data Scientist
Category: Analytics & Experimentation
Difficulty: hard
Interview Round: Onsite
##### Scenario
Meta is exploring a new group video-call feature.
##### Question
Beyond the provided tables, what additional data sources or surveys would you use to measure demand for group video calls? How would you define and track success metrics once the feature is launched? What framework would you apply to decide an upper bound on participants allowed in one group call?
##### Hints
Consider task completion, concurrency, call quality, user satisfaction, infra cost.
Quick Answer: Evaluates how to estimate demand and define success for a new Meta group video-call feature. Strong answers combine behavioral proxies, survey or research data, launch metrics, participant-cap analysis, call-quality guardrails, and network-effect-aware experimentation.
Determine Success Metrics for a New Group Video-Call Feature
Meta is exploring a new group video-call feature. You need to estimate demand before launch, define success after launch, and recommend how to think about the maximum number of participants allowed in one call.
Constraints & Assumptions
Assume standard product logs are available for existing messaging, one-to-one calls, group chats, invites, joins, leaves, and call quality.
The feature should create incremental value, not just move users from existing calls or messages into a new surface.
Participant limits should account for user value, call quality, device constraints, abuse risk, and infrastructure cost.
You may propose surveys or research, but the answer should be grounded in measurable data.
Clarifying Questions to Ask Guidance
Is the feature for a messaging app, a social app, or both?
Are group audio calls already available, or is this the first group calling product?
What does leadership care about most: engagement, retention, meaningful interactions, revenue, or strategic parity?
Are there hard engineering limits on media quality, latency, or maximum participants?
Part 1 - Measure Pre-Launch Demand
Beyond existing tables or logs, what additional data sources or surveys would you use to measure demand for group video calls?
What This Part Should Cover Guidance
Behavioral proxies such as group-chat activity, sequential one-to-one calls, failed coordination, call invitations, and repeated multi-person planning.
Qualitative research and surveys that ask about use cases, group size, frequency, willingness to use, and current substitutes.
Market or competitive signals, support tickets, search queries, and beta/waitlist demand when available.
Segmentation by region, device capability, network conditions, social graph density, and existing call behavior.
Part 2 - Define Launch Success
How would you define and track success metrics once the feature is launched?
What This Part Should Cover Guidance
A primary metric such as incremental weekly active group callers, successful group-call minutes, or quality group-call sessions.
Funnel metrics for creation, invitations, joins, connection success, retention, repeat usage, and call completion.
Guardrails for one-to-one call cannibalization, message engagement, crashes, latency, dropped calls, abuse reports, and cost per minute.
Cohort and network-effect measurement so the analysis does not undercount multi-user spillovers.
Part 3 - Choose a Participant Limit
What framework would you apply to decide an upper bound on participants allowed in one group call?
What This Part Should Cover Guidance
Distribution of intended group sizes, call completion by size, quality degradation, device/network constraints, and infrastructure cost.
Experimentation or staged rollout across size caps where feasible.
A decision rule that chooses the smallest cap that covers most valuable use cases while protecting reliability and cost.
What a Strong Answer Covers Guidance
A strong answer links demand estimation, metric design, and participant-cap decisions into one framework, and explicitly handles quality, cost, and network spillovers.
Follow-up Questions Guidance
How would you randomize an experiment when users in the same group influence each other?
What would you do if large calls drive minutes but have poor quality?
Which one metric would you put on an executive dashboard?