Determine Metrics for Group-Video Calling Experiment Success
Overview
This question evaluates a data scientist's competency in experiment design, product analytics, metric selection, statistical power analysis, and capacity-aware feature evaluation within the Analytics & Experimentation domain.
Scenario
You are the data scientist for a large consumer messaging app that currently supports 1:1 video calls. The product team is debating whether to launch group video calling (3+ participants). You must propose how to evaluate this decision with data and experiments.
Tasks
(a) Which additional data sources or pre-read reports would you review before deciding to add group calls?
(b) What primary success metric would you track and why?
(c) How would you determine an initial maximum participant limit?
(d) Outline an A/B-test design, including: control vs treatment definition, metric definitions, guardrail metrics, power/sample-size planning, and how to compare treatment to control (avoid comparing the treatment group to itself).
Note: Address user need, baseline CTR, capacity constraints, guardrails, and statistical power.
Solution
Login required