Define Success Metrics for a Group Video Conference Feature Using 1:1 Call Data

A product already lets users make one-to-one video calls. The team now wants to launch video conferencing: video calls with more than two participants. Based on the information the team already has about 1:1 calls, define what success would look like for the video conference feature. The call data available today: - `video_calls` has one row per call, and every call is between exactly two people: `caller` (the user who made the call), `recipient` (the user who was called), `ds` (date of the call), `call_id` (unique per call) and `duration` (length of the call in seconds). - `dim_all_users` has one row per user per date: `user_id`, `age_bucket` (age category), `country`, `dau_flag` (1 if the user was a daily active user on that date, else 0) and `ds`. ### Clarifying Questions - Does a video conference mean any call with three or more participants, and is there a maximum group size? - Will the feature launch to everyone at once, or through an experiment that keeps a holdout group? - Is the product goal more people talking to each other in real time, more time spent in calls, or more overall activity in the app? - Over what period after launch should success be judged? ### Part 1 — Define success State the goal of the video conference feature, then define the metrics that would show it is succeeding: a primary metric, supporting metrics and guardrails. ```hint What group calls replace Think about what a user who wants to talk with several people at once does today with only 1:1 calls, and what should change for that user if the feature works. ``` #### What This Part Should Cover - A goal for the feature tied to the user problem it solves - One primary metric with an exact definition: unit, time window, numerator and denominator - Supporting adoption, engagement and call-quality metrics - Guardrails, including what happens to 1:1 calling and to overall activity ### Part 2 — Ground the metrics in the 1:1 call data Explain how you would use the existing 1:1 call data before launch to set baselines, size the opportunity and find the users most likely to adopt group calls. Then explain what the current log cannot record about a call with more than two people, and sketch the per-user daily summary you would build to track the feature after launch. ```hint Read the existing log Ask what the 1:1 table already reveals about users who talk to several different people within a short window, and which facts about a multi-party call cannot fit in a row with one caller and one recipient. ``` #### What This Part Should Cover - Baselines and opportunity sizing computed from 1:1 calls and the user table - The gaps in the current log for multi-party calls, and the data needed to fill them - A concrete per-user daily summary with segment columns such as country and age bucket ### What a Strong Answer Covers - A metric framework that connects the goal to a precisely defined primary metric - A clear view on cannibalization: group calls that replace 1:1 calls versus calls that are new - Segmentation by country and age bucket, and attention to novelty effects in early data - A plan to read the result causally, such as a holdout experiment or staged rollout, with a decision rule - Recognition that the data model must change for calls with more than two people ### Follow-up Questions - After launch, group call minutes are growing but total call minutes per user are flat. Is the feature a success? - A group call can include users from both the test group and the control group. How would you design the experiment so that the comparison stays valid? - Group calls are much longer on average than 1:1 calls. Would you use average call duration as a success metric? - Which metrics would you watch to catch group calls that users did not want to join?