Design analytics and experiment for group video calls

Quick Overview

This question evaluates product analytics and experimentation competencies, including tie-strength and group-affinity measurement, event-level instrumentation design, proxy-product forecasting, participant-cap constraints and operational trade-offs, and experiment design that accounts for network effects and contamination.

Design analytics and experiment for group video calls

Company: Meta

Role: Data Scientist

Category: Analytics & Experimentation

Difficulty: hard

Interview Round: Technical Screen

You are evaluating a new Group Video Call feature in a messaging app. Using only product analytics (no user surveys), answer the following in depth: 1) Analyze relationships between users: Given access to historical messaging and 1:1 call data, propose a concrete plan to quantify tie strength and group affinity to identify who actually needs group calls. Specify at least five metrics (e.g., mutual message frequency, recency-weighted interaction score, common-neighbor count/triadic closure, thread-size distribution, co-call co-occurrence, clustering coefficient). Describe how you would segment users/threads and produce a ranked list of candidate groups to target at launch. 2) Instrumentation wishlist (recipient-focused): You cannot run a survey. Draft the exact event logs you want engineering to add to infer the recipient’s need for a group call. For each event (e.g., recipient sees incoming group call, attempts to add >1 participant from a 1:1 thread, attempts multi-dial, abandons add-participant flow, retries after failure), specify: event name, triggering UI action, required fields (user_id, thread_id, call_id, selected_participant_count, addressable_friend_count, prior_missed_calls_7d, network_type, device_os, error_code, latency_ms, local_time), and sampling/PII constraints. Explain how you would transform these logs into leading-indicator metrics of latent demand. 3) Choose a Meta proxy product: Pick one existing Meta product (e.g., Messenger, WhatsApp, Instagram, or Workplace) to use as a proxy for forecasting adoption and operational risks. Justify the choice with overlap of audience, call topology, encryption/infra constraints, and historical group-call usage. Define the exact proxy metrics you will read (activation, 1/7/28-day retention of group callers, average group size, call minutes/user, failure rate) and how you will translate them into targets for this product while adjusting for platform differences. 4) Set or reject a participant cap: Should the feature enforce a maximum group size at launch? Provide a data-backed decision framework using historical distribution of multi-party interactions, infra capacity, call quality trade-offs, and moderation/safety. Propose an initial cap and a playbook to ramp it (or remove it) using guardrail metrics (setup success, join success, end-to-end latency, crash rate, abuse rate) and stop conditions. 5) Experiment design with network effects and contamination: Design the primary experiment to measure incremental value. Specify: unit of randomization (user, thread, ego-network cluster, or community), how you will prevent/mitigate interference, and the exposure policy when a treated user calls a control friend (block vs. allow with shadow treatment vs. invite-only gate). Quantify expected contamination given average degree d and treatment share p; propose a method (e.g., graph clustering or household-level hashing) to keep cross-edge spillovers under X%. Include primary KPIs (e.g., incremental daily callers, minutes, sender/recipient satisfaction proxy), guardrails (delivery latency, crash, spam/abuse), power analysis inputs (MDE, variance, sample size, test duration), and a staged ramp plan. Finally, explain how you would measure and interpret direct vs. indirect network effects (e.g., k-factor, changes in clustering coefficient) within this experiment.

Quick Answer: This question evaluates product analytics and experimentation competencies, including tie-strength and group-affinity measurement, event-level instrumentation design, proxy-product forecasting, participant-cap constraints and operational trade-offs, and experiment design that accounts for network effects and contamination.

|Home/Analytics & Experimentation/Meta
Meta logo
Meta
Oct 13, 2025, 9:49 PM
hardData ScientistTechnical ScreenAnalytics & Experimentation
7
0

Evaluate and Launch Group Video Calls — Product Analytics Plan

Context: You are evaluating a new Group Video Call feature in a large-scale consumer messaging app at Meta. You must rely solely on behavioral/product analytics (no user surveys).

1) Quantify Relationships and Identify Who Needs Group Calls

Using historical messaging and 1:1 call data, propose a concrete plan to quantify tie strength and group affinity. Specify at least five metrics and explain how to segment users/threads and produce a ranked list of candidate groups to target at launch.

2) Instrumentation Wishlist (Recipient-Focused)

You cannot run a survey. Draft the exact event logs you want engineering to add to infer the recipient’s need for a group call. For each event, specify:

  • Event name
  • Triggering UI action
  • Required fields: user_id, thread_id, call_id, selected_participant_count, addressable_friend_count, prior_missed_calls_7d, network_type, device_os, error_code, latency_ms, local_time
  • Sampling/PII constraints Explain how you would transform these logs into leading-indicator metrics of latent demand.

Examples of events to consider: recipient sees incoming group call, attempts to add >1 participant from a 1:1 thread, attempts multi-dial, abandons add-participant flow, retries after failure.

3) Choose a Meta Proxy Product

Pick one existing Meta product (e.g., Messenger, WhatsApp, Instagram, or Workplace) as a proxy for forecasting adoption and operational risks. Justify the choice with overlap of audience, call topology, encryption/infra constraints, and historical group-call usage. Define the exact proxy metrics you will read (activation, 1/7/28-day retention of group callers, average group size, call minutes/user, failure rate) and how you will translate them into targets for this product while adjusting for platform differences.

4) Set or Reject a Participant Cap

Should the feature enforce a maximum group size at launch? Provide a data-backed decision framework using historical distribution of multi-party interactions, infra capacity, call quality trade-offs, and moderation/safety. Propose an initial cap and a playbook to ramp it (or remove it) using guardrail metrics (setup success, join success, end-to-end latency, crash rate, abuse rate) and stop conditions.

5) Experiment Design with Network Effects and Contamination

Design the primary experiment to measure incremental value. Specify the unit of randomization (user, thread, ego-network cluster, or community), how you will prevent/mitigate interference, and the exposure policy when a treated user calls a control friend (block vs. allow with shadow treatment vs. invite-only gate). Quantify expected contamination given average degree d and treatment share p; propose a method (e.g., graph clustering or household-level hashing) to keep cross-edge spillovers under X%. Include primary KPIs (e.g., incremental daily callers, minutes, sender/recipient satisfaction proxy), guardrails (delivery latency, crash, spam/abuse), power analysis inputs (MDE, variance, sample size, test duration), and a staged ramp plan. Explain how you would measure and interpret direct vs. indirect network effects (e.g., k-factor, changes in clustering coefficient) within this experiment.

Loading comments...