Choose group-call size cap via experiment
Company: Meta
Role: Data Scientist
Category: Analytics & Experimentation
Difficulty: hard
Interview Round: Technical Screen
You must decide the maximum participants per Group Call. Propose an experiment to select the cap among {4, 8, 16}. Cover: (a) Design choice (fixed multi-arm vs bandit) under a 4-week deadline; justify exploration vs exploitation trade-offs. (b) Primary metric (e.g., successful starts per eligible user) and reliability/latency guardrails (start success rate, join latency p95/p99, crash rate). (c) Stratified randomization to balance device class, network type, and region; how to handle cluster/household effects. (d) Detect non-monotonic effects (8 may outperform 4 and 16) and heterogeneity by segment (friends vs workgroups); specify an analysis that controls false discovery (e.g., hierarchical modeling or Holm-Bonferroni). (e) Stopping rules, interim looks, and rollback thresholds if tail latency SLOs are breached; include how you would cap exposure to protect reliability while still making a decision.
Overview: This question evaluates a candidate's competency in experimental design, A/B testing and statistical analysis for product decisions, specifically testing metric selection, reliability and latency guardrails, stratified randomization, interference and clustering effects, heterogeneity detection, and operational stopping rules.