Choose a precise A/B test primary metric
Company: Google
Role: Data Scientist
Category: Analytics & Experimentation
Difficulty: hard
Interview Round: Technical Screen
An app changes the color of its primary home-screen CTA. You must pick exactly one primary metric for an A/B test and defend it—no hedging. Provide: 1) The metric’s exact definition: numerator, denominator, unit of analysis (user/session), inclusion/exclusion rules (new vs. returning, employees/testers), exposure definition, attribution window, and time horizon. 2) Why this metric is most sensitive and business-aligned for a visual color change, and why alternatives (e.g., CTR, raw revenue) are inferior here. 3) Guardrail metrics (e.g., bounce, crash rate, latency, retention) with explicit thresholds and what you’ll do if they breach. 4) How you will mitigate novelty effects and returning-user contamination within the first 48 hours (e.g., holdbacks, cooldowns, or analysis windows). 5) The minimum sample size or MDE you require to make a ship/no-ship decision within 48 hours, given baseline funnel rates you state explicitly.
Quick Answer: This question evaluates a data scientist's competency in experimental design, metric selection, and statistical reasoning for A/B testing, requiring precise definitions of a primary metric (numerator, denominator, unit of analysis), inclusion/exclusion and exposure rules, attribution windows, guardrails, and sample size/MDE considerations.