Choose a precise A/B test primary metric
Overview
This question evaluates a data scientist's competency in experimental design, metric selection, and statistical reasoning for A/B testing, requiring precise definitions of a primary metric (numerator, denominator, unit of analysis), inclusion/exclusion and exposure rules, attribution windows, guardrails, and sample size/MDE considerations.
Google
Oct 13, 2025, 9:49 PM
Data Scientist
Technical Screen
Analytics & Experimentation
0
0
A/B Test: Choose One Primary Metric for a Home-Screen CTA Color Change
You are running an A/B test for an app that changes the color of its primary home-screen call-to-action (CTA). You must select exactly one primary metric and defend it. Provide the following:
-
Primary Metric Definition
- Exact definition: numerator, denominator, unit of analysis (user or session)
- Inclusion/exclusion rules (e.g., new vs. returning, employees/testers)
- Exposure definition (what counts as seeing the CTA)
- Attribution window and time horizon
-
Rationale
- Why this metric is the most sensitive and business-aligned for a visual color change
- Why alternatives (e.g., CTR variants, raw revenue) are inferior here
-
Guardrail Metrics
- List guardrails (e.g., bounce, crash rate, latency, retention)
- Explicit thresholds and actions if breached
-
Novelty and Contamination Mitigation (first 48 hours)
- How to mitigate novelty effects and returning-user contamination (e.g., holdbacks, cooldowns, analysis windows)
-
Sample Size/MDE for 48-Hour Decision
- State baseline funnel rates explicitly
- Provide the minimum sample size or minimum detectable effect (MDE) required to ship/no-ship within 48 hours
Solution
Login required
Show