PracHub
QuestionsPremiumCoachesLearningGuidesInterview Prep
|Home/Analytics & Experimentation/Google

Choose a precise A/B test primary metric

Last updated: Mar 29, 2026

Quick Overview

This question evaluates a data scientist's competency in experimental design, metric selection, and statistical reasoning for A/B testing, requiring precise definitions of a primary metric (numerator, denominator, unit of analysis), inclusion/exclusion and exposure rules, attribution windows, guardrails, and sample size/MDE considerations.

  • hard
  • Google
  • Analytics & Experimentation
  • Data Scientist

Choose a precise A/B test primary metric

Company: Google

Role: Data Scientist

Category: Analytics & Experimentation

Difficulty: hard

Interview Round: Technical Screen

An app changes the color of its primary home-screen CTA. You must pick exactly one primary metric for an A/B test and defend it—no hedging. Provide: 1) The metric’s exact definition: numerator, denominator, unit of analysis (user/session), inclusion/exclusion rules (new vs. returning, employees/testers), exposure definition, attribution window, and time horizon. 2) Why this metric is most sensitive and business-aligned for a visual color change, and why alternatives (e.g., CTR, raw revenue) are inferior here. 3) Guardrail metrics (e.g., bounce, crash rate, latency, retention) with explicit thresholds and what you’ll do if they breach. 4) How you will mitigate novelty effects and returning-user contamination within the first 48 hours (e.g., holdbacks, cooldowns, or analysis windows). 5) The minimum sample size or MDE you require to make a ship/no-ship decision within 48 hours, given baseline funnel rates you state explicitly.

Quick Answer: This question evaluates a data scientist's competency in experimental design, metric selection, and statistical reasoning for A/B testing, requiring precise definitions of a primary metric (numerator, denominator, unit of analysis), inclusion/exclusion and exposure rules, attribution windows, guardrails, and sample size/MDE considerations.

Related Interview Questions

  • Evaluate AI Workflow Product Metrics - Google (hard)
  • Design an A/B test for search ranking - Google (easy)
  • Design an Unbiased Upgrade Experiment - Google (hard)
  • Design a Causal Upgrade Experiment - Google (hard)
  • How would you use propensity score matching here - Google (medium)
Google logo
Google
Oct 13, 2025, 9:49 PM
Data Scientist
Technical Screen
Analytics & Experimentation
5
0

A/B Test: Choose One Primary Metric for a Home-Screen CTA Color Change

You are running an A/B test for an app that changes the color of its primary home-screen call-to-action (CTA). You must select exactly one primary metric and defend it. Provide the following:

  1. Primary Metric Definition
    • Exact definition: numerator, denominator, unit of analysis (user or session)
    • Inclusion/exclusion rules (e.g., new vs. returning, employees/testers)
    • Exposure definition (what counts as seeing the CTA)
    • Attribution window and time horizon
  2. Rationale
    • Why this metric is the most sensitive and business-aligned for a visual color change
    • Why alternatives (e.g., CTR variants, raw revenue) are inferior here
  3. Guardrail Metrics
    • List guardrails (e.g., bounce, crash rate, latency, retention)
    • Explicit thresholds and actions if breached
  4. Novelty and Contamination Mitigation (first 48 hours)
    • How to mitigate novelty effects and returning-user contamination (e.g., holdbacks, cooldowns, analysis windows)
  5. Sample Size/MDE for 48-Hour Decision
    • State baseline funnel rates explicitly
    • Provide the minimum sample size or minimum detectable effect (MDE) required to ship/no-ship within 48 hours

Solution

Show

Submit Your Answer to Earn 20XP

Sign in to leave a comment

Loading comments...

Browse More Questions

More Analytics & Experimentation•More Google•More Data Scientist•Google Data Scientist•Google Analytics & Experimentation•Data Scientist Analytics & Experimentation
PracHub

Master your tech interviews with 8,000+ real questions from top companies.

Product

  • Questions
  • Learning Tracks
  • Interview Guides
  • Resources
  • Premium
  • For Universities
  • Student Access

Browse

  • By Company
  • By Role
  • By Category
  • Topic Hubs
  • SQL Questions
  • Compare Platforms
  • Discord Community

Support

  • support@prachub.com
  • (916) 541-4762

Legal

  • Privacy Policy
  • Terms of Service
  • About Us

© 2026 PracHub. All rights reserved.