PracHub
QuestionsCoachesLearningGuidesInterview Prep
|Home/Analytics & Experimentation/Google

Design an A/B test with guardrails and SRM checks

Last updated: Mar 29, 2026

Quick Overview

This question evaluates competency in experimental design, metric engineering, statistical power and sample-size calculation, variance-reduction techniques, and monitoring and guardrail design for online A/B tests.

  • Medium
  • Google
  • Analytics & Experimentation
  • Data Scientist

Design an A/B test with guardrails and SRM checks

Company: Google

Role: Data Scientist

Category: Analytics & Experimentation

Difficulty: Medium

Interview Round: Technical Screen

You are launching a new personalized ranking on the product listing page. Define: (a) the primary success metric and its exact formula (include numerator/denominator and data filters); (b) at least three guardrail metrics with thresholds (e.g., latency p95, refund rate, bounce rate) and why each protects against specific failure modes. Then design the experiment: unit of randomization (and why), exposure rules, pre-exposure filtering, blocking/stratification, and handling repeat visitors across devices. Compute the required sample size to detect a 1.5% relative lift in the primary metric with 90% power and two-sided alpha=0.05, given a baseline mean of 3.2 and SD of 2.1 per user-day; show the formula you use. Specify your stopping rule (fixed horizon vs alpha-spending), how you'll apply CUPED or re-randomization to reduce variance, and the exact SRM test you’ll run each day. Describe how you will visualize results and diagnose issues (e.g., quantile treatment effects, funnel breakouts, time-since-exposure plots), and how you will interpret heterogeneous effects without p-hacking.

Quick Answer: This question evaluates competency in experimental design, metric engineering, statistical power and sample-size calculation, variance-reduction techniques, and monitoring and guardrail design for online A/B tests.

Related Interview Questions

  • Evaluate AI Workflow Product Metrics - Google (hard)
  • Design an A/B test for search ranking - Google (easy)
  • Design an Unbiased Upgrade Experiment - Google (hard)
  • Design a Causal Upgrade Experiment - Google (hard)
  • How do you diagnose a ratio metric change - Google (medium)
|Home/Analytics & Experimentation/Google

Design an A/B test with guardrails and SRM checks

Google logo
Google
Oct 13, 2025, 9:49 PM
MediumData ScientistTechnical ScreenAnalytics & Experimentation
5
0

You are launching a new personalized ranking on the product listing page. Define: (a) the primary success metric and its exact formula (include numerator/denominator and data filters); (b) at least three guardrail metrics with thresholds (e.g., latency p95, refund rate, bounce rate) and why each protects against specific failure modes. Then design the experiment: unit of randomization (and why), exposure rules, pre-exposure filtering, blocking/stratification, and handling repeat visitors across devices. Compute the required sample size to detect a 1.5% relative lift in the primary metric with 90% power and two-sided alpha=0.05, given a baseline mean of 3.2 and SD of 2.1 per user-day; show the formula you use. Specify your stopping rule (fixed horizon vs alpha-spending), how you'll apply CUPED or re-randomization to reduce variance, and the exact SRM test you’ll run each day. Describe how you will visualize results and diagnose issues (e.g., quantile treatment effects, funnel breakouts, time-since-exposure plots), and how you will interpret heterogeneous effects without p-hacking.

Loading comments...

Browse More Questions

More Analytics & Experimentation•More Google•More Data Scientist•Google Data Scientist•Google Analytics & Experimentation•Data Scientist Analytics & Experimentation

Write your answer

Your first approved answer each day earns 20 XP.

Sign in to write your answer.
PracHub

Master your tech interviews with 8,000+ real questions from top companies.

Product

  • Questions
  • Learning Tracks
  • Interview Guides
  • Resources
  • Premium
  • For Universities
  • Student Access

Browse

  • By Company
  • By Role
  • By Category
  • Topic Hubs
  • SQL Questions
  • AI Coding Questions
  • Compare Platforms
  • Discord Community

Support

  • support@prachub.com
  • (916) 541-4762

Legal

  • Privacy Policy
  • Terms of Service
  • About Us

© 2026 PracHub. All rights reserved.