Design an A/B test with guardrails and SRM checks

Quick Overview

This question evaluates competency in experimental design, metric engineering, statistical power and sample-size calculation, variance-reduction techniques, and monitoring and guardrail design for online A/B tests.

Design an A/B test with guardrails and SRM checks

Company: Google

Role: Data Scientist

Category: Analytics & Experimentation

Difficulty: medium

Interview Round: Technical Screen

You are launching a new personalized ranking on the product listing page. Define: (a) the primary success metric and its exact formula (include numerator/denominator and data filters); (b) at least three guardrail metrics with thresholds (e.g., latency p95, refund rate, bounce rate) and why each protects against specific failure modes. Then design the experiment: unit of randomization (and why), exposure rules, pre-exposure filtering, blocking/stratification, and handling repeat visitors across devices. Compute the required sample size to detect a 1.5% relative lift in the primary metric with 90% power and two-sided alpha=0.05, given a baseline mean of 3.2 and SD of 2.1 per user-day; show the formula you use. Specify your stopping rule (fixed horizon vs alpha-spending), how you'll apply CUPED or re-randomization to reduce variance, and the exact SRM test you’ll run each day. Describe how you will visualize results and diagnose issues (e.g., quantile treatment effects, funnel breakouts, time-since-exposure plots), and how you will interpret heterogeneous effects without p-hacking.

Quick Answer: This question evaluates competency in experimental design, metric engineering, statistical power and sample-size calculation, variance-reduction techniques, and monitoring and guardrail design for online A/B tests.

|Home/Analytics & Experimentation/Google
Google logo
Google
Oct 13, 2025, 9:49 PM
mediumData ScientistTechnical ScreenAnalytics & Experimentation
9
0

You are launching a new personalized ranking on the product listing page. Define: (a) the primary success metric and its exact formula (include numerator/denominator and data filters); (b) at least three guardrail metrics with thresholds (e.g., latency p95, refund rate, bounce rate) and why each protects against specific failure modes. Then design the experiment: unit of randomization (and why), exposure rules, pre-exposure filtering, blocking/stratification, and handling repeat visitors across devices. Compute the required sample size to detect a 1.5% relative lift in the primary metric with 90% power and two-sided alpha=0.05, given a baseline mean of 3.2 and SD of 2.1 per user-day; show the formula you use. Specify your stopping rule (fixed horizon vs alpha-spending), how you'll apply CUPED or re-randomization to reduce variance, and the exact SRM test you’ll run each day. Describe how you will visualize results and diagnose issues (e.g., quantile treatment effects, funnel breakouts, time-since-exposure plots), and how you will interpret heterogeneous effects without p-hacking.

Loading comments...