Design an A/B test with guardrails and SRM checks

Q: Design an A/B test with guardrails and SRM checks

This is a Analytics & Experimentation interview question from Google for Data Scientist roles. View the full question and solution on PracHub.

Q: How do I approach Analytics & Experimentation interview questions?

Analytics & Experimentation questions require understanding of core concepts and practice. PracHub provides solutions with explanations to help you master analytics & experimentation interviews.

Question

You are launching a new personalized ranking on the product listing page. Define: (a) the primary success metric and its exact formula (include numerator/denominator and data filters); (b) at least three guardrail metrics with thresholds (e.g., latency p95, refund rate, bounce rate) and why each protects against specific failure modes. Then design the experiment: unit of randomization (and why), exposure rules, pre-exposure filtering, blocking/stratification, and handling repeat visitors across devices. Compute the required sample size to detect a 1.5% relative lift in the primary metric with 90% power and two-sided alpha=0.05, given a baseline mean of 3.2 and SD of 2.1 per user-day; show the formula you use. Specify your stopping rule (fixed horizon vs alpha-spending), how you'll apply CUPED or re-randomization to reduce variance, and the exact SRM test you’ll run each day. Describe how you will visualize results and diagnose issues (e.g., quantile treatment effects, funnel breakouts, time-since-exposure plots), and how you will interpret heterogeneous effects without p-hacking.

Design an A/B test with guardrails and SRM checks

Comments (0)