Design success and guardrail metrics
Overview
This question evaluates a data scientist's ability to design rigorous metric frameworks and randomized experiments, covering competencies such as defining primary success and guardrail metrics with attribution rules, randomization and bucketing strategies, SRM and event-loss detection, handling novelty and seasonality, segmentation and multiple-comparison control, and decision rubrics. It is asked in Analytics & Experimentation interviews to assess rigor in causal measurement, trade-off reasoning, and product-metrics literacy, testing both conceptual understanding and practical application of statistical and experimentation principles.
You are launching a new recommendation module on a content platform. Design a metrics framework and an experiment plan. A) Define one primary success metric with an exact formula (include unit of randomization, numerator, denominator, inclusion/exclusion criteria, and a 28-day attribution rule). Define at least two guardrail metrics with clear thresholds and directions (e.g., bounce rate, crash rate, page latency) and explain trade-offs. B) Specify the randomization unit and bucketing strategy to avoid contamination across surfaces and sessions. Describe how you would detect and triage Sample Ratio Mismatch (SRM) and event-loss issues (e.g., via canary dashboards, heartbeat events), including a concrete SRM test and an acceptable p-value threshold. C) Explain how you will handle novelty effects and seasonality (e.g., ramp schedule, pre-period covariates/CUPED, calendar alignment) and propose a minimal test duration rule. Include a plan for segmentation and multiple-comparison control across 5 planned cuts. D) Outline a decision rubric that ties metric movement to ship/hold/iterate decisions, including what to do if the primary metric is flat but a guardrail regresses.