This interview question evaluates metric design, causal reasoning, experiment setup, diagnostics, SQL/statistical checks, and recommendations in a realistic interview setting. A strong answer for Diagnose Traffic Allocation in A/B Test Results states assumptions, handles edge cases, explains trade-offs, and shows how to validate the result clearly.
##### Scenario
Online experimentation and causal analysis for a consumer app.
##### Question
An A/B test changed a call-to-action button from green (control) to red (treatment) and retention dropped. Describe the diagnostics you would run to decide whether uneven traffic allocation or other experiment-quality issues drove the result. You are asked to measure the causal impact of receiving negative reviews on a merchant’s coupon repurchase rate. Outline the data you need and the methodology you would use to obtain an unbiased estimate.
##### Hints
Think about sample-ratio mismatch checks, covariate balance, time windows, difference-in-differences, matching/propensity scores, or holdout experiments.
Quick Answer: This interview question evaluates metric design, causal reasoning, experiment setup, diagnostics, SQL/statistical checks, and recommendations in a realistic interview setting. A strong answer for Diagnose Traffic Allocation in A/B Test Results states assumptions, handles edge cases, explains trade-offs, and shows how to validate the result clearly.
A consumer app ran an A/B test that changed a call-to-action (CTA) button from green (control) to red (treatment). Retention decreased in treatment.
You need to:
Diagnose whether uneven traffic allocation or experiment-quality issues could explain the observed drop in retention.
Separately, estimate the causal impact of receiving negative reviews on a merchant's coupon repurchase rate.
Assume retention is a k-day retention metric (e.g., 7-day retention), and the platform has standard experimentation infrastructure with event logs, feature flags, and user-level randomization. For the reviews question, assume we have time-stamped purchases and reviews at user–merchant level.
Part A — A/B Test Diagnostics for Retention Drop
Describe the diagnostics you would run to determine if uneven traffic allocation or other experiment-quality issues drove the result, and how you would decide whether to trust the result as causal.
Part B — Causal Impact of Negative Reviews on Coupon Repurchase
Outline the data you need and a methodology to obtain an unbiased estimate of the impact of receiving negative reviews on a merchant’s coupon repurchase rate.
Constraints & Assumptions
Preserve the scope, facts, inputs, and requested outputs from the prompt above.
If the prompt leaves a detail unspecified, state a reasonable assumption before relying on it.
Keep the answer interview-ready: concise enough to present, but concrete enough to implement or evaluate.
Clarifying Questions to Ask Guidance
Clarify the business objective, unit of analysis, time window, exposure definition, and primary metric.
State assumptions about instrumentation, randomization, sample size, and data quality.
Separate descriptive analysis from causal claims.
What a Strong Answer Covers Guidance
A metric framework with primary, guardrail, and diagnostic metrics.
A credible analysis or experiment design with clear assumptions and bias checks.
SQL/statistical logic for segmentation, variance, confidence, and data validation where relevant.
An actionable recommendation that explains trade-offs and next steps.
Follow-up Questions Guidance
What sanity checks would you run before trusting the result?
How would you handle novelty effects, seasonality, or selection bias?