Diagnose Traffic Allocation in A/B Test Results

Q: Diagnose Traffic Allocation in A/B Test Results

This question evaluates a data scientist's proficiency in experiment diagnostics and causal inference, testing understanding of experimentation infrastructure, retention metrics, data quality, randomization integrity and identification of causal effects in observational settings, and is commonly asked to assess the ability to distinguish true treatment impacts from allocation or instrumentation issues. It falls under Analytics & Experimentation and causal inference within data science, requiring both conceptual understanding of bias and identification and practical application of experiment-quality diagnostics and identification strategies using user-level, time-stamped event logs.

Q: How do I approach Analytics & Experimentation interview questions?

Analytics & Experimentation questions require understanding of core concepts and practice. PracHub provides solutions with explanations to help you master analytics & experimentation interviews.

Question

Scenario

A consumer app ran an A/B test that changed a call-to-action (CTA) button from green (control) to red (treatment). Retention decreased in treatment.

You need to:

Diagnose whether uneven traffic allocation or experiment-quality issues could explain the observed drop in retention.
Separately, estimate the causal impact of receiving negative reviews on a merchant's coupon repurchase rate.

Assume retention is a k-day retention metric (e.g., 7-day retention), and the platform has standard experimentation infrastructure with event logs, feature flags, and user-level randomization. For the reviews question, assume we have time-stamped purchases and reviews at user–merchant level.

Part A — A/B Test Diagnostics for Retention Drop

Describe the diagnostics you would run to determine if uneven traffic allocation or other experiment-quality issues drove the result, and how you would decide whether to trust the result as causal.

Part B — Causal Impact of Negative Reviews on Coupon Repurchase

Outline the data you need and a methodology to obtain an unbiased estimate of the impact of receiving negative reviews on a merchant’s coupon repurchase rate.

Diagnose Traffic Allocation in A/B Test Results

Scenario

Part A — A/B Test Diagnostics for Retention Drop

Part B — Causal Impact of Negative Reviews on Coupon Repurchase

Solution

Comments (0)