Diagnose Discrepancy in A/B Test Conversion Rate Results

Q: Diagnose Discrepancy in A/B Test Conversion Rate Results

This question evaluates a candidate's skills in A/B test design, causal inference, power analysis, instrumentation validation, and diagnostic troubleshooting for conversion-lift experiments.

Q: How do I approach Analytics & Experimentation interview questions?

Analytics & Experimentation questions require understanding of core concepts and practice. PracHub provides solutions with explanations to help you master analytics & experimentation interviews.

Question

A/B Test Design: Personalized Marketing Emails and Conversion Lift

Scenario

An e-commerce firm wants to send personalized marketing emails to increase purchase conversions and evaluate the impact rigorously.

Tasks

Design an A/B test to measure whether personalized emails increase conversion rate. Specify:
- Unit of randomization and exposure rules
- Control vs. treatment definition
- Primary metric and guardrail metrics
- Statistical test and analysis approach
- Minimum Detectable Effect (MDE)
- Required sample size and expected test duration
After rolling out to the full user base, a new director reruns the test and observes only a 2% lift versus the original 20%. List plausible causes and the specific analyses you would run to diagnose the discrepancy.

Hints

Consider experiment design, power analysis, instrumentation issues, seasonality, user overlap/interference, novelty effects, and segmentation cuts.

Diagnose Discrepancy in A/B Test Conversion Rate Results

A/B Test Design: Personalized Marketing Emails and Conversion Lift

Scenario

Tasks

Hints

Solution

Comments (0)