Determine Discount's Effect on Conversion Rate with A/B Testing
Overview
This question evaluates proficiency in A/B test design and causal inference, including hypothesis specification, primary and guardrail metric selection, randomization and exposure rules, sample size and power reasoning, experimental setup and quality checks, statistical analysis methods, and interpretation of conversion, revenue, and margin impacts within the Analytics & Experimentation domain. It is commonly asked to assess practical application combined with conceptual statistical understanding for designing robust experiments that balance statistical rigor, business constraints, data quality, and heterogeneity of treatment effects.
A/B Test Design: 10% Discount Impact on Conversion
Scenario
An e-commerce retailer wants to evaluate whether offering a 10% sitewide discount increases conversion rate and overall business outcomes.
Task
Design and analyze an A/B test to measure the discount’s impact. Provide:
- Hypotheses (null/alternative) and test direction.
-
Metrics:
- Primary outcome.
- Secondary and guardrail metrics (business and technical).
-
Randomization and exposure:
- Randomization unit and assignment.
- Eligibility/exposure rules to minimize dilution and contamination.
-
Sample size and power:
- Assumptions (baseline, MDE, alpha, power).
- Formula and a worked numeric example.
-
Experiment setup:
- Bucketing, ramp plan, duration, traffic splits.
- Concurrency controls, QA, and data quality checks (e.g., SRM).
-
Analysis plan:
- Estimation and significance testing.
- Variance reduction, clustering, sequential monitoring.
- Handling novelty effects and dilution (ITT vs. TOT).
-
Segmentation and heterogeneity:
- Which segments to pre-specify; multiplicity control.
-
Interpretation and decisioning:
- How to interpret results across conversion, revenue, and margin.
- Actions for positive/neutral/negative outcomes.
Hints to consider: randomization unit, power, guardrails, significance testing, segmentation, dilution, and novelty effects.
Solution
Login required