Determine Discount's Effect on Conversion Rate with A/B Testing

Quick Overview

This interview question evaluates metric design, causal reasoning, experiment setup, diagnostics, SQL/statistical checks, and recommendations in a realistic interview setting. A strong answer for Determine Discount's Effect on Conversion Rate with A/B Testing states assumptions, handles edge cases, explains trade-offs, and shows how to validate the result clearly.

Determine Discount's Effect on Conversion Rate with A/B Testing

Company: Amazon

Role: Data Scientist

Category: Analytics & Experimentation

Difficulty: medium

Interview Round: Technical Screen

##### Scenario E-commerce retailer wants to understand if a 10 % discount increases conversion rate. ##### Question Design and analyze an A/B test to measure the discount’s impact: state hypotheses, choose metrics, estimate sample size, outline experiment setup, analysis, and how you would interpret results. ##### Hints Discuss randomization unit, power, guardrails, significance testing, segmentation, dilution and novelty effects.

Quick Answer: This interview question evaluates metric design, causal reasoning, experiment setup, diagnostics, SQL/statistical checks, and recommendations in a realistic interview setting. A strong answer for Determine Discount's Effect on Conversion Rate with A/B Testing states assumptions, handles edge cases, explains trade-offs, and shows how to validate the result clearly.

|Home/Analytics & Experimentation/Amazon
Amazon logo
Amazon
Aug 4, 2025, 10:55 AM
mediumData ScientistTechnical ScreenAnalytics & Experimentation
4
0

Determine Discount's Effect on Conversion Rate with A/B Testing

A/B Test Design: 10% Discount Impact on Conversion

Scenario

An e-commerce retailer wants to evaluate whether offering a 10% sitewide discount increases conversion rate and overall business outcomes.

Task

Design and analyze an A/B test to measure the discount’s impact. Provide:

  1. Hypotheses (null/alternative) and test direction.
  2. Metrics:
    • Primary outcome.
    • Secondary and guardrail metrics (business and technical).
  3. Randomization and exposure:
    • Randomization unit and assignment.
    • Eligibility/exposure rules to minimize dilution and contamination.
  4. Sample size and power:
    • Assumptions (baseline, MDE, alpha, power).
    • Formula and a worked numeric example.
  5. Experiment setup:
    • Bucketing, ramp plan, duration, traffic splits.
    • Concurrency controls, QA, and data quality checks (e.g., SRM).
  6. Analysis plan:
    • Estimation and significance testing.
    • Variance reduction, clustering, sequential monitoring.
    • Handling novelty effects and dilution (ITT vs. TOT).
  7. Segmentation and heterogeneity:
    • Which segments to pre-specify; multiplicity control.
  8. Interpretation and decisioning:
    • How to interpret results across conversion, revenue, and margin.
    • Actions for positive/neutral/negative outcomes.

Hints to consider: randomization unit, power, guardrails, significance testing, segmentation, dilution, and novelty effects.

Clarifying Questions to Ask Guidance

  • Clarify the business objective, unit of analysis, time window, exposure definition, and primary metric.
  • State assumptions about instrumentation, randomization, sample size, and data quality.
  • Separate descriptive analysis from causal claims.

What a Strong Answer Covers Guidance

  • A metric framework with primary, guardrail, and diagnostic metrics.
  • A credible analysis or experiment design with clear assumptions and bias checks.
  • SQL/statistical logic for segmentation, variance, confidence, and data validation where relevant.
  • An actionable recommendation that explains trade-offs and next steps.

Follow-up Questions Guidance

  • What sanity checks would you run before trusting the result?
  • How would you handle novelty effects, seasonality, or selection bias?
  • What decision would you make if metrics disagree?
Loading comments...

Browse More Questions

More Analytics & Experimentation•More Amazon•More Data Scientist•Amazon Data Scientist•Amazon Analytics & Experimentation•Data Scientist Analytics & Experimentation