How do I approach Analytics & Experimentation interview questions?

Analytics & Experimentation questions require understanding of core concepts and practice. PracHub provides solutions with explanations to help you master analytics & experimentation interviews.

What difficulty level is this interview question?

This is a medium difficulty Analytics & Experimentation question, commonly asked during Technical Screen rounds at Amazon.

What role is this question designed for?

This question is commonly asked for Data Scientist candidates at Amazon during technical interviews.

Determine Discount's Effect on Conversion Rate with A/B Testing

Quick Overview

This question evaluates proficiency in A/B test design and causal inference, including hypothesis specification, primary and guardrail metric selection, randomization and exposure rules, sample size and power reasoning, experimental setup and quality checks, statistical analysis methods, and interpretation of conversion, revenue, and margin impacts within the Analytics & Experimentation domain. It is commonly asked to assess practical application combined with conceptual statistical understanding for designing robust experiments that balance statistical rigor, business constraints, data quality, and heterogeneity of treatment effects.

A/B Test Design: 10% Discount Impact on Conversion

Scenario

An e-commerce retailer wants to evaluate whether offering a 10% sitewide discount increases conversion rate and overall business outcomes.

Task

Design and analyze an A/B test to measure the discount’s impact. Provide:

Hypotheses (null/alternative) and test direction.
Metrics:
- Primary outcome.
- Secondary and guardrail metrics (business and technical).
Randomization and exposure:
- Randomization unit and assignment.
- Eligibility/exposure rules to minimize dilution and contamination.
Sample size and power:
- Assumptions (baseline, MDE, alpha, power).
- Formula and a worked numeric example.
Experiment setup:
- Bucketing, ramp plan, duration, traffic splits.
- Concurrency controls, QA, and data quality checks (e.g., SRM).
Analysis plan:
- Estimation and significance testing.
- Variance reduction, clustering, sequential monitoring.
- Handling novelty effects and dilution (ITT vs. TOT).
Segmentation and heterogeneity:
- Which segments to pre-specify; multiplicity control.
Interpretation and decisioning:
- How to interpret results across conversion, revenue, and margin.
- Actions for positive/neutral/negative outcomes.

Hints to consider: randomization unit, power, guardrails, significance testing, segmentation, dilution, and novelty effects.

Quick Overview

Task

Design and analyze an A/B test to measure the discount’s impact. Provide:

Hypotheses (null/alternative) and test direction.

Metrics:

Primary outcome.
Secondary and guardrail metrics (business and technical).

Randomization and exposure:

Randomization unit and assignment.
Eligibility/exposure rules to minimize dilution and contamination.

Sample size and power:

Assumptions (baseline, MDE, alpha, power).
Formula and a worked numeric example.

Experiment setup:

Bucketing, ramp plan, duration, traffic splits.
Concurrency controls, QA, and data quality checks (e.g., SRM).

Analysis plan:

Estimation and significance testing.
Variance reduction, clustering, sequential monitoring.
Handling novelty effects and dilution (ITT vs. TOT).

Segmentation and heterogeneity:

Which segments to pre-specify; multiplicity control.

Interpretation and decisioning:

How to interpret results across conversion, revenue, and margin.
Actions for positive/neutral/negative outcomes.

Hints to consider: randomization unit, power, guardrails, significance testing, segmentation, dilution, and novelty effects.

Determine Discount's Effect on Conversion Rate with A/B Testing

Quick Overview

A/B Test Design: 10% Discount Impact on Conversion

Scenario

Task

Solution

Submit Your Answer to Earn 20XP

Determine Discount's Effect on Conversion Rate with A/B Testing

Quick Overview

A/B Test Design: 10% Discount Impact on Conversion

Scenario

Task

Solution

Submit Your Answer to Earn 20XP