A/B Testing Interview Questions

Q: How do you determine the sample size for an A/B test?

Use a power analysis with inputs: baseline metric, minimum detectable effect (MDE), significance level (alpha, usually 0.05), and power (usually 0.80). Larger effects need fewer samples. For small MDE on rare events, you may need millions of users.

Q: What is the difference between statistical and practical significance?

Statistical significance means the observed difference is unlikely due to chance (p-value < alpha). Practical significance means the effect is large enough to matter for the business. A statistically significant 0.01% lift may not be worth the engineering cost to ship.

Question 1

A/B Testing Interview Questions

A/B Testing Interview Questions

Common A/B testing interview patterns

A/B testing interview questions

Present Piracy Trends to a PM

Investigate Falling Successful Orders in LA

Should the mulch promotion continue?

Design causal study for airport cancellation reduction

Design an A/B test with guardrails

Evaluate Facebook Dating launch and validate success

Choose cashback segment and model post-launch impact

Design metrics and geo A/B for new feature

Identify research to improve business

Investigate Declining ROI and Propose Effective Solutions

Investigate Causes of Decline in Facebook Group Comments

Investigate Falling Brand-Ad Spend

Design experiment for ads in chat with budgets

Measure impact of ads-manager automation feature

Evaluate Stripe Capital Lending Strategy

Assess Stripe Capital Strategy

Estimate shuttle impact with robust causal design

Validate friends vs unconnected; design rollout experiment

Diagnose a failing campaign

Define metrics and design experiments for notifications

Common mistakes in A/B testing interviews

How A/B testing questions are evaluated

Related analytics concepts

A/B Testing Interview FAQs

How do you determine the sample size for an A/B test?

What is the difference between statistical and practical significance?