A/B Testing Interview Questions

Q: How do you determine the sample size for an A/B test?

Use a power analysis with inputs: baseline metric, minimum detectable effect (MDE), significance level (alpha, usually 0.05), and power (usually 0.80). Larger effects need fewer samples. For small MDE on rare events, you may need millions of users.

Q: What is the difference between statistical and practical significance?

Statistical significance means the observed difference is unlikely due to chance (p-value < alpha). Practical significance means the effect is large enough to matter for the business. A statistically significant 0.01% lift may not be worth the engineering cost to ship.

A/B Testing Interview Questions

A/B Testing Interview Questions

Common A/B testing interview patterns

A/B testing interview questions

Choose alternatives when randomization fails

Evaluate brand ads effectiveness on social media causally

Design an experiment for spam filtering impact

Launch and measure a TV campaign

Design experiment on culture memo emphasis

Diagnose rising delivery cost precisely

Assess launching a vegan burger

Prove friends outperform unconnected; design experiments and metrics

Determine A/B test sample size drivers

Diagnose Decline in Successful Orders

Design experiment for fake accounts impact

Choose KPIs and prove impact with experiments

Calculate Break-Even Point and Profit Impact Analysis

Define Success with Contact Syncing for Growth and Evaluation

Determine Channel Performance with Additional Metrics Needed

Design A/B Test for Short-Video Recommendation Algorithm

Diagnose drop in shopper accepted orders

Compute DiD and validate parallel trends

Design video-ads experiment and handle null results

Design and analyze an A/B test

Common mistakes in A/B testing interviews

How A/B testing questions are evaluated

Related analytics concepts

A/B Testing Interview FAQs

How do you determine the sample size for an A/B test?

What is the difference between statistical and practical significance?