Meta
Medium
Data ScientistDesign and validate ad model launch
Analytics & Experimentation
1
0
October 13, 2025
A/B Testing Interview Questions
A/B testing questions are central to data science and product analytics interviews at companies like Meta, Google, Netflix, and Airbnb.
Expect questions on experiment design, randomization units, sample size calculation, multiple comparisons, and metric selection.
Interviewers evaluate your statistical rigor, practical judgment, and ability to communicate experiment results.
837Questions
79Companies
Easy 99 • Medium 59 • Hard 12Difficulty mix
Common A/B testing interview patterns
- Designing an experiment for a product change
- Calculating sample size and experiment duration
- Choosing between one-sided and two-sided tests
- Handling multiple comparisons and peeking
- Interpreting results with novelty or primacy effects
- Network effects and interference between test groups
A/B testing interview questions
Filter and sort
Reddit
Hard
Data ScientistDesign a causal evaluation without A/B testing
Analytics & Experimentation
0
0
October 13, 2025
TikTok
Hard
Data ScientistDesign and evaluate a dasher bike rollout
Analytics & Experimentation
1
0
October 13, 2025
Amazon
Hard
Data ScientistEstimate live sports impact on subscriptions
Analytics & Experimentation
1
0
October 13, 2025
Thumbtack
Hard
Data ScientistDesign and evaluate an A/B test for launch
Analytics & Experimentation
1
0
October 13, 2025
Meta
Hard
Data ScientistDesign Messenger spam experiment with clustering
Analytics & Experimentation
1
0
October 13, 2025
Apple
Hard
Data ScientistEvaluate a model and choose metrics
Analytics & Experimentation
1
0
October 13, 2025
TikTok
Hard
Software EngineerDefine and measure project metrics
Analytics & Experimentation
1
0
September 6, 2025
Coinbase
Medium
Data ScientistEstimate QR Code Scan Rate for Super Bowl Ad
Analytics & Experimentation
1
0
August 4, 2025
PayPal
Hard
Data ScientistDesign A/B Test to Measure PayPal Cashback Value
Analytics & Experimentation
0
0
August 4, 2025
Meta
Medium
Data ScientistAnalyze Key Metrics for Notification System Success
Analytics & Experimentation
1
0
August 4, 2025
Roblox
Hard
Software EngineerDefine success metrics and monitoring
Analytics & Experimentation
1
0
July 31, 2025
Pinterest
Medium
Data ScientistEvaluate Carousel and Billboard Lift
Analytics & Experimentation
1
0
January 11, 2026
Uber
Easy
Data ScientistDesign an Uber feature and analyze safety
Analytics & Experimentation
2
0
January 8, 2026
Amazon
Medium
Machine Learning EngineerExplain why CTR rises but CVR unchanged
Analytics & Experimentation
1
0
January 6, 2026
Instacart
Easy
Data ScientistInvestigate marketplace metrics and experiment rollout
Analytics & Experimentation
1
0
December 18, 2025
PayPal
Easy
Data ScientistDesign metrics and an experiment for Eats donations
Analytics & Experimentation
0
0
November 20, 2025
Meta
Hard
Data ScientistHow would you evaluate upranking Shop ads?
Analytics & Experimentation
1
0
October 16, 2025
Capital One
Medium
Data ScientistAnalyze your favorite app and improve it
Analytics & Experimentation
1
0
October 13, 2025
Meta
Hard
Data ScientistDesign and critique teen-parent impact experiment
Analytics & Experimentation
1
0
October 13, 2025
Common mistakes in A/B testing interviews
- Not specifying the randomization unit (user vs session vs page)
- Peeking at results before reaching the required sample size
- Ignoring practical significance when statistical significance is achieved
- Not considering guardrail metrics
- Failing to account for novelty effects in short experiments
How A/B testing questions are evaluated
Structure your experiment design: hypothesis, metrics, unit, sample size, duration.
Discuss what could go wrong and how you would detect it.
Show ability to make a recommendation even when results are ambiguous.
Related analytics concepts
A/B Testing Interview FAQs
How do you determine the sample size for an A/B test?
Use a power analysis with inputs: baseline metric, minimum detectable effect (MDE), significance level (alpha, usually 0.05), and power (usually 0.80). Larger effects need fewer samples. For small MDE on rare events, you may need millions of users.
What is the difference between statistical and practical significance?
Statistical significance means the observed difference is unlikely due to chance (p-value < alpha). Practical significance means the effect is large enough to matter for the business. A statistically significant 0.01% lift may not be worth the engineering cost to ship.