Airtable
Easy
Data ScientistDiagnose metric anomalies and evaluate new algorithm
Analytics & Experimentation
2
0
October 14, 2025
A/B Testing Interview Questions
A/B testing questions are central to data science and product analytics interviews at companies like Meta, Google, Netflix, and Airbnb.
Expect questions on experiment design, randomization units, sample size calculation, multiple comparisons, and metric selection.
Interviewers evaluate your statistical rigor, practical judgment, and ability to communicate experiment results.
837Questions
79Companies
Easy 99 • Medium 59 • Hard 12Difficulty mix
Common A/B testing interview patterns
- Designing an experiment for a product change
- Calculating sample size and experiment duration
- Choosing between one-sided and two-sided tests
- Handling multiple comparisons and peeking
- Interpreting results with novelty or primacy effects
- Network effects and interference between test groups
A/B testing interview questions
Filter and sort
Meta
Medium
Data ScientistDetect and address Simpson’s paradox
Analytics & Experimentation
3
0
October 13, 2025
Graham Capital
Medium
Software EngineerExplain project assumptions and validation methods
Analytics & Experimentation
1
0
September 6, 2025
PayPal
Medium
Data ScientistAnalyze Success Metrics and Diagnose Crypto Feature Issues
Analytics & Experimentation
2
0
August 4, 2025
Meta
Medium
Data ScientistDetermine Value of Prioritizing Accounts by Unread Notifications
Analytics & Experimentation
2
0
August 4, 2025
Meta
Medium
Data EngineerDesign visualizations for streaming metrics
Analytics & Experimentation
3
0
August 1, 2025
Meta
Medium
Data ScientistInterpreting metrics when autoplay videos reduce time‑spent but increase DAU
Analytics & Experimentation
11
0
July 12, 2025
Amazon
Medium
Data ScientistHow would you evaluate adding video ads?
Analytics & Experimentation
2
0
November 4, 2025
Airwallex
Medium
Data ScientistDiagnose a dip in approval/conversion rate
Analytics & Experimentation
3
0
October 14, 2025
Meta
Medium
Data ScientistDiagnose a sudden KPI drop
Analytics & Experimentation
2
0
October 13, 2025
Coinbase
Hard
Data ScientistDesign KYC experiment amid crypto volatility
Analytics & Experimentation
3
0
October 13, 2025
Meta
Medium
Data ScientistDetermine Impact of Re-share Button on User Engagement
Analytics & Experimentation
3
0
August 4, 2025
Uber
Medium
Software EngineerDefine and integrate room ranking factors
Analytics & Experimentation
1
0
August 1, 2025
Meta
Hard
Data EngineerDefine and validate product metrics
Analytics & Experimentation
2
0
July 15, 2025
Meta
Easy
Product AnalystHow would you grow key product metrics?
Analytics & Experimentation
2
0
February 2, 2026
DoorDash
Medium
Data ScientistAssess Adding Bicycle Dashers
Analytics & Experimentation
1
0
February 2, 2026
Google
Hard
Data ScientistMeasure outage impact; choose fix vs build
Analytics & Experimentation
3
0
October 13, 2025
TikTok
Hard
Data ScientistDiagnose a watch-time drop and design experiments
Analytics & Experimentation
3
0
October 13, 2025
Instacart
Medium
Data ScientistDiagnose and fix low conversion rigorously
Analytics & Experimentation
2
0
October 13, 2025
TikTok
Medium
Data ScientistTroubleshoot Sudden KPI Drop After Recent Product Release
Analytics & Experimentation
3
0
August 4, 2025
Common mistakes in A/B testing interviews
- Not specifying the randomization unit (user vs session vs page)
- Peeking at results before reaching the required sample size
- Ignoring practical significance when statistical significance is achieved
- Not considering guardrail metrics
- Failing to account for novelty effects in short experiments
How A/B testing questions are evaluated
Structure your experiment design: hypothesis, metrics, unit, sample size, duration.
Discuss what could go wrong and how you would detect it.
Show ability to make a recommendation even when results are ambiguous.
Related analytics concepts
A/B Testing Interview FAQs
How do you determine the sample size for an A/B test?
Use a power analysis with inputs: baseline metric, minimum detectable effect (MDE), significance level (alpha, usually 0.05), and power (usually 0.80). Larger effects need fewer samples. For small MDE on rare events, you may need millions of users.
What is the difference between statistical and practical significance?
Statistical significance means the observed difference is unlikely due to chance (p-value < alpha). Practical significance means the effect is large enough to matter for the business. A statistically significant 0.01% lift may not be worth the engineering cost to ship.