Pinterest
Medium
Data ScientistEvaluate New Feed-Ranking Algorithm with A/B Testing
Analytics & Experimentation
76
0
July 12, 2025
A/B Testing Interview Questions
A/B testing questions are central to data science and product analytics interviews at companies like Meta, Google, Netflix, and Airbnb.
Expect questions on experiment design, randomization units, sample size calculation, multiple comparisons, and metric selection.
Interviewers evaluate your statistical rigor, practical judgment, and ability to communicate experiment results.
837Questions
79Companies
Easy 99 • Medium 59 • Hard 12Difficulty mix
Common A/B testing interview patterns
- Designing an experiment for a product change
- Calculating sample size and experiment duration
- Choosing between one-sided and two-sided tests
- Handling multiple comparisons and peeking
- Interpreting results with novelty or primacy effects
- Network effects and interference between test groups
A/B testing interview questions
Filter and sort
Meta
Hard
Data ScientistDetermine Group Call Feature Need and Evaluation Methods
Analytics & Experimentation
89
0
July 12, 2025
Meta
Hard
Data ScientistTrack Success and Guardrail Metrics for Push Notifications
Analytics & Experimentation
90
0
July 12, 2025
Confluent
Medium
Data ScientistEvaluate Metrics and Randomization for Onboarding Tutorial Change
Analytics & Experimentation
86
0
August 4, 2025
Meta
Medium
Data ScientistValidate Friend Content's Social Impact with Engagement Metrics
Analytics & Experimentation
84
0
July 12, 2025
TikTok
Medium
Data ScientistInvestigate Traffic Distribution Impact on Retention Decrease
Analytics & Experimentation
89
0
July 12, 2025
DoorDash
Medium
Data ScientistInvestigate Causes of Cold Food Deliveries and Solutions
Analytics & Experimentation
73
0
August 4, 2025
LinkedIn
Medium
Data ScientistMeasure Success of New B2B Product
Analytics & Experimentation
80
0
July 12, 2025
Google
Easy
Data ScientistDesign an A/B test for search ranking
Analytics & Experimentation
18
0
February 7, 2026
Uber
Hard
Data ScientistMeasure Impact of Updated Rider ETA Algorithm
Analytics & Experimentation
49
0
July 12, 2025
DoorDash
Easy
Data ScientistHow would you diagnose a completed orders drop?
Analytics & Experimentation
10
0
February 14, 2026
Uber
Hard
Data ScientistEstimate causal effect with interference
Analytics & Experimentation
19
0
October 13, 2025
Meta
Medium
Data ScientistExplain Algorithm's Disproportionate Impact on Demographic Segments
Analytics & Experimentation
73
0
August 4, 2025
Snapchat
Medium
Data ScientistDesign A/B Test for New Recommendation Algorithm Launch
Analytics & Experimentation
80
0
August 4, 2025
Uber
Easy
Data ScientistHow to experiment on ETA reduction
Analytics & Experimentation
12
0
February 6, 2026
Yelp
Medium
Data ScientistModify Instagram Feature: Track User Engagement Metric
Analytics & Experimentation
89
0
August 4, 2025
OpenAI
Medium
Data ScientistDetermine Metrics to Measure Free-Trial Impact on Subscriptions
Analytics & Experimentation
52
0
August 4, 2025
Attentive
Medium
Data ScientistApply Multiple Testing Corrections for Valid Results Analysis
Analytics & Experimentation
78
0
August 4, 2025
Meta
Medium
Data ScientistValidate Friends' Content Engagement with Experimental Design Metrics
Analytics & Experimentation
81
0
July 12, 2025
TikTok
Hard
Data ScientistEvaluate Home-Feed Diversity's Impact on User Engagement Metrics
Analytics & Experimentation
83
0
July 12, 2025
Common mistakes in A/B testing interviews
- Not specifying the randomization unit (user vs session vs page)
- Peeking at results before reaching the required sample size
- Ignoring practical significance when statistical significance is achieved
- Not considering guardrail metrics
- Failing to account for novelty effects in short experiments
How A/B testing questions are evaluated
Structure your experiment design: hypothesis, metrics, unit, sample size, duration.
Discuss what could go wrong and how you would detect it.
Show ability to make a recommendation even when results are ambiguous.
Related analytics concepts
A/B Testing Interview FAQs
How do you determine the sample size for an A/B test?
Use a power analysis with inputs: baseline metric, minimum detectable effect (MDE), significance level (alpha, usually 0.05), and power (usually 0.80). Larger effects need fewer samples. For small MDE on rare events, you may need millions of users.
What is the difference between statistical and practical significance?
Statistical significance means the observed difference is unlikely due to chance (p-value < alpha). Practical significance means the effect is large enough to matter for the business. A statistically significant 0.01% lift may not be worth the engineering cost to ship.