Design and assess an A/B test

Q: Design and assess an A/B test

This is a Analytics & Experimentation interview question from Airbnb for Data Scientist roles. View the full question and solution on PracHub.

Q: How do I approach Analytics & Experimentation interview questions?

Analytics & Experimentation questions require understanding of core concepts and practice. PracHub provides solutions with explanations to help you master analytics & experimentation interviews.

Question

Experiment Design: New Onboarding Flow to Improve D7 Retention

You are testing a new onboarding flow for a consumer marketplace app available on iOS, Android, and Web. The new flow is shown only to new users. The goal is to increase Day-7 (D7) retention by 3 percentage points (absolute) relative to the current onboarding.

Design an A/B test and specify:

Primary and guardrail metrics
Unit of randomization and analysis
Sample size and duration assumptions (state your baseline and power assumptions; show how you compute the sample size)
Bucketing and exposure control (sticky assignment, pre-/post-login considerations, ramp plan)
Segmentation plan (which slices, why, and how to interpret)
Stopping rules and decision criteria
Novelty effects and seasonality considerations
Analysis plan (statistical tests, variance reduction if any, ITT vs. per-protocol)
How you would communicate the results and decision to stakeholders

Design and assess an A/B test

Experiment Design: New Onboarding Flow to Improve D7 Retention

Solution (Locked)

Comments (0)