How do I approach Analytics & Experimentation interview questions?

Analytics & Experimentation questions require understanding of core concepts and practice. PracHub provides solutions with explanations to help you master analytics & experimentation interviews.

What difficulty level is this interview question?

This is a hard difficulty Analytics & Experimentation question, commonly asked during Onsite rounds at Airbnb.

What role is this question designed for?

This question is commonly asked for Data Scientist candidates at Airbnb during technical interviews.

Design and assess an A/B test | Airbnb Interview Question

Quick Overview

This question evaluates a data scientist's competency in experimental design, statistical power and sample-size estimation, metric definition and guardrails, unit-of-randomization and exposure control, segmentation and analysis planning for product experiments.

Experiment Design: New Onboarding Flow to Improve D7 Retention

You are testing a new onboarding flow for a consumer marketplace app available on iOS, Android, and Web. The new flow is shown only to new users. The goal is to increase Day-7 (D7) retention by 3 percentage points (absolute) relative to the current onboarding.

Design an A/B test and specify:

Primary and guardrail metrics
Unit of randomization and analysis
Sample size and duration assumptions (state your baseline and power assumptions; show how you compute the sample size)
Bucketing and exposure control (sticky assignment, pre-/post-login considerations, ramp plan)
Segmentation plan (which slices, why, and how to interpret)
Stopping rules and decision criteria
Novelty effects and seasonality considerations
Analysis plan (statistical tests, variance reduction if any, ITT vs. per-protocol)
How you would communicate the results and decision to stakeholders

Experiment Design: New Onboarding Flow to Improve D7 Retention

Design an A/B test and specify:

Primary and guardrail metrics
Unit of randomization and analysis
Sample size and duration assumptions (state your baseline and power assumptions; show how you compute the sample size)
Bucketing and exposure control (sticky assignment, pre-/post-login considerations, ramp plan)
Segmentation plan (which slices, why, and how to interpret)
Stopping rules and decision criteria
Novelty effects and seasonality considerations
Analysis plan (statistical tests, variance reduction if any, ITT vs. per-protocol)
How you would communicate the results and decision to stakeholders

Design and assess an A/B test

Quick Overview

Experiment Design: New Onboarding Flow to Improve D7 Retention

Solution

Comments (0)

Design and assess an A/B test

Quick Overview

Experiment Design: New Onboarding Flow to Improve D7 Retention

Solution

Comments (0)