Design and assess an A/B test

Q: Design and assess an A/B test

This question evaluates a data scientist's competency in experimental design, statistical power and sample-size estimation, metric definition and guardrails, unit-of-randomization and exposure control, segmentation and analysis planning for product experiments.

Q: How do I approach Analytics & Experimentation interview questions?

Analytics & Experimentation questions require understanding of core concepts and practice. PracHub provides solutions with explanations to help you master analytics & experimentation interviews.

Question

Loading...

Experiment Design: New Onboarding Flow to Improve D7 Retention

You are testing a new onboarding flow for a consumer marketplace app available on iOS, Android, and Web. The new flow is shown only to new users. The goal is to increase Day-7 (D7) retention by 3 percentage points (absolute) relative to the current onboarding.

Design an A/B test and specify:

Primary and guardrail metrics
Unit of randomization and analysis
Sample size and duration assumptions (state your baseline and power assumptions; show how you compute the sample size)
Bucketing and exposure control (sticky assignment, pre-/post-login considerations, ramp plan)
Segmentation plan (which slices, why, and how to interpret)
Stopping rules and decision criteria
Novelty effects and seasonality considerations
Analysis plan (statistical tests, variance reduction if any, ITT vs. per-protocol)
How you would communicate the results and decision to stakeholders

Design and assess an A/B test

Experiment Design: New Onboarding Flow to Improve D7 Retention

Solution

Comments (0)

Design and assess an A/B test

Overview

Experiment Design: New Onboarding Flow to Improve D7 Retention

Solution

Comments (0)