##### Scenario Product team plans to launch a redesigned onboarding flow and needs evidence it increases activation. ##### Question Design an A/B test for the new onboarding. State hypothesis, unit of randomization, key metrics, guardrail metrics, and runtime calculation. If early results show uplift but increased support tickets, how would you decide whether to launch? ##### Hints Address sample size, power, sequential checks, and balancing primary vs. secondary metrics.

Evaluates A/B test design for a redesigned onboarding flow in a subscription app. Strong answers define activation, user-level randomization, primary and guardrail metrics, sample size, sequential monitoring, and decisions when activation improves but support burden rises.

How do I approach Analytics & Experimentation interview questions?

Analytics & Experimentation questions require understanding of core concepts and practice. PracHub provides solutions with explanations to help you master analytics & experimentation interviews.

What difficulty level is this interview question?

This is a medium difficulty Analytics & Experimentation question, commonly asked during Onsite rounds at Netflix.

What role is this question designed for?

This question is commonly asked for Data Scientist candidates at Netflix during technical interviews.

How to Design Effective A/B Tests for Onboarding

Design Effective A/B Tests for Onboarding

A consumer subscription app is launching a redesigned onboarding flow for newly registered users. The goal is to increase activation, defined for this prompt as starting to play any title within 7 days of signup unless your organization uses a different definition.

Constraints & Assumptions

Users should be randomized at signup and assigned persistently.
Define exposure, eligibility, and activation precisely.
Include primary metrics, guardrails, power, runtime, monitoring, and decision rules.
Handle early mixed results carefully.

Clarifying Questions to Ask

What is the current baseline activation rate?
What onboarding steps changed?
Are users exposed across multiple devices?
What support, retention, or subscription outcomes could be affected?

Part 1 - Experiment Setup

State the hypothesis, unit of randomization, and treatment/control definitions.

What This Part Should Cover

New-user eligibility, sticky user-level randomization, exposure definition, treatment and control flows, and hypothesis.

Part 2 - Metrics

What primary, secondary, and guardrail metrics would you use?

What This Part Should Cover

Activation within 7 days, onboarding completion, first play, subscription conversion, retention, and engagement.
Guardrails such as support tickets, cancellations, latency, crashes, user complaints, and long-term retention.

Part 3 - Power and Monitoring

How would you calculate sample size/runtime and monitor the test?

What This Part Should Cover

Baseline rate, MDE, alpha, power, traffic volume, experiment duration, sequential monitoring rules, SRM, and instrumentation checks.

Part 4 - Decision Framework

What would you do if early results show activation uplift but support tickets increase?

What This Part Should Cover

Predefined guardrail thresholds, severity analysis, segment diagnostics, practical significance, iteration, holdout, or ramp decision.

What a Strong Answer Covers

A strong answer defines the activation metric and exposure cleanly, designs a powered user-level experiment, monitors guardrails, and makes decisions based on both user value and operational risk.

Follow-up Questions

What if activation rises but 30-day retention falls?
How would you avoid peeking bias?
How would you analyze treatment effects by user segment?

Design Effective A/B Tests for Onboarding

Constraints & Assumptions

Users should be randomized at signup and assigned persistently.
Define exposure, eligibility, and activation precisely.
Include primary metrics, guardrails, power, runtime, monitoring, and decision rules.
Handle early mixed results carefully.

Clarifying Questions to Ask

What is the current baseline activation rate?
What onboarding steps changed?
Are users exposed across multiple devices?
What support, retention, or subscription outcomes could be affected?

Part 1 - Experiment Setup

State the hypothesis, unit of randomization, and treatment/control definitions.

What This Part Should Cover

New-user eligibility, sticky user-level randomization, exposure definition, treatment and control flows, and hypothesis.

Part 2 - Metrics

What primary, secondary, and guardrail metrics would you use?

What This Part Should Cover

Activation within 7 days, onboarding completion, first play, subscription conversion, retention, and engagement.
Guardrails such as support tickets, cancellations, latency, crashes, user complaints, and long-term retention.

Part 3 - Power and Monitoring

How would you calculate sample size/runtime and monitor the test?

What This Part Should Cover

Baseline rate, MDE, alpha, power, traffic volume, experiment duration, sequential monitoring rules, SRM, and instrumentation checks.

Part 4 - Decision Framework

What would you do if early results show activation uplift but support tickets increase?

What This Part Should Cover

Predefined guardrail thresholds, severity analysis, segment diagnostics, practical significance, iteration, holdout, or ramp decision.

What a Strong Answer Covers

A strong answer defines the activation metric and exposure cleanly, designs a powered user-level experiment, monitors guardrails, and makes decisions based on both user value and operational risk.

Follow-up Questions

What if activation rises but 30-day retention falls?
How would you avoid peeking bias?
How would you analyze treatment effects by user segment?

How to Design Effective A/B Tests for Onboarding

Quick Overview

How to Design Effective A/B Tests for Onboarding

Design Effective A/B Tests for Onboarding

Constraints & Assumptions

Clarifying Questions to Ask

Part 1 - Experiment Setup

What This Part Should Cover

Part 2 - Metrics

What This Part Should Cover

Part 3 - Power and Monitoring

What This Part Should Cover

Part 4 - Decision Framework

What This Part Should Cover

What a Strong Answer Covers

Follow-up Questions

Write your answer

How to Design Effective A/B Tests for Onboarding

Quick Overview

How to Design Effective A/B Tests for Onboarding

Design Effective A/B Tests for Onboarding

Constraints & Assumptions

Clarifying Questions to Ask

Part 1 - Experiment Setup

What This Part Should Cover

Part 2 - Metrics

What This Part Should Cover

Part 3 - Power and Monitoring

What This Part Should Cover

Part 4 - Decision Framework

What This Part Should Cover

What a Strong Answer Covers

Follow-up Questions

Write your answer