Evaluates A/B test design for a redesigned onboarding flow in a subscription app. Strong answers define activation, user-level randomization, primary and guardrail metrics, sample size, sequential monitoring, and decisions when activation improves but support burden rises.
##### Scenario
Product team plans to launch a redesigned onboarding flow and needs evidence it increases activation.
##### Question
Design an A/B test for the new onboarding. State hypothesis, unit of randomization, key metrics, guardrail metrics, and runtime calculation. If early results show uplift but increased support tickets, how would you decide whether to launch?
##### Hints
Address sample size, power, sequential checks, and balancing primary vs. secondary metrics.
Quick Answer: Evaluates A/B test design for a redesigned onboarding flow in a subscription app. Strong answers define activation, user-level randomization, primary and guardrail metrics, sample size, sequential monitoring, and decisions when activation improves but support burden rises.
A consumer subscription app is launching a redesigned onboarding flow for newly registered users. The goal is to increase activation, defined for this prompt as starting to play any title within 7 days of signup unless your organization uses a different definition.
Constraints & Assumptions
Users should be randomized at signup and assigned persistently.
Define exposure, eligibility, and activation precisely.
Include primary metrics, guardrails, power, runtime, monitoring, and decision rules.
Handle early mixed results carefully.
Clarifying Questions to Ask Guidance
What is the current baseline activation rate?
What onboarding steps changed?
Are users exposed across multiple devices?
What support, retention, or subscription outcomes could be affected?
Part 1 - Experiment Setup
State the hypothesis, unit of randomization, and treatment/control definitions.
What This Part Should Cover Guidance
New-user eligibility, sticky user-level randomization, exposure definition, treatment and control flows, and hypothesis.
Part 2 - Metrics
What primary, secondary, and guardrail metrics would you use?
What This Part Should Cover Guidance
Activation within 7 days, onboarding completion, first play, subscription conversion, retention, and engagement.
Guardrails such as support tickets, cancellations, latency, crashes, user complaints, and long-term retention.
Part 3 - Power and Monitoring
How would you calculate sample size/runtime and monitor the test?
A strong answer defines the activation metric and exposure cleanly, designs a powered user-level experiment, monitors guardrails, and makes decisions based on both user value and operational risk.
Follow-up Questions Guidance
What if activation rises but 30-day retention falls?
How would you avoid peeking bias?
How would you analyze treatment effects by user segment?