Scenario
In a QuickBooks-like onboarding flow, the product team adds a progress bar.
The onboarding has 4 steps:
-
Onboard
-
Send payment
-
Set up bank account
-
File the tax
Question
How would you evaluate whether the progress bar helps?
Your answer should include:
-
Hypothesis and what “helps” means
-
Primary metric + diagnostic metrics + guardrails (with tradeoffs)
-
Experiment design (randomization unit, sample size/power considerations at a high level, duration)
-
Key pitfalls (novelty effects, instrumentation, segment heterogeneity) and how you’d address them