Evaluate Metrics and Randomization for Onboarding Tutorial Change
Overview
This question evaluates a data scientist's skills in product experimentation, including selection of step-specific metrics (micro-conversions and time-to-complete), decisions about randomization and unit-of-analysis, and statistical inference with sample-size considerations.
Scenario
A single step within Confluent’s multi-step user-onboarding tutorial was modified. The product team wants to run an experiment to determine whether the change improves the user experience specifically at that step, while ensuring no negative side effects on the overall onboarding flow.
Assumptions for clarity:
- The tutorial consists of ordered steps (1…k). Only step i was changed; all other steps remain unchanged.
- We can instrument events at the step level: step_i_view, step_i_submit, step_i_success, step_i_error, help_click, backtrack, abandon, timestamps.
- Users may belong to accounts (organizations) with multiple users.
Questions
- Metrics
- Which primary and secondary metrics would you track that are highly specific to the modified step?
- Experiment design
- At which level would you randomize (user vs. account), and what covariates would you examine to verify comparable groups?
- Inference and sizing
- Which statistical test(s) would you use? How would you compute required sample size and expected runtime? What alternative test would you prefer if the sample size turns out to be very small?
Hints
Think micro-conversion rates, time-to-complete, event drop-offs; discuss unit-of-analysis alignment and balance checks; consider t/Z tests, nonparametrics or Bayesian for small samples.
Solution
Login required