Product team changed one specific step in Confluent’s user-onboarding tutorial and wants to evaluate whether the change improves the experience.

Which primary and secondary metrics would you track that are highly specific to the modified tutorial step? 2. At which level would you randomize (user vs. account) and what covariates would you examine to verify comparable groups? 3. Which statistical test(s) would you use, how would you compute required sample size and expected runtime, and what alternative test would you prefer if the sample size turns out to be very small?

Think micro-conversion rates, time-to-complete, event drop-offs; discuss unit-of-analysis alignment and balance checks; consider t/Z tests, nonparametrics or Bayesian for small samples.

Confluent Data Scientist interview question: Evaluate Metrics and Randomization for Onboarding Tutorial Change. {"blocks": [{"key": "8b0ea358", "text": "Scenario", "type": "header-two", "depth": 0, "inlineStyleRanges": [], "entityRanges": [], "data": {...