##### Scenario Product team changed one specific step in Confluent’s user-onboarding tutorial and wants to evaluate whether the change improves the experience. ##### Question Which primary and secondary metrics would you track that are highly specific to the modified tutorial step? 2. At which level would you randomize (user vs. account) and what covariates would you examine to verify comparable groups? 3. Which statistical test(s) would you use, how would you compute required sample size and expected runtime, and what alternative test would you prefer if the sample size turns out to be very small? ##### Hints Think micro-conversion rates, time-to-complete, event drop-offs; discuss unit-of-analysis alignment and balance checks; consider t/Z tests, nonparametrics or Bayesian for small samples.

Evaluate Metrics and Randomization for Onboarding Tutorial Change evaluates metric design, causal reasoning, experiment setup, diagnostics, SQL/statistical checks, and recommendations in a realistic interview setting. A strong answer states assumptions, handles edge cases, explains trade-offs, and shows how to validate the result clearly.

How do I approach Analytics & Experimentation interview questions?

Analytics & Experimentation questions require understanding of core concepts and practice. PracHub provides solutions with explanations to help you master analytics & experimentation interviews.

What difficulty level is this interview question?

This is a medium difficulty Analytics & Experimentation question, commonly asked during Onsite rounds at Confluent.

What role is this question designed for?

This question is commonly asked for Data Scientist candidates at Confluent during technical interviews.

Evaluate Metrics and Randomization for Onboarding Tutorial Change

Scenario

A single step within Confluent’s multi-step user-onboarding tutorial was modified. The product team wants to run an experiment to determine whether the change improves the user experience specifically at that step, while ensuring no negative side effects on the overall onboarding flow.

Assumptions for clarity:

The tutorial consists of ordered steps (1…k). Only step i was changed; all other steps remain unchanged.
We can instrument events at the step level: step_i_view, step_i_submit, step_i_success, step_i_error, help_click, backtrack, abandon, timestamps.
Users may belong to accounts (organizations) with multiple users.

Questions

Metrics

Which primary and secondary metrics would you track that are highly specific to the modified step?

Experiment design

At which level would you randomize (user vs. account), and what covariates would you examine to verify comparable groups?

Inference and sizing

Which statistical test(s) would you use? How would you compute required sample size and expected runtime? What alternative test would you prefer if the sample size turns out to be very small?

Hints

Think micro-conversion rates, time-to-complete, event drop-offs; discuss unit-of-analysis alignment and balance checks; consider t/Z tests, nonparametrics or Bayesian for small samples.

Constraints & Assumptions

Preserve the scope, facts, inputs, and requested outputs from the prompt above.
If the prompt leaves a detail unspecified, state a reasonable assumption before relying on it.
Keep the answer interview-ready: concise enough to present, but concrete enough to implement or evaluate.

Clarifying Questions to Ask

Clarify the business objective, unit of analysis, time window, exposure definition, and primary metric.
State assumptions about instrumentation, randomization, sample size, and data quality.
Separate descriptive analysis from causal claims.

What a Strong Answer Covers

A metric framework with primary, guardrail, and diagnostic metrics.
A credible analysis or experiment design with clear assumptions and bias checks.
SQL/statistical logic for segmentation, variance, confidence, and data validation where relevant.
An actionable recommendation that explains trade-offs and next steps.

Follow-up Questions

What sanity checks would you run before trusting the result?
How would you handle novelty effects, seasonality, or selection bias?
What decision would you make if metrics disagree?