Define Churn and Design Onboarding Experiment
Company: Gusto
Role: Data Scientist
Category: Analytics & Experimentation
Difficulty: medium
Interview Round: Onsite
Imagine you are the product data scientist for a consumer app. The team is evaluating a redesigned onboarding flow, and some variants may include an email that encourages new users to complete onboarding.
Answer the following:
1. Define churn rate for this product. Specify the denominator, churn event, time window, observation window, timezone, and how to handle reactivation.
2. Explain how you would use historical data to choose the churn event and time window rather than picking them arbitrarily.
3. Design an A/B test for the onboarding change. Specify the randomization unit, treatment and control setup, especially if email is part of the treatment, primary success metric, secondary metrics, guardrail metrics, and how to interpret short-term versus long-term effects.
Quick Answer: This question evaluates competency in product analytics and experimentation, including defining churn metrics (denominator, event and observation windows, reactivation rules), leveraging historical data to select meaningful churn definitions, and designing randomized A/B tests with clear treatment/control assignments and metric hierarchies.