Analyze Retention Data for Geo-Targeted Feature Launch
Overview
This question evaluates a data scientist's competency in experimental design, causal inference, metric definition, statistical power/sample-size reasoning, and interpretation of retention curves in a geo-targeted, two-sided marketplace with limited traffic and operational implications.
Business Case: Geo-Targeted Feature with Limited Traffic and Retention Curves
Scenario
You are evaluating whether to launch a new geo-targeted product feature in a two-sided marketplace context (supply and demand). You have limited traffic in candidate test regions and a performance chart with retention curves for test vs. control cohorts over several weeks.
Task
Walk through how you would structure the business case and experiment plan:
- State the objective and key hypotheses for a geo-targeted feature.
- Specify the minimal data you need to make a decision.
- Define primary success metrics, secondary/diagnostic metrics, and guardrail metrics.
- Propose success criteria (thresholds, power/significance or Bayesian thresholds) appropriate for limited traffic.
- Interpret a retention chart showing test vs. control curves: draw insights or hypotheses for typical patterns (e.g., early lift that fades, late lift, crossing curves, no effect, negative effect).
- Recommend next steps: design choices (e.g., geo-experiment vs. user-level randomization), analysis methods suitable for limited traffic, follow-up tests, and operational checks.
Assume the feature selectively surfaces local content or benefits by neighborhood/zone and could influence user engagement, order frequency, and operational load in targeted geographies.
Solution
Login required