Assess free-month promotion impact
Quick Overview
This question evaluates a data scientist's competency in experimental design, causal inference, metric definition and prioritization, bias identification, incremental impact estimation, and business ROI assessment for promotional offers.
You are evaluating a free one-month promotion for a subscription product. Eligible users can either see the normal paid signup flow or receive the first month free. The business wants to know whether the promotion should be rolled out more broadly.
Assume you have user-level data with: experiment assignment or targeting flags, signup date, activation events, acquisition channel, geo, device, historical engagement, subscription start/end dates, payments, refunds, and a limited follow-up window. Some users may have been targeted non-randomly, and long-term lifetime value is only partially observed.
Discuss how you would:
- Define the primary success metric, secondary metrics, and guardrails. Consider tradeoffs among signup conversion, activation, 30/60/90-day retention, paid conversion after the free month, gross revenue, net revenue, contribution margin, and ROI.
- Decide whether this should be analyzed as a randomized experiment or as an observational causal inference problem.
- Identify the major pitfalls and sources of bias, including selection effects, pull-forward or cannibalization, delayed outcomes, survivorship bias, seasonality, and users who churn and later resubscribe.
- Estimate the incremental impact and ROI when not all desired data are available, stating any assumptions needed to project long-term value.
- Summarize how you would present a recommendation to non-technical stakeholders if some metrics improve while others worsen.