Measure free-month promotion impact
Company: OpenAI
Role: Data Scientist
Category: Analytics & Experimentation
Difficulty: hard
Interview Round: Technical Screen
You are a data scientist evaluating a one-month free subscription promotion for new users. The business wants to know whether the promotion should be rolled out broadly.
Assume the company can run a randomized experiment on eligible users. Available user-level data include: `user_id`, `assignment_date`, `treatment_flag`, `signup_date`, `trial_start_date`, `first_paid_date`, monthly renewal and cancellation events, realized revenue, acquisition channel, country, and device. However, you do **not** have perfect long-term lifetime value data, full fraud labels, or complete marketing cost attribution, and the latest cohorts have only 60 days of follow-up.
Design an analysis plan to estimate the **causal impact** of the promotion and determine whether it creates positive business value. In your answer, address:
- the unit of randomization and the main estimand;
- primary, secondary, and guardrail metrics;
- why signup rate and retention rate alone are insufficient;
- how to define incremental revenue, cost, and ROI when there is no single universally correct ROI formula;
- how to handle limited follow-up, right-censoring, users who churn and later re-subscribe, and other edge cases;
- what assumptions you would make if key data are missing;
- how you would present the results and recommendation to stakeholders;
- what you would do differently if the promotion could not be randomized and you had to rely on observational data instead.
Quick Answer: This question evaluates causal inference and experimentation skills, including randomized design and estimand framing, metric and ROI attribution, handling censoring and incomplete data, and judgment under partial follow-up.