Evaluate a free-trial A/B test
Company: OpenAI
Role: Data Scientist
Category: Analytics & Experimentation
Difficulty: easy
Interview Round: Technical Screen
## Scenario
A marketing team ran an A/B test offering **a free 1-month trial** to users.
- **Control (A):** Standard offer (no free month)
- **Treatment (B):** Free 1-month trial offer
- Randomization unit: **user** (assume one assignment per user)
The team cares about:
1) **Signup rate** (do more users start a subscription/trial?)
2) **Retention** (do users stick around after the trial / over time?)
## Tasks
1. **Experiment design & setup**
- State the hypothesis and the key product/business risk.
- Specify the **primary metric**, **diagnostic metrics**, and **guardrail metrics**. Discuss tradeoffs (e.g., signup lift vs low-quality signups).
- Define precisely what counts as **“signup”** and what counts as **“retained”** (e.g., D30 retention, post-trial paid conversion, activity-based retention).
- Explain how you would handle:
- users assigned but never exposed to the offer (assignment vs exposure)
- users with insufficient follow-up time (censoring)
- seasonality or concurrent campaigns
2. **Analysis plan**
- Describe how you would estimate the treatment effect on signup rate and retention.
- Which statistical tests/models would you use (e.g., difference in proportions, logistic regression, survival analysis)?
- How would you compute confidence intervals and communicate uncertainty?
- What checks would you run before trusting results (e.g., SRM, balance checks, instrumentation validation)?
3. **Common implementation/logic issues (Python code review style)**
In a typical experiment analysis codebase, list **likely logic bugs or setup mistakes** you would look for, such as:
- incorrect attribution window
- filtering/conditioning on post-treatment behavior
- mixing per-event vs per-user denominators
- incorrectly defining “eligible population”
- double-counting users or handling cross-device users
- peeking / stopping rules
4. **Decision & next steps**
- Given possible outcomes (signup up, retention down; signup flat, retention up; etc.), outline what you would recommend to stakeholders.
- Propose at least one follow-up experiment or segmentation to understand heterogeneous effects (e.g., new vs returning users, geo, acquisition channel).
Quick Answer: This question evaluates skills in experimental design, causal inference and statistical analysis, metric definition and instrumentation, data-quality debugging, and interpretation of product/business trade-offs in A/B testing.