Evaluate a free-trial A/B test

Scenario

A marketing team ran an A/B test offering a free 1-month trial to users.

• Control (A): Standard offer (no free month)
• Treatment (B): Free 1-month trial offer
• Randomization unit: user (assume one assignment per user)

The team cares about:

1. Signup rate (do more users start a subscription/trial?)
2. Retention (do users stick around after the trial / over time?)

Tasks

1. Experiment design & setup
   • State the hypothesis and the key product/business risk.
   • Specify the primary metric , diagnostic metrics , and guardrail metrics . Discuss tradeoffs (e.g., signup lift vs low-quality signups).
   • Define precisely what counts as “signup” and what counts as “retained” (e.g., D30 retention, post-trial paid conversion, activity-based retention).
   • Explain how you would handle:
     • users assigned but never exposed to the offer (assignment vs exposure)
     • users with insufficient follow-up time (censoring)
     • seasonality or concurrent campaigns
2. Analysis plan
   • Describe how you would estimate the treatment effect on signup rate and retention.
   • Which statistical tests/models would you use (e.g., difference in proportions, logistic regression, survival analysis)?
   • How would you compute confidence intervals and communicate uncertainty?
   • What checks would you run before trusting results (e.g., SRM, balance checks, instrumentation validation)?
3. Common implementation/logic issues (Python code review style) In a typical experiment analysis codebase, list likely logic bugs or setup mistakes you would look for, such as:
   • incorrect attribution window
   • filtering/conditioning on post-treatment behavior
   • mixing per-event vs per-user denominators
   • incorrectly defining “eligible population”
   • double-counting users or handling cross-device users
   • peeking / stopping rules
4. Decision & next steps
   • Given possible outcomes (signup up, retention down; signup flat, retention up; etc.), outline what you would recommend to stakeholders.
   • Propose at least one follow-up experiment or segmentation to understand heterogeneous effects (e.g., new vs returning users, geo, acquisition channel).