A/B Test: Free Trial Offer Impact on Subscription Behavior

Scenario

You are analyzing a randomized A/B test in which free users are offered a free trial of the paid plan (treatment) versus no trial offer (control). Each user is assigned exactly once at first exposure.

Available Data (assumed minimal schema)

• user_id
• variant: "treatment" or "control"
• assigned_at: timestamp of assignment/exposure
• subscribed_at: timestamp when a user becomes a paid subscriber (null if never subscribes)
• canceled_at: timestamp when a user cancels their paid subscription (null if never cancels)

Assume you have at least 30 days of observation after assignment for all users included in the analysis (or you censor/exclude users without a full window consistently across arms).

Tasks

1. Define the primary metrics you would track to measure the free-trial’s impact on subscription behavior. Use precise, unambiguous definitions with clear numerators, denominators, and time windows.
2. Calculate the lift in conversion to paid subscription between treatment and control.
3. Test whether the difference in conversion is statistically significant (state the test and how you’d compute it).
4. Estimate the treatment effect on churn (cancels) within 30 days and discuss whether the effect is practically meaningful.
5. Based on your results, would you recommend rolling the free-trial out to all users? Justify your answer considering both statistical and business significance.

Hints

• Use consistent denominators (intent-to-treat where appropriate).
• Use two-sample tests with appropriate confidence intervals.
• Discuss both statistical and practical/business significance.