Analyze A/B Test Results for Subscription Conversion Rates
Overview
This question evaluates a data scientist's competency in A/B testing, statistical inference, experiment design, and product-metrics analysis, including conversion-rate computation, significance testing, churn attribution, and estimating net subscriber impact over time.
A/B Test: Free-Trial Offer Impact on Paid Subscriptions and Churn
Context
You are analyzing an A/B test where free users in the Treatment arm are offered a limited‑time trial of the paid plan. The Control arm sees the status quo (no trial). The goal is to evaluate whether the trial offer increases paid subscriptions and reduces churn.
Assume the analysis window includes day 0 (assignment) through at least 60 days. Define:
- N_T, N_C: users exposed in Treatment and Control.
- Trial length L (days), paid billing cycle length B = 30 days.
- Trial starters in Treatment: T_T (optional if available).
- Paid conversions by day t: S_T_paid(t), S_C_paid(t).
- Cancels during trial (Treatment): X_T_trial_cancel.
- Cancels within first paid billing cycle (among those who become paid): X_T_paid_cycle1, X_C_paid_cycle1.
Tasks
- Compute the signup (paid-conversion) rate for Treatment vs. Control and the percentage lift.
- Test whether the lift is statistically significant (state test, null/alt hypotheses, and p‑value or CI).
- Calculate and compare cancel rates during the trial and after the first paid billing cycle.
- Estimate net paid‑subscriber change after 30 and 60 days, incorporating both signups and cancels.
- Identify additional metrics or user segments to examine before recommending a full roll‑out.
- Summarize the experiment outcome and provide a go / no‑go recommendation with supporting numbers.
Hint: Use a standard A/B testing framework: define metrics clearly, check randomization/SRM, use two‑proportion tests or the delta method, and segment by tenure and time buckets.
Solution
Login required