Analyze A/B Test Results for Subscription Conversion Rates
Quick Overview
Evaluates A/B test analysis for a limited-time free trial offer. Strong answers compute paid-conversion lift, account for trial starts and churn, test statistical significance, and make a launch decision using retained paid value, revenue, guardrails, and long-term holdouts.
Analyze A/B Test Results for Subscription Conversion Rates
Company: OpenAI
Role: Data Scientist
Category: Analytics & Experimentation
Difficulty: medium
Interview Round: Take-home Project
##### Scenario
An A/B test offers free users a limited-time trial of the paid plan to see whether it increases paid subscriptions and reduces churn.
##### Question
Compute the signup (paid-conversion) rate for treatment vs. control and the percentage lift. 2. Test whether the lift is statistically significant (state your test, null/alt hypotheses, p-value or CI). 3. Calculate and compare cancel rates during the trial and after the first paid billing cycle. 4. Estimate net paid-subscriber change after 30 days and 60 days, incorporating both signups and cancels. 5. What additional metrics or user segments would you examine before recommending a full roll-out? 6. Summarize the experiment outcome and give a go / no-go recommendation with supporting numbers.
##### Hints
Standard A/B-testing framework: define metrics, check randomization, use proportion test or delta method, segment by tenure, bucketed time windows.
Quick Answer: Evaluates A/B test analysis for a limited-time free trial offer. Strong answers compute paid-conversion lift, account for trial starts and churn, test statistical significance, and make a launch decision using retained paid value, revenue, guardrails, and long-term holdouts.
Analyze A/B Test Results for Subscription Conversion Rates
An A/B test offers free users a limited-time trial of the paid plan to see whether it increases paid subscriptions and reduces churn. The control group receives the status quo experience without the trial offer.
Constraints & Assumptions
Treat assignment as user-level randomized unless stated otherwise.
The analysis window should include trial start, trial end, first paid billing cycle, and churn after conversion.
Distinguish trial starts, paid conversions, and retained paid subscribers.
Include statistical significance, practical significance, and guardrails.
Clarifying Questions to Ask Guidance
What is the trial length, and when is a user counted as paid?
Are users eligible only once, or can they receive multiple offers?
What is the primary metric: paid conversion by day 30, retained paid conversion by day 60, revenue, or churn?
Are there refunds, cancellations, or involuntary churn events?
Part 1 - Compute Conversion Metrics
Compute the paid-conversion rate for treatment versus control and the percentage lift.
What This Part Should Cover Guidance
Treatment and control conversion rates using exposed users as denominators.
Absolute lift and relative percentage lift.
Clear conversion window and distinction between trial starters and paid subscribers.
Part 2 - Compute Churn and Retention
How would you measure trial cancellation, first-cycle churn, and retained paid conversion?
What This Part Should Cover Guidance
Trial cancel rate, paid-cycle cancel rate, retained paid rate, and revenue or LTV if available.
Why raw trial starts can be misleading if users churn after the free period.
Part 3 - Test Statistical Significance
How would you test whether the conversion-rate difference is statistically significant?
What This Part Should Cover Guidance
Two-proportion z test or logistic regression with covariate adjustment.
Confidence intervals for absolute and relative lift.
Power, minimum detectable effect, sample-ratio mismatch, and multiple metrics.
Part 4 - Make a Launch Decision
How would you decide whether to launch the trial offer?
What This Part Should Cover Guidance
Primary metric, guardrails, revenue/LTV, churn, user experience, support cost, and segment effects.
Practical significance and long-term holdout considerations.
Recommendation framework for launch, iterate, or stop.
What a Strong Answer Covers Guidance
A strong answer separates trial adoption from paid conversion and retained value, computes lift correctly, tests significance, and makes a launch decision using conversion, churn, revenue, and guardrails together.
Follow-up Questions Guidance
What if treatment increases paid conversion but also increases first-month churn?
How would you analyze users who start a trial but never become paid?
How would delayed conversions affect the analysis window?