##### Scenario An A/B test offers free users a limited-time trial of the paid plan to see whether it increases paid subscriptions and reduces churn. ##### Question Compute the signup (paid-conversion) rate for treatment vs. control and the percentage lift. 2. Test whether the lift is statistically significant (state your test, null/alt hypotheses, p-value or CI). 3. Calculate and compare cancel rates during the trial and after the first paid billing cycle. 4. Estimate net paid-subscriber change after 30 days and 60 days, incorporating both signups and cancels. 5. What additional metrics or user segments would you examine before recommending a full roll-out? 6. Summarize the experiment outcome and give a go / no-go recommendation with supporting numbers. ##### Hints Standard A/B-testing framework: define metrics, check randomization, use proportion test or delta method, segment by tenure, bucketed time windows.

Evaluates A/B test analysis for a limited-time free trial offer. Strong answers compute paid-conversion lift, account for trial starts and churn, test statistical significance, and make a launch decision using retained paid value, revenue, guardrails, and long-term holdouts.

How do I approach Analytics & Experimentation interview questions?

Analytics & Experimentation questions require understanding of core concepts and practice. PracHub provides solutions with explanations to help you master analytics & experimentation interviews.

What difficulty level is this interview question?

This is a medium difficulty Analytics & Experimentation question, commonly asked during Take-home Project rounds at OpenAI.

What role is this question designed for?

This question is commonly asked for Data Scientist candidates at OpenAI during technical interviews.

Analyze A/B Test Results for Subscription Conversion Rates

An A/B test offers free users a limited-time trial of the paid plan to see whether it increases paid subscriptions and reduces churn. The control group receives the status quo experience without the trial offer.

Constraints & Assumptions

Treat assignment as user-level randomized unless stated otherwise.
The analysis window should include trial start, trial end, first paid billing cycle, and churn after conversion.
Distinguish trial starts, paid conversions, and retained paid subscribers.
Include statistical significance, practical significance, and guardrails.

Clarifying Questions to Ask

What is the trial length, and when is a user counted as paid?
Are users eligible only once, or can they receive multiple offers?
What is the primary metric: paid conversion by day 30, retained paid conversion by day 60, revenue, or churn?
Are there refunds, cancellations, or involuntary churn events?

Part 1 - Compute Conversion Metrics

Compute the paid-conversion rate for treatment versus control and the percentage lift.

What This Part Should Cover

Treatment and control conversion rates using exposed users as denominators.
Absolute lift and relative percentage lift.
Clear conversion window and distinction between trial starters and paid subscribers.

Part 2 - Compute Churn and Retention

How would you measure trial cancellation, first-cycle churn, and retained paid conversion?

What This Part Should Cover

Trial cancel rate, paid-cycle cancel rate, retained paid rate, and revenue or LTV if available.
Why raw trial starts can be misleading if users churn after the free period.

Part 3 - Test Statistical Significance

How would you test whether the conversion-rate difference is statistically significant?

What This Part Should Cover

Two-proportion z test or logistic regression with covariate adjustment.
Confidence intervals for absolute and relative lift.
Power, minimum detectable effect, sample-ratio mismatch, and multiple metrics.

Part 4 - Make a Launch Decision

How would you decide whether to launch the trial offer?

What This Part Should Cover

Primary metric, guardrails, revenue/LTV, churn, user experience, support cost, and segment effects.
Practical significance and long-term holdout considerations.
Recommendation framework for launch, iterate, or stop.

What a Strong Answer Covers

A strong answer separates trial adoption from paid conversion and retained value, computes lift correctly, tests significance, and makes a launch decision using conversion, churn, revenue, and guardrails together.

Follow-up Questions

What if treatment increases paid conversion but also increases first-month churn?
How would you analyze users who start a trial but never become paid?
How would delayed conversions affect the analysis window?