Determine Metrics to Measure Free-Trial Impact on Subscriptions
A/B Test: Free Trial Offer Impact on Subscription Behavior
Scenario
You are analyzing a randomized A/B test in which free users are offered a free trial of the paid plan (treatment) versus no trial offer (control). Each user is assigned exactly once at first exposure.
Available Data (assumed minimal schema)
-
user_id
-
variant: "treatment" or "control"
-
assigned_at: timestamp of assignment/exposure
-
subscribed_at: timestamp when a user becomes a paid subscriber (null if never subscribes)
-
canceled_at: timestamp when a user cancels their paid subscription (null if never cancels)
Assume you have at least 30 days of observation after assignment for all users included in the analysis (or you censor/exclude users without a full window consistently across arms).
Tasks
-
Define the primary metrics you would track to measure the free-trial’s impact on subscription behavior. Use precise, unambiguous definitions with clear numerators, denominators, and time windows.
-
Calculate the lift in conversion to paid subscription between treatment and control.
-
Test whether the difference in conversion is statistically significant (state the test and how you’d compute it).
-
Estimate the treatment effect on churn (cancels) within 30 days and discuss whether the effect is practically meaningful.
-
Based on your results, would you recommend rolling the free-trial out to all users? Justify your answer considering both statistical and business significance.
Hints
-
Use consistent denominators (intent-to-treat where appropriate).
-
Use two-sample tests with appropriate confidence intervals.
-
Discuss both statistical and practical/business significance.
Constraints & Assumptions
-
Preserve the scope, facts, inputs, and requested outputs from the prompt above.
-
If the prompt leaves a detail unspecified, state a reasonable assumption before relying on it.
-
Keep the answer interview-ready: concise enough to present, but concrete enough to implement or evaluate.
Clarifying Questions to Ask
-
Clarify the business objective, unit of analysis, time window, exposure definition, and primary metric.
-
State assumptions about instrumentation, randomization, sample size, and data quality.
-
Separate descriptive analysis from causal claims.
What a Strong Answer Covers
-
A metric framework with primary, guardrail, and diagnostic metrics.
-
A credible analysis or experiment design with clear assumptions and bias checks.
-
SQL/statistical logic for segmentation, variance, confidence, and data validation where relevant.
-
An actionable recommendation that explains trade-offs and next steps.
Follow-up Questions
-
What sanity checks would you run before trusting the result?
-
How would you handle novelty effects, seasonality, or selection bias?
-
What decision would you make if metrics disagree?