Determine Metrics to Measure Free-Trial Impact on Subscriptions

Q: How do I approach Analytics & Experimentation interview questions?

Analytics & Experimentation questions require understanding of core concepts and practice. PracHub provides solutions with explanations to help you master analytics & experimentation interviews.

Q: What difficulty level is this interview question?

This is a medium difficulty Analytics & Experimentation question, commonly asked during Take-home Project rounds at OpenAI.

Q: What role is this question designed for?

This question is commonly asked for Data Scientist candidates at OpenAI during technical interviews.

Question

Determine Metrics to Measure Free-Trial Impact on Subscriptions

A/B Test: Free Trial Offer Impact on Subscription Behavior

Scenario

You are analyzing a randomized A/B test in which free users are offered a free trial of the paid plan (treatment) versus no trial offer (control). Each user is assigned exactly once at first exposure.

Available Data (assumed minimal schema)

user_id
variant: "treatment" or "control"
assigned_at: timestamp of assignment/exposure
subscribed_at: timestamp when a user becomes a paid subscriber (null if never subscribes)
canceled_at: timestamp when a user cancels their paid subscription (null if never cancels)

Assume you have at least 30 days of observation after assignment for all users included in the analysis (or you censor/exclude users without a full window consistently across arms).

Tasks

Define the primary metrics you would track to measure the free-trial’s impact on subscription behavior. Use precise, unambiguous definitions with clear numerators, denominators, and time windows.
Calculate the lift in conversion to paid subscription between treatment and control.
Test whether the difference in conversion is statistically significant (state the test and how you’d compute it).
Estimate the treatment effect on churn (cancels) within 30 days and discuss whether the effect is practically meaningful.
Based on your results, would you recommend rolling the free-trial out to all users? Justify your answer considering both statistical and business significance.

Hints

Use consistent denominators (intent-to-treat where appropriate).
Use two-sample tests with appropriate confidence intervals.
Discuss both statistical and practical/business significance.

Clarifying Questions to Ask Guidance

Clarify the business objective, unit of analysis, time window, exposure definition, and primary metric.
State assumptions about instrumentation, randomization, sample size, and data quality.
Separate descriptive analysis from causal claims.

What a Strong Answer Covers Guidance

A metric framework with primary, guardrail, and diagnostic metrics.
A credible analysis or experiment design with clear assumptions and bias checks.
SQL/statistical logic for segmentation, variance, confidence, and data validation where relevant.
An actionable recommendation that explains trade-offs and next steps.

Follow-up Questions Guidance

What sanity checks would you run before trusting the result?
How would you handle novelty effects, seasonality, or selection bias?
What decision would you make if metrics disagree?

Determine Metrics to Measure Free-Trial Impact on Subscriptions

Quick Overview

Determine Metrics to Measure Free-Trial Impact on Subscriptions

Determine Metrics to Measure Free-Trial Impact on Subscriptions

A/B Test: Free Trial Offer Impact on Subscription Behavior

Scenario

Available Data (assumed minimal schema)

Tasks

Hints

Clarifying Questions to Ask Guidance

What a Strong Answer Covers Guidance

Follow-up Questions Guidance

Write your answer