Explain Treatment Results and Recommend Launch Criteria for Experiments

Q: How do I approach Analytics & Experimentation interview questions?

Analytics & Experimentation questions require understanding of core concepts and practice. PracHub provides solutions with explanations to help you master analytics & experimentation interviews.

Q: What difficulty level is this interview question?

This is a hard difficulty Analytics & Experimentation question, commonly asked during Technical Screen rounds at Upstart.

Q: What role is this question designed for?

This question is commonly asked for Data Scientist candidates at Upstart during technical interviews.

Question

A/B Test Interpretation, Launch Decision, Segmentation, and Multiple-Testing Control

Context

You ran an experiment with two treatments (t1, t2) against a control. Two core business metrics were tracked:

Gross Booking (GB): a volume/topline metric (e.g., GMV, loan originations, order value).
Variable Consideration (VC): a monetization metric (e.g., revenue/take-rate/fees tied to transactions).

Observed results:

t1: No statistically significant change in GB or VC.
t2: Statistically significant increase in GB and statistically significant decrease in VC.

Confidence intervals for t2 (vs. control):

GB: +0.1% to +2.3%; point-estimated lift +$0.48.
VC: −2.5% to −1.5%; point-estimated loss −$0.20.

Questions

How would you explain these results to the PM and recommend next steps?
Given the confidence intervals above, would you launch t2? Justify your decision in terms of business objectives and risk.
How would you segment users or orders to identify cohorts with positive GB impact and no negative VC impact?
If 20 different experiments run simultaneously, how would you define portfolio-level launch criteria to control false discoveries and ensure reliable decisions?

Hint: Discuss trade-offs, statistical power, cost–benefit, cohort analysis, and multiple-testing corrections (e.g., FDR).

Clarifying Questions to Ask Guidance

Clarify the business objective, unit of analysis, time window, exposure definition, and primary metric.
State assumptions about instrumentation, randomization, sample size, and data quality.
Separate descriptive analysis from causal claims.

What a Strong Answer Covers Guidance

A metric framework with primary, guardrail, and diagnostic metrics.
A credible analysis or experiment design with clear assumptions and bias checks.
SQL/statistical logic for segmentation, variance, confidence, and data validation where relevant.
An actionable recommendation that explains trade-offs and next steps.

Follow-up Questions Guidance

What sanity checks would you run before trusting the result?
How would you handle novelty effects, seasonality, or selection bias?
What decision would you make if metrics disagree?

Explain Treatment Results and Recommend Launch Criteria for Experiments

Quick Overview

Explain Treatment Results and Recommend Launch Criteria for Experiments

A/B Test Interpretation, Launch Decision, Segmentation, and Multiple-Testing Control

Context

Questions

Clarifying Questions to Ask Guidance

What a Strong Answer Covers Guidance

Follow-up Questions Guidance

Write your answer