##### Scenario Evaluating two treatments (t1, t 2) on Gross Booking (GB) and Variable Consideration (VC) ##### Question Treatment t1 shows no significant change in GB or VC, while t2 shows a significant increase in GB but a significant decrease in VC. How would you explain these results to the PM and recommend next steps? Given confidence intervals GB [+0.1%, +2.3%] with +$0.48 lift, and VC [-2.5%, -1.5%] with -$0.20 loss, would you launch t2? Justify your decision. How would you segment users or orders to identify cohorts with positive GB impact and no negative VC impact? If 20 different experiments run simultaneously, how would you define launch criteria to control false discoveries and ensure reliable decisions? ##### Hints Discuss trade-offs, statistical power, cost-benefit, cohort analysis, multiple-testing corrections (e.g., FDR).

Explain Treatment Results and Recommend Launch Criteria for Experiments evaluates metric design, causal reasoning, experiment setup, diagnostics, SQL/statistical checks, and recommendations in a realistic interview setting. A strong answer states assumptions, handles edge cases, explains trade-offs, and shows how to validate the result clearly.

How do I approach Analytics & Experimentation interview questions?

Analytics & Experimentation questions require understanding of core concepts and practice. PracHub provides solutions with explanations to help you master analytics & experimentation interviews.

What difficulty level is this interview question?

This is a hard difficulty Analytics & Experimentation question, commonly asked during Technical Screen rounds at Upstart.

What role is this question designed for?

This question is commonly asked for Data Scientist candidates at Upstart during technical interviews.

Explain Treatment Results and Recommend Launch Criteria for Experiments

A/B Test Interpretation, Launch Decision, Segmentation, and Multiple-Testing Control

Context

You ran an experiment with two treatments (t1, t2) against a control. Two core business metrics were tracked:

Gross Booking (GB): a volume/topline metric (e.g., GMV, loan originations, order value).
Variable Consideration (VC): a monetization metric (e.g., revenue/take-rate/fees tied to transactions).

Observed results:

t1: No statistically significant change in GB or VC.
t2: Statistically significant increase in GB and statistically significant decrease in VC.

Confidence intervals for t2 (vs. control):

GB: +0.1% to +2.3%; point-estimated lift +$0.48.
VC: −2.5% to −1.5%; point-estimated loss −$0.20.

Questions

How would you explain these results to the PM and recommend next steps?
Given the confidence intervals above, would you launch t2? Justify your decision in terms of business objectives and risk.
How would you segment users or orders to identify cohorts with positive GB impact and no negative VC impact?
If 20 different experiments run simultaneously, how would you define portfolio-level launch criteria to control false discoveries and ensure reliable decisions?

Hint: Discuss trade-offs, statistical power, cost–benefit, cohort analysis, and multiple-testing corrections (e.g., FDR).

Constraints & Assumptions

Preserve the scope, facts, inputs, and requested outputs from the prompt above.
If the prompt leaves a detail unspecified, state a reasonable assumption before relying on it.
Keep the answer interview-ready: concise enough to present, but concrete enough to implement or evaluate.

Clarifying Questions to Ask

Clarify the business objective, unit of analysis, time window, exposure definition, and primary metric.
State assumptions about instrumentation, randomization, sample size, and data quality.
Separate descriptive analysis from causal claims.

What a Strong Answer Covers

A metric framework with primary, guardrail, and diagnostic metrics.
A credible analysis or experiment design with clear assumptions and bias checks.
SQL/statistical logic for segmentation, variance, confidence, and data validation where relevant.
An actionable recommendation that explains trade-offs and next steps.

Follow-up Questions

What sanity checks would you run before trusting the result?
How would you handle novelty effects, seasonality, or selection bias?
What decision would you make if metrics disagree?

A/B Test Interpretation, Launch Decision, Segmentation, and Multiple-Testing Control

Context

You ran an experiment with two treatments (t1, t2) against a control. Two core business metrics were tracked:

Gross Booking (GB): a volume/topline metric (e.g., GMV, loan originations, order value).
Variable Consideration (VC): a monetization metric (e.g., revenue/take-rate/fees tied to transactions).

Observed results:

t1: No statistically significant change in GB or VC.
t2: Statistically significant increase in GB and statistically significant decrease in VC.

Confidence intervals for t2 (vs. control):

GB: +0.1% to +2.3%; point-estimated lift +$0.48.
VC: −2.5% to −1.5%; point-estimated loss −$0.20.

Questions

How would you explain these results to the PM and recommend next steps?
Given the confidence intervals above, would you launch t2? Justify your decision in terms of business objectives and risk.
How would you segment users or orders to identify cohorts with positive GB impact and no negative VC impact?
If 20 different experiments run simultaneously, how would you define portfolio-level launch criteria to control false discoveries and ensure reliable decisions?

Hint: Discuss trade-offs, statistical power, cost–benefit, cohort analysis, and multiple-testing corrections (e.g., FDR).

Constraints & Assumptions

Preserve the scope, facts, inputs, and requested outputs from the prompt above.
If the prompt leaves a detail unspecified, state a reasonable assumption before relying on it.
Keep the answer interview-ready: concise enough to present, but concrete enough to implement or evaluate.

Clarifying Questions to Ask

Clarify the business objective, unit of analysis, time window, exposure definition, and primary metric.
State assumptions about instrumentation, randomization, sample size, and data quality.
Separate descriptive analysis from causal claims.

What a Strong Answer Covers

A metric framework with primary, guardrail, and diagnostic metrics.
A credible analysis or experiment design with clear assumptions and bias checks.
SQL/statistical logic for segmentation, variance, confidence, and data validation where relevant.
An actionable recommendation that explains trade-offs and next steps.

Follow-up Questions

What sanity checks would you run before trusting the result?
How would you handle novelty effects, seasonality, or selection bias?
What decision would you make if metrics disagree?

Explain Treatment Results and Recommend Launch Criteria for Experiments

Quick Overview

Explain Treatment Results and Recommend Launch Criteria for Experiments

A/B Test Interpretation, Launch Decision, Segmentation, and Multiple-Testing Control

Context

Questions

Constraints & Assumptions

Clarifying Questions to Ask

What a Strong Answer Covers

Follow-up Questions

Write your answer

Explain Treatment Results and Recommend Launch Criteria for Experiments

Quick Overview

Explain Treatment Results and Recommend Launch Criteria for Experiments

A/B Test Interpretation, Launch Decision, Segmentation, and Multiple-Testing Control

Context

Questions

Constraints & Assumptions

Clarifying Questions to Ask

What a Strong Answer Covers

Follow-up Questions

Write your answer