##### Scenario Evaluating two treatments (t1, t 2) on Gross Booking (GB) and Variable Consideration (VC) ##### Question Treatment t1 shows no significant change in GB or VC, while t2 shows a significant increase in GB but a significant decrease in VC. How would you explain these results to the PM and recommend next steps? Given confidence intervals GB [+0.1%, +2.3%] with +$0.48 lift, and VC [-2.5%, -1.5%] with -$0.20 loss, would you launch t2? Justify your decision. How would you segment users or orders to identify cohorts with positive GB impact and no negative VC impact? If 20 different experiments run simultaneously, how would you define launch criteria to control false discoveries and ensure reliable decisions? ##### Hints Discuss trade-offs, statistical power, cost-benefit, cohort analysis, multiple-testing corrections (e.g., FDR).

This question evaluates a data scientist's ability to interpret A/B test results, balance trade-offs between volume and monetization metrics, apply multiple-testing corrections, and design segmentation to identify heterogeneous treatment effects.

How do I approach Analytics & Experimentation interview questions?

Analytics & Experimentation questions require understanding of core concepts and practice. PracHub provides solutions with explanations to help you master analytics & experimentation interviews.

What difficulty level is this interview question?

This is a hard difficulty Analytics & Experimentation question, commonly asked during Technical Screen rounds at Upstart.

What role is this question designed for?

This question is commonly asked for Data Scientist candidates at Upstart during technical interviews.

Explain Treatment Results and Recommend Launch Criteria for Experiments

A/B Test Interpretation, Launch Decision, Segmentation, and Multiple-Testing Control

Context

You ran an experiment with two treatments (t1, t2) against a control. Two core business metrics were tracked:

Gross Booking (GB): a volume/topline metric (e.g., GMV, loan originations, order value).
Variable Consideration (VC): a monetization metric (e.g., revenue/take-rate/fees tied to transactions).

Observed results:

t1: No statistically significant change in GB or VC.
t2: Statistically significant increase in GB and statistically significant decrease in VC.

Confidence intervals for t2 (vs. control):

GB: +0.1% to +2.3%; point-estimated lift +$0.48.
VC: −2.5% to −1.5%; point-estimated loss −$0.20.

Questions

How would you explain these results to the PM and recommend next steps?
Given the confidence intervals above, would you launch t2? Justify your decision in terms of business objectives and risk.
How would you segment users or orders to identify cohorts with positive GB impact and no negative VC impact?
If 20 different experiments run simultaneously, how would you define portfolio-level launch criteria to control false discoveries and ensure reliable decisions?

Hint: Discuss trade-offs, statistical power, cost–benefit, cohort analysis, and multiple-testing corrections (e.g., FDR).

Context

You ran an experiment with two treatments (t1, t2) against a control. Two core business metrics were tracked:

Gross Booking (GB): a volume/topline metric (e.g., GMV, loan originations, order value).

Variable Consideration (VC): a monetization metric (e.g., revenue/take-rate/fees tied to transactions).

Observed results:

t1: No statistically significant change in GB or VC.

t2: Statistically significant increase in GB and statistically significant decrease in VC.

Confidence intervals for t2 (vs. control):

GB: +0.1% to +2.3%; point-estimated lift +$0.48.

VC: −2.5% to −1.5%; point-estimated loss −$0.20.

Questions

How would you explain these results to the PM and recommend next steps?

Given the confidence intervals above, would you launch t2? Justify your decision in terms of business objectives and risk.

How would you segment users or orders to identify cohorts with positive GB impact and no negative VC impact?

If 20 different experiments run simultaneously, how would you define portfolio-level launch criteria to control false discoveries and ensure reliable decisions?

Hint: Discuss trade-offs, statistical power, cost–benefit, cohort analysis, and multiple-testing corrections (e.g., FDR).

Explain Treatment Results and Recommend Launch Criteria for Experiments

Quick Overview

A/B Test Interpretation, Launch Decision, Segmentation, and Multiple-Testing Control

Context

Questions

Solution

Comments (0)

Explain Treatment Results and Recommend Launch Criteria for Experiments

Quick Overview

A/B Test Interpretation, Launch Decision, Segmentation, and Multiple-Testing Control

Context

Questions

Solution

Comments (0)