How do I approach Analytics & Experimentation interview questions?

Analytics & Experimentation questions require understanding of core concepts and practice. PracHub provides solutions with explanations to help you master analytics & experimentation interviews.

What difficulty level is this interview question?

This is a hard difficulty Analytics & Experimentation question, commonly asked during Technical Screen rounds at Uber.

What role is this question designed for?

This question is commonly asked for Data Scientist candidates at Uber during technical interviews.

Analyze T2 Results and Recommend Launch Strategy

Quick Overview

This question evaluates experimental design and A/B test interpretation skills, including statistical versus practical significance, trade-offs between growth (gross bookings) and margin (variable contribution), segmentation for heterogeneous effects, and portfolio-level multiple-testing and error-control competency.

A/B Test Interpretation, Launch Decision, Segmentation, and Multi-Experiment Error Control

Context

You ran two A/B tests on an e-commerce platform:

T1 and T2 are feature variants intended to impact two business metrics:
- Gross Bookings (GB): pre-fee, pre-incentive order value (a growth metric).
- VC: Variable Contribution (margin) per order (i.e., contribution margin after variable costs). Assumption: A decrease in VC is margin-negative. If your org defines VC differently (e.g., as a contra-revenue where a decrease is good), flip the sign logic accordingly.

Observed Results

T1: No statistically significant change in GB or VC.
T2: Statistically significant increase in GB but statistically significant decrease in VC.
- T2 confidence intervals:
  - GB: [+0.1%, +2.3%] ≈ +$0.48 per order
  - VC: [–2.5%, –1.5%] ≈ –$0.20 per order

Tasks

Explain these results to the PM (statistical vs practical significance; growth vs margin trade-offs; plausible mechanisms).
Decide whether to launch T2 using the given CIs and per-order impacts, and justify the decision.
Design a segmentation analysis to identify cohorts where GB lifts without hurting VC.
If you will run 20 parallel feature experiments, define:
- Launch criteria and statistical thresholds for the primary and guardrail metrics.
- How you will control false discoveries and error rates across the portfolio.

Hints

Contrast statistical vs practical significance.
Weigh revenue (GB) vs margin (VC) trade-offs.
Apply multiple-testing corrections where appropriate.
Use principled cohort discovery techniques that avoid p-hacking.

Quick Overview

Context

You ran two A/B tests on an e-commerce platform:

T1 and T2 are feature variants intended to impact two business metrics:

Gross Bookings (GB): pre-fee, pre-incentive order value (a growth metric).
VC: Variable Contribution (margin) per order (i.e., contribution margin after variable costs). Assumption: A decrease in VC is margin-negative. If your org defines VC differently (e.g., as a contra-revenue where a decrease is good), flip the sign logic accordingly.

Tasks

Explain these results to the PM (statistical vs practical significance; growth vs margin trade-offs; plausible mechanisms).

Decide whether to launch T2 using the given CIs and per-order impacts, and justify the decision.

Design a segmentation analysis to identify cohorts where GB lifts without hurting VC.

If you will run 20 parallel feature experiments, define:

Launch criteria and statistical thresholds for the primary and guardrail metrics.
How you will control false discoveries and error rates across the portfolio.

Analyze T2 Results and Recommend Launch Strategy

Quick Overview

A/B Test Interpretation, Launch Decision, Segmentation, and Multi-Experiment Error Control

Context

Observed Results

Tasks

Hints

Solution

Submit Your Answer to Earn 20XP

Analyze T2 Results and Recommend Launch Strategy

Quick Overview

A/B Test Interpretation, Launch Decision, Segmentation, and Multi-Experiment Error Control

Context

Observed Results

Tasks

Hints

Solution

Submit Your Answer to Earn 20XP