Analyze T2 Results and Recommend Launch Strategy
Company: Uber
Role: Data Scientist
Category: Analytics & Experimentation
Difficulty: hard
Interview Round: Technical Screen
##### Scenario
E-commerce platform tests two treatments (T1, T
2) that affect Gross Bookings (GB) and Variable Consideration (VC)
##### Question
T1 shows no significant change in GB or VC, while T2 shows a significant GB increase but significant VC decrease. Explain these results to the PM and recommend next steps. Given T2 confidence intervals (GB [+0.1%, +2.3%] ≈ +$0.48/order; VC [–2.5%, –1.5%] ≈ –$0.20/order), decide whether to launch and justify. Design a segmentation analysis to identify cohorts where GB lifts without hurting VC. If we will run 20 parallel feature experiments, define launch criteria, statistical thresholds, and how you will control error rates.
##### Hints
Contrast statistical vs practical significance, revenue vs margin trade-offs, multiple-testing corrections, and cohort discovery techniques.
Overview: Analyze T2 Results and Recommend Launch Strategy evaluates metric design, causal reasoning, experiment setup, diagnostics, SQL/statistical checks, and recommendations in a realistic interview setting. A strong answer states assumptions, handles edge cases, explains trade-offs, and shows how to validate the result clearly.
Community answers
Answer by SS
False Discovery Control
Multiple Testing Problem:
Running 20 tests at α = 0.05 → Expected 1 false positive even if nothing works
Need to control Family-Wise Error Rate (FWER) or False Discovery Rate (FDR)
Approach: Benjamini-Hochberg Procedure (FDR Control)
Rank all 20 p-values: p₁ ≤ p₂ ≤ ... ≤ p₂₀
Set target FDR = 0.05 (tolerate 5% false discoveries)
Find largest k where: p_k ≤ (k/20) × 0.05
Reject all hypotheses H₁, H₂, ..., H_k
Alternative: Bonferroni (More Conservative)
Adjusted α = 0.05/20 = 0.0025 per test
Controls FWER but reduces power