Interpreting confidence intervals to choose a treatment
Quick Overview
Evaluates confidence-interval interpretation for choosing among feed-ranking treatments with engagement losses. Strong answers compare statistical significance, magnitude, uncertainty, business thresholds, downside risk, and launch recommendations.
Interpreting confidence intervals to choose a treatment
Company: Meta
Role: Data Scientist
Category: Analytics & Experimentation
Difficulty: medium
Interview Round: Onsite
Scenario:
Three feed‑ranking tweaks have negative engagement effects with known 95 % CIs. Interpret intervals, assess risk, and pick a launch candidate.
Question:
Treatments yield CIs: -5(-7.5,-2.
5), -15(-17,-
13), -12(-28,-
4). Interpret and choose which to launch. (Hint: statistical significance, business threshold)
Overview: Evaluates confidence-interval interpretation for choosing among feed-ranking treatments with engagement losses. Strong answers compare statistical significance, magnitude, uncertainty, business thresholds, downside risk, and launch recommendations.
Feed-ranking Tweaks: Interpret Confidence Intervals and Choose a Treatment
You ran online experiments for three feed-ranking tweaks. The primary metric is percent change in engagement versus control, where negative means worse. Leadership wants a recommendation that weighs statistical significance and business risk tolerance.
Assume the 95% confidence intervals are already adjusted for multiplicity or peeking where relevant.
Data:
Treatment A: -5% with 95% CI [-7.5%, -2.5%]
Treatment B: -15% with 95% CI [-17%, -13%]
Treatment C: -12% with 95% CI [-28%, -4%]
Constraints & Assumptions
Interpret both statistical significance and business significance.
Compare expected effect, worst-case downside, and uncertainty.
Use an acceptable short-term engagement-loss threshold if one is provided; otherwise state the threshold assumption.
Do not recommend a launch solely because a result is statistically significant.
Clarifying Questions to Ask Guidance
What short-term engagement loss is acceptable for the business?
Are there long-term benefits or secondary metrics not shown here?
Are guardrails such as retention, satisfaction, safety, or revenue available?
Are the CIs for percentage change or percentage-point change?
Part 1 - Interpret Each CI
Interpret the significance, magnitude, and uncertainty for treatments A, B, and C.
What This Part Should Cover Guidance
Note that all three CIs are below zero, indicating statistically significant engagement decreases.
Compare magnitude of expected loss.
Compare uncertainty: B is precise and harmful, C is highly uncertain with large downside, A is moderate and narrower.
Part 2 - Risk and Business Threshold
Assess each treatment against a business threshold for acceptable short-term loss.
What This Part Should Cover Guidance
Use the upper and lower bounds to discuss best plausible and worst plausible outcomes.
Reject treatments whose confidence interval or expected loss exceeds the acceptable threshold.
Consider secondary benefits only if they are credible and measured.
Part 3 - Launch Recommendation
Choose which, if any, to launch and justify the decision.
What This Part Should Cover Guidance
If the goal is engagement and no offsetting long-term benefit exists, recommend launching none.
If a small short-term loss is acceptable for strategic reasons, A is the least risky candidate for cautious ramp or further testing.
Do not launch B or C without strong offsetting evidence because B is clearly harmful and C has wide downside.
Recommend more data, longer-term metrics, or targeted ramp where needed.
Follow-up Questions Guidance
How would your recommendation change if Treatment A improves retention?
What if leadership can tolerate at most a 3% short-term loss?
How would you communicate CI uncertainty to non-technical stakeholders?