Interpreting confidence intervals to choose a treatment

Q: Interpreting confidence intervals to choose a treatment

This question evaluates a data scientist's ability to interpret confidence intervals, assess statistical significance and uncertainty, and incorporate business risk tolerance into experiment-based decisions.

Q: How do I approach Analytics & Experimentation interview questions?

Analytics & Experimentation questions require understanding of core concepts and practice. PracHub provides solutions with explanations to help you master analytics & experimentation interviews.

Question

Feed-Ranking Tweaks: Interpret 95% CIs and Choose a Launch Candidate

Context

You ran online experiments for three feed-ranking tweaks. The primary metric is percent change in engagement vs. control (negative = worse). For each treatment, you have a 95% confidence interval (CI) for the treatment effect. Leadership wants a recommendation that weighs statistical significance and business risk tolerance ("business threshold" for acceptable short-term loss).

Assumption: Effects are percentage-point changes in engagement; 95% CIs are already adjusted for multiplicity/peeking where relevant.

Data

Treatment A: −5% (95% CI: [−7.5%, −2.5%])
Treatment B: −15% (95% CI: [−17%, −13%])
Treatment C: −12% (95% CI: [−28%, −4%])

Task

Interpret each CI (significance, magnitude, uncertainty).
Assess risk vs. an acceptable business threshold for short-term engagement loss.
Choose which (if any) to launch and justify the decision.

Interpreting confidence intervals to choose a treatment

Feed-Ranking Tweaks: Interpret 95% CIs and Choose a Launch Candidate

Context

Data

Task

Solution

Comments (0)