Scenario: Three feed‑ranking tweaks have negative engagement effects with known 95 % CIs. Interpret intervals, assess risk, and pick a launch candidate. Question: Treatments yield CIs: -5(-7.5,-2. 5), -15(-17,- 13), -12(-28,- 4). Interpret and choose which to launch. (Hint: statistical significance, business threshold)

This question evaluates a data scientist's ability to interpret confidence intervals, assess statistical significance and uncertainty, and incorporate business risk tolerance into experiment-based decisions.

How do I approach Analytics & Experimentation interview questions?

Analytics & Experimentation questions require understanding of core concepts and practice. PracHub provides solutions with explanations to help you master analytics & experimentation interviews.

What difficulty level is this interview question?

This is a medium difficulty Analytics & Experimentation question, commonly asked during Onsite rounds at Meta.

What role is this question designed for?

This question is commonly asked for Data Scientist candidates at Meta during technical interviews.

Interpreting confidence intervals to choose a treatment

Feed-Ranking Tweaks: Interpret 95% CIs and Choose a Launch Candidate

Context

You ran online experiments for three feed-ranking tweaks. The primary metric is percent change in engagement vs. control (negative = worse). For each treatment, you have a 95% confidence interval (CI) for the treatment effect. Leadership wants a recommendation that weighs statistical significance and business risk tolerance ("business threshold" for acceptable short-term loss).

Assumption: Effects are percentage-point changes in engagement; 95% CIs are already adjusted for multiplicity/peeking where relevant.

Data

Treatment A: −5% (95% CI: [−7.5%, −2.5%])
Treatment B: −15% (95% CI: [−17%, −13%])
Treatment C: −12% (95% CI: [−28%, −4%])

Task

Interpret each CI (significance, magnitude, uncertainty).
Assess risk vs. an acceptable business threshold for short-term engagement loss.
Choose which (if any) to launch and justify the decision.

Feed-Ranking Tweaks: Interpret 95% CIs and Choose a Launch Candidate

Context

Assumption: Effects are percentage-point changes in engagement; 95% CIs are already adjusted for multiplicity/peeking where relevant.

Data

Treatment A: −5% (95% CI: [−7.5%, −2.5%])
Treatment B: −15% (95% CI: [−17%, −13%])
Treatment C: −12% (95% CI: [−28%, −4%])

Task

Interpret each CI (significance, magnitude, uncertainty).
Assess risk vs. an acceptable business threshold for short-term engagement loss.
Choose which (if any) to launch and justify the decision.

Interpreting confidence intervals to choose a treatment

Quick Overview

Feed-Ranking Tweaks: Interpret 95% CIs and Choose a Launch Candidate

Context

Data

Task

Solution

Submit Your Answer

Interpreting confidence intervals to choose a treatment

Quick Overview

Feed-Ranking Tweaks: Interpret 95% CIs and Choose a Launch Candidate

Context

Data

Task

Solution

Submit Your Answer