You are changing the Friend Recommendation algorithm to prioritize higher-quality connections. Design and analyze an end-to-end A/B test: 1) define primary success metrics and at most two counter-metrics, plus guardrails (latency, crash, abuse), and a cannibalization read on Stories; 2) pre-register MDEs and the minimum launch bar; 3) choose randomization unit and bucketing consistency for long-lived effects; 4) plan duration and ramp strategy with a holdback for long-term read; 5) specify how you’ll handle multiple looks and stopping rules to avoid alpha inflation; 6) interpret this realistic outcome: MAU +5% (p=0.03), per-user comments −3% (p=0.07), time spent +1% (p=0.20), Story creation −2% (p=0.01). Would you ship, partially ship, iterate, or stop? Justify to product, growth, and integrity stakeholders in one paragraph each.

This question evaluates experimentation design and analysis competencies for A/B testing, including metric selection, statistical interpretation, integrity and performance guardrails, and cross-functional stakeholder communication.

How do I approach Analytics & Experimentation interview questions?

Analytics & Experimentation questions require understanding of core concepts and practice. PracHub provides solutions with explanations to help you master analytics & experimentation interviews.

What difficulty level is this interview question?

This is a hard difficulty Analytics & Experimentation question, commonly asked during Technical Screen rounds at Meta.

What role is this question designed for?

This question is commonly asked for Data Scientist candidates at Meta during technical interviews.

Design and analyze end-to-end A/B test | Meta Interview Question

A/B Test Design: Higher-Quality Friend Recommendations

Context: You are updating the Friend Recommendation ("People You May Know") ranking to prioritize higher-quality connections on a large consumer social platform. Design and analyze an end-to-end A/B test that evaluates user value, growth impact, and integrity risks.

Tasks

Metrics

Define primary success metrics and at most two counter-metrics.
Specify guardrails for latency, crash, and abuse.
Include a cannibalization read on Stories (creation/consumption).

Pre-registration

Pre-register minimum detectable effects (MDEs) and a clear minimum launch bar (decision thresholds).

Experimental Unit and Bucketing

Choose the randomization unit.
Specify bucketing consistency to support long-lived effects.

Duration and Ramp

Provide a ramp strategy (including a small canary) and total duration.
Include a persistent holdback for long-term reads.

Multiple Looks and Stopping Rules

Describe how you will handle multiple looks and stopping to avoid alpha inflation.

Outcome Interpretation

Given this outcome, decide whether to ship, partially ship, iterate, or stop, and justify to three stakeholder groups:
- MAU +5% (p=0.03)
- Per-user comments −3% (p=0.07)
- Time spent +1% (p=0.20)
- Story creation −2% (p=0.01)
Provide one short paragraph each for Product, Growth, and Integrity stakeholders.

Tasks

Metrics

Define primary success metrics and at most two counter-metrics.

Specify guardrails for latency, crash, and abuse.

Include a cannibalization read on Stories (creation/consumption).

Pre-registration

Pre-register minimum detectable effects (MDEs) and a clear minimum launch bar (decision thresholds).

Experimental Unit and Bucketing

Choose the randomization unit.

Specify bucketing consistency to support long-lived effects.

Duration and Ramp

Provide a ramp strategy (including a small canary) and total duration.

Include a persistent holdback for long-term reads.

Multiple Looks and Stopping Rules

Describe how you will handle multiple looks and stopping to avoid alpha inflation.

Outcome Interpretation

Given this outcome, decide whether to ship, partially ship, iterate, or stop, and justify to three stakeholder groups:

MAU +5% (p=0.03)
Per-user comments −3% (p=0.07)
Time spent +1% (p=0.20)
Story creation −2% (p=0.01)

Provide one short paragraph each for Product, Growth, and Integrity stakeholders.

Design and analyze end-to-end A/B test

Quick Overview

A/B Test Design: Higher-Quality Friend Recommendations

Tasks

Solution

Submit Your Answer to Earn 20XP

Design and analyze end-to-end A/B test

Quick Overview

A/B Test Design: Higher-Quality Friend Recommendations

Tasks

Solution

Submit Your Answer to Earn 20XP