How do I approach Analytics & Experimentation interview questions?

Analytics & Experimentation questions require understanding of core concepts and practice. PracHub provides solutions with explanations to help you master analytics & experimentation interviews.

What difficulty level is this interview question?

This is a hard difficulty Analytics & Experimentation question, commonly asked during Onsite rounds at Meta.

What role is this question designed for?

This question is commonly asked for Data Scientist candidates at Meta during technical interviews.

Design an Experiment to Evaluate New Recommendation Model | Meta Interview Question

Design an Experiment to Evaluate New Recommendation Model

Last updated: Mar 29, 2026

Quick Overview

This question evaluates proficiency in online experiment design, A/B testing, metric selection and interpretation, statistical power and significance, heterogeneity analysis, and translating CTR lifts into quantified business impact.

Experiment Design: New Ads Ranking Model vs. Current System

Context

You are evaluating a newly built ML ranking model for an ads recommendation surface. The goal is to determine whether the new model should replace the current system based on rigorous online experimentation.

Task

Design an experiment to evaluate the new recommendation model. Specify:

Control and treatment assignment, traffic split, contamination avoidance, and duration.
Primary, guard-rail, and long-term metrics (with rationale) and success criteria.
Sample size and statistical significance considerations; address heterogeneous effects.
How to quantify business gain if an A/B test shows a 5% lift in overall CTR.
Interpretation of a reported 100% CTR increase for males aged 18–55 in India.
Decision-making: If the experiment shows +5% CTR and +5% revenue with no negative guard-rail impact, would you roll out? Explain risk mitigation.

Solution

Show

Comments (0)

Loading comments...

Browse More Questions

More Analytics & Experimentation•More Meta•More Data Scientist•Meta Data Scientist•Meta Analytics & Experimentation•Data Scientist Analytics & Experimentation