Run a clean A/B test for recommendations

Quick Overview

This question evaluates a data scientist's competency in experimental design, causal inference, and statistical power analysis for A/B tests, covering randomization unit choice, contamination prevention, metric and guardrail selection, clustering adjustments, and bias-control techniques.

Run a clean A/B test for recommendations

Company: Meta

Role: Data Scientist

Category: Analytics & Experimentation

Difficulty: medium

Interview Round: Onsite

You must run an A/B test to evaluate the new hashtag recommender starting on 2025‑09‑01. 1) Define the randomization unit (user/session/impression) and justify it given potential interference (users can see and follow the same hashtag across sessions). 2) Specify treatment and control selection, rollout/ramp plan, and how you will prevent cross‑over and contamination (e.g., sticky bucketing, holdout of creators/hashtags if needed). 3) Define a primary metric (e.g., 24‑hour follow‑through rate per exposed user) and at least three guardrails (e.g., session length, violating‑hashtag follow rate, crash rate). 4) Powering: with a baseline follow rate of 4.0%, two‑sided alpha=0.05, power=0.80, and a minimum detectable effect of +5% relative, compute the required users per arm assuming user‑level randomization and intraclass correlation ρ=0.02 from repeated sessions. Show formulas and adjustments for clustering and expected sample‑ratio mismatch detection. 5) Outline bias controls: CUPED or pre‑period stratification, novelty/winner’s curse handling, sequential monitoring rules, minimum test duration, and diagnostics for interference or network effects.

Quick Answer: This question evaluates a data scientist's competency in experimental design, causal inference, and statistical power analysis for A/B tests, covering randomization unit choice, contamination prevention, metric and guardrail selection, clustering adjustments, and bias-control techniques.

|Home/Analytics & Experimentation/Meta
Meta logo
Meta
Oct 13, 2025, 9:49 PM
mediumData ScientistOnsiteAnalytics & Experimentation
3
0

You must run an A/B test to evaluate the new hashtag recommender starting on 2025‑09‑01. 1) Define the randomization unit (user/session/impression) and justify it given potential interference (users can see and follow the same hashtag across sessions). 2) Specify treatment and control selection, rollout/ramp plan, and how you will prevent cross‑over and contamination (e.g., sticky bucketing, holdout of creators/hashtags if needed). 3) Define a primary metric (e.g., 24‑hour follow‑through rate per exposed user) and at least three guardrails (e.g., session length, violating‑hashtag follow rate, crash rate). 4) Powering: with a baseline follow rate of 4.0%, two‑sided alpha=0.05, power=0.80, and a minimum detectable effect of +5% relative, compute the required users per arm assuming user‑level randomization and intraclass correlation ρ=0.02 from repeated sessions. Show formulas and adjustments for clustering and expected sample‑ratio mismatch detection. 5) Outline bias controls: CUPED or pre‑period stratification, novelty/winner’s curse handling, sequential monitoring rules, minimum test duration, and diagnostics for interference or network effects.

Loading comments...