Design a feed ads A/B test with guardrails

Q: Design a feed ads A/B test with guardrails

This question evaluates a data scientist's competency in experimental design, causal inference, statistical power analysis, and product-metric trade-offs within the Analytics & Experimentation domain, and is commonly asked to assess the ability to increase monetization while protecting user and creator experience.

Q: How do I approach Analytics & Experimentation interview questions?

Analytics & Experimentation questions require understanding of core concepts and practice. PracHub provides solutions with explanations to help you master analytics & experimentation interviews.

Q: What difficulty level is this interview question?

This is a hard difficulty Analytics & Experimentation question, commonly asked during Onsite rounds at Meta.

Q: What role is this question designed for?

This question is commonly asked for Data Scientist candidates at Meta during technical interviews.

Question

Experiment Design: Insert One Extra Ad Every 8 Organic Posts in Main Feed

Context

You want to increase ad load by inserting one additional ad for every 8 organic posts in the main feed. The goal is to evaluate impact on revenue while protecting user experience and creator ecosystem health.

Tasks

(a) Choose the unit of randomization (user-level vs session-level) and justify.

(b) Define primary metrics (e.g., ad revenue per user) and guardrail metrics (e.g., session length, 7-day retention, creator impressions, complaint rate), including directionality and decision thresholds.

(c) Perform a power analysis given a +2.0% MDE on revenue with a 0.8 SD at user-day granularity. State assumptions and alternative interpretations if needed. Include variance reduction methods.

(d) Specify SRM checks and how you would detect and handle “novelty effects” of the new ad (early-time vs steady-state responses).

(e) Propose a ramp strategy and the use of geographic/age holdouts.

(f) Plan a long-term holdback to detect delayed churn.

Additionally: Address possible network spillovers (e.g., users share screenshots of ads) and how to avoid biased estimates from dynamic ranking reacting to the treatment.

Design a feed ads A/B test with guardrails

Quick Overview

Design a feed ads A/B test with guardrails

Experiment Design: Insert One Extra Ad Every 8 Organic Posts in Main Feed

Context

Tasks

Write your answer