Evaluate New Ad Model with A/B Testing Experiment

Quick Overview

This interview question evaluates metric design, causal reasoning, experiment setup, diagnostics, SQL/statistical checks, and recommendations in a realistic interview setting. A strong answer for Evaluate New Ad Model with A/B Testing Experiment states assumptions, handles edge cases, explains trade-offs, and shows how to validate the result clearly.

Evaluate New Ad Model with A/B Testing Experiment

Company: Meta

Role: Data Scientist

Category: Analytics & Experimentation

Difficulty: hard

Interview Round: Onsite

##### Scenario As part of the Ads team you have trained a new ad-recommendation model and need to decide if it should replace the current model. ##### Question Design an experiment to evaluate the new recommendation model against the incumbent. What primary and guardrail metrics would you track? How would these differ when presenting results to the CFO versus the CGO (growth)? State your final recommendation framework for launch or rollback. ##### Hints Cover experiment design (A/B, traffic split, duration), metrics (revenue, CTR, ROI, user retention), statistical significance and trade-offs for finance vs growth.

Quick Answer: This interview question evaluates metric design, causal reasoning, experiment setup, diagnostics, SQL/statistical checks, and recommendations in a realistic interview setting. A strong answer for Evaluate New Ad Model with A/B Testing Experiment states assumptions, handles edge cases, explains trade-offs, and shows how to validate the result clearly.

|Home/Analytics & Experimentation/Meta
Meta logo
Meta
Aug 4, 2025, 10:55 AM
hardData ScientistOnsiteAnalytics & Experimentation
2
0

Evaluate New Ad Model with A/B Testing Experiment

Evaluate a New Ads Recommendation Model via Online Experimentation

Scenario

You have trained a new ad-recommendation model and must decide whether it should replace the incumbent model that currently ranks/serves ads in a large-scale, auction-based ads system.

Task

Design an experiment to evaluate the new model against the incumbent and answer:

  1. Experiment design: unit of randomization, traffic split/ramp, duration, and how to handle auction/marketplace interference.
  2. Metrics: define primary success metric(s) and guardrails (revenue, CTR, ROI, user retention, latency, etc.).
  3. Stakeholder readouts: how the story and metrics differ for a CFO vs a CGO (growth).
  4. Decision framework: launch or rollback criteria, with monitoring and risk mitigation.

Requirements

  • Cover A/B design, traffic allocation, expected duration, statistical significance/power, and trade-offs between finance and growth.
  • Include clear metric definitions and assumptions where needed.

Constraints & Assumptions

  • Preserve the scope, facts, inputs, and requested outputs from the prompt above.
  • If the prompt leaves a detail unspecified, state a reasonable assumption before relying on it.
  • Keep the answer interview-ready: concise enough to present, but concrete enough to implement or evaluate.

Clarifying Questions to Ask Guidance

  • Clarify the business objective, unit of analysis, time window, exposure definition, and primary metric.
  • State assumptions about instrumentation, randomization, sample size, and data quality.
  • Separate descriptive analysis from causal claims.

What a Strong Answer Covers Guidance

  • A metric framework with primary, guardrail, and diagnostic metrics.
  • A credible analysis or experiment design with clear assumptions and bias checks.
  • SQL/statistical logic for segmentation, variance, confidence, and data validation where relevant.
  • An actionable recommendation that explains trade-offs and next steps.

Follow-up Questions Guidance

  • What sanity checks would you run before trusting the result?
  • How would you handle novelty effects, seasonality, or selection bias?
  • What decision would you make if metrics disagree?
Loading comments...

Browse More Questions

More Analytics & Experimentation•More Meta•More Data Scientist•Meta Data Scientist•Meta Analytics & Experimentation•Data Scientist Analytics & Experimentation