Determine Key Metrics and Design A/B Test for Ad Ranking

Quick Overview

This interview question evaluates metric design, causal reasoning, experiment setup, diagnostics, SQL/statistical checks, and recommendations in a realistic interview setting. A strong answer for Determine Key Metrics and Design A/B Test for Ad Ranking states assumptions, handles edge cases, explains trade-offs, and shows how to validate the result clearly.

Determine Key Metrics and Design A/B Test for Ad Ranking

Company: Meta

Role: Data Scientist

Category: Analytics & Experimentation

Difficulty: hard

Interview Round: Onsite

##### Scenario Ads team is replacing a rule-based ad ranking with a new recommendation system; UI and auction rules stay the same. ##### Question Which primary and guardrail metrics would you track for this launch and why? How would you design the A/B test so control and treatment populations are truly comparable? Write the formulas you would use to estimate required sample size and test duration. If treatment CTR rises 5%, will advertisers necessarily spend more? Explain the causal path and additional analyses you would run. Below is a chart where treatment CTR is already higher than control during the pre-launch period. What is wrong with this picture and how would you fix the visualization? How would you summarize results, next steps, and risks for senior leadership? ##### Hints Think metric hierarchy, randomization, statistical power, business incentives, visualization best-practices, and executive storytelling.

Quick Answer: This interview question evaluates metric design, causal reasoning, experiment setup, diagnostics, SQL/statistical checks, and recommendations in a realistic interview setting. A strong answer for Determine Key Metrics and Design A/B Test for Ad Ranking states assumptions, handles edge cases, explains trade-offs, and shows how to validate the result clearly.

|Home/Analytics & Experimentation/Meta
Meta logo
Meta
Aug 4, 2025, 10:55 AM
hardData ScientistOnsiteAnalytics & Experimentation
5
0

Determine Key Metrics and Design A/B Test for Ad Ranking

Experiment Design: Replacing Rule-Based Ad Ranking with a Recommender

Context

You are launching a new machine-learning–based ad ranking system to replace a rule-based ranker. The UI and auction rules remain unchanged. You must plan metrics, testing, analysis, visualization, and executive communication for the launch.

Tasks

  1. Metrics
  • Define the primary success metric for the launch and the key guardrail metrics across user experience, advertiser outcomes, marketplace health, and system reliability. Explain why.
  1. A/B Test Design
  • Describe how you would randomize and analyze the experiment so control and treatment populations are truly comparable, accounting for seasonality and marketplace interference (e.g., budgets, pacing, frequency caps).
  1. Sample Size and Duration
  • Provide the formulas you would use to estimate required sample size and test duration for: (a) a proportion metric like CTR; (b) a continuous metric like revenue per user-day (or RPM). State assumptions.
  1. CTR vs Advertiser Spend
  • If treatment CTR rises 5%, will advertisers necessarily spend more? Explain the causal path and what additional analyses you would run to diagnose and forecast spend effects.
  1. Visualization Critique
  • You are given a line chart where treatment CTR is already higher than control during the pre-launch period. What is wrong with this picture? How would you fix the visualization and/or the design?
  1. Executive Summary
  • How would you summarize results, next steps, and risks for senior leadership? Provide a concise structure.

Hints

Think metric hierarchy, randomization unit and stratification, power and MDE, advertiser incentives and auction dynamics, visualization best practices, and executive storytelling.

Constraints & Assumptions

  • Preserve the scope, facts, inputs, and requested outputs from the prompt above.
  • If the prompt leaves a detail unspecified, state a reasonable assumption before relying on it.
  • Keep the answer interview-ready: concise enough to present, but concrete enough to implement or evaluate.

Clarifying Questions to Ask Guidance

  • Clarify the business objective, unit of analysis, time window, exposure definition, and primary metric.
  • State assumptions about instrumentation, randomization, sample size, and data quality.
  • Separate descriptive analysis from causal claims.

What a Strong Answer Covers Guidance

  • A metric framework with primary, guardrail, and diagnostic metrics.
  • A credible analysis or experiment design with clear assumptions and bias checks.
  • SQL/statistical logic for segmentation, variance, confidence, and data validation where relevant.
  • An actionable recommendation that explains trade-offs and next steps.

Follow-up Questions Guidance

  • What sanity checks would you run before trusting the result?
  • How would you handle novelty effects, seasonality, or selection bias?
  • What decision would you make if metrics disagree?
Loading comments...

Browse More Questions

More Analytics & Experimentation•More Meta•More Data Scientist•Meta Data Scientist•Meta Analytics & Experimentation•Data Scientist Analytics & Experimentation