How do I approach Analytics & Experimentation interview questions?

Analytics & Experimentation questions require understanding of core concepts and practice. PracHub provides solutions with explanations to help you master analytics & experimentation interviews.

What difficulty level is this interview question?

This is a hard difficulty Analytics & Experimentation question, commonly asked during Onsite rounds at Meta.

What role is this question designed for?

This question is commonly asked for Data Scientist candidates at Meta during technical interviews.

Determine Key Metrics and Design A/B Test for Ad Ranking

Q: Determine Key Metrics and Design A/B Test for Ad Ranking

This question evaluates experimental design, metric definition, causal inference, power analysis, visualization critique, and executive communication competencies within Analytics & Experimentation for a data scientist role.

Experiment Design: Replacing Rule-Based Ad Ranking with a Recommender

Context

You are launching a new machine-learning–based ad ranking system to replace a rule-based ranker. The UI and auction rules remain unchanged. You must plan metrics, testing, analysis, visualization, and executive communication for the launch.

Tasks

Metrics

Define the primary success metric for the launch and the key guardrail metrics across user experience, advertiser outcomes, marketplace health, and system reliability. Explain why.

A/B Test Design

Describe how you would randomize and analyze the experiment so control and treatment populations are truly comparable, accounting for seasonality and marketplace interference (e.g., budgets, pacing, frequency caps).

Sample Size and Duration

Provide the formulas you would use to estimate required sample size and test duration for: (a) a proportion metric like CTR; (b) a continuous metric like revenue per user-day (or RPM). State assumptions.

CTR vs Advertiser Spend

If treatment CTR rises 5%, will advertisers necessarily spend more? Explain the causal path and what additional analyses you would run to diagnose and forecast spend effects.

Visualization Critique

You are given a line chart where treatment CTR is already higher than control during the pre-launch period. What is wrong with this picture? How would you fix the visualization and/or the design?

Executive Summary

How would you summarize results, next steps, and risks for senior leadership? Provide a concise structure.

Hints

Think metric hierarchy, randomization unit and stratification, power and MDE, advertiser incentives and auction dynamics, visualization best practices, and executive storytelling.

Tasks

Metrics

Define the primary success metric for the launch and the key guardrail metrics across user experience, advertiser outcomes, marketplace health, and system reliability. Explain why.

A/B Test Design

Describe how you would randomize and analyze the experiment so control and treatment populations are truly comparable, accounting for seasonality and marketplace interference (e.g., budgets, pacing, frequency caps).

Sample Size and Duration

Provide the formulas you would use to estimate required sample size and test duration for: (a) a proportion metric like CTR; (b) a continuous metric like revenue per user-day (or RPM). State assumptions.

CTR vs Advertiser Spend

If treatment CTR rises 5%, will advertisers necessarily spend more? Explain the causal path and what additional analyses you would run to diagnose and forecast spend effects.

Visualization Critique

You are given a line chart where treatment CTR is already higher than control during the pre-launch period. What is wrong with this picture? How would you fix the visualization and/or the design?

Executive Summary

How would you summarize results, next steps, and risks for senior leadership? Provide a concise structure.

Determine Key Metrics and Design A/B Test for Ad Ranking

Quick Overview

Experiment Design: Replacing Rule-Based Ad Ranking with a Recommender

Context

Tasks

Hints

Solution

Submit Your Answer to Earn 20XP

Determine Key Metrics and Design A/B Test for Ad Ranking

Quick Overview

Experiment Design: Replacing Rule-Based Ad Ranking with a Recommender

Context

Tasks

Hints

Solution

Submit Your Answer to Earn 20XP