How do I approach Analytics & Experimentation interview questions?

Analytics & Experimentation questions require understanding of core concepts and practice. PracHub provides solutions with explanations to help you master analytics & experimentation interviews.

What difficulty level is this interview question?

This is a medium difficulty Analytics & Experimentation question, commonly asked during Technical Screen rounds at Meta.

What role is this question designed for?

This question is commonly asked for Data Scientist candidates at Meta during technical interviews.

Design Experiment to Evaluate New Video-Ad Effectiveness

Last updated: Mar 29, 2026

Quick Overview

This question evaluates competency in experimental design and causal inference, including randomized assignment, metric definition, power and sample-size reasoning, interference management in auctioned ad ecosystems, and interpretation of sequential tests and rollout criteria.

Experiment Design: Measuring the Impact of a New Video‑Ad Format

Context

You are designing an experiment for a large consumer app that runs an ad auction. A new video‑ad format (UI/creative/interaction changes) is being considered. The goal is to estimate its causal impact on advertiser effectiveness and revenue while protecting user experience, accounting for potential interference in the ad marketplace.

Task

Design an experiment to evaluate the new format. Specify:

Unit of randomization (and why), including options (A/B vs multivariate, user vs impression vs geo) and how to manage auction/interference.
Primary and secondary metrics (with definitions and guardrails).
Sample size and power considerations (including variance reduction, clustering, and sequential testing).
Success criteria and a rollout plan.
If the primary metric shows no statistically significant improvement, list follow‑up analyses and alternative actions.

Solution

Show

Comments (0)

Loading comments...

Browse More Questions

More Analytics & Experimentation•More Meta•More Data Scientist•Meta Data Scientist•Meta Analytics & Experimentation•Data Scientist Analytics & Experimentation