How do I approach Analytics & Experimentation interview questions?

Analytics & Experimentation questions require understanding of core concepts and practice. PracHub provides solutions with explanations to help you master analytics & experimentation interviews.

What difficulty level is this interview question?

This is a hard difficulty Analytics & Experimentation question, commonly asked during Onsite rounds at Meta.

What role is this question designed for?

This question is commonly asked for Data Scientist candidates at Meta during technical interviews.

Design and analyze an A/B test | Meta Interview Question

Q: Design and analyze an A/B test

This question evaluates a candidate's ability to design and analyze randomized experiments, covering statistical power and sample-size calculation, cluster-robust variance adjustments, covariate adjustment (CUPED), hypothesis and guardrail specification, ramping and monitoring strategies, and causal inference methods such as geo-level difference-in-differences, and is situated in the Analytics & Experimentation domain for data scientist roles. It is commonly asked to assess practical application of experimental statistics and operational decision-making under real-world constraints—balancing statistical rigor, multiple-testing control, and issues like repeat users, seasonality, and delayed attribution—requiring both conceptual understanding of causal inference and hands-on analytical application.

Experiment Design: Proximity-Weighted Search Ranking A/B Test

You are designing a 14-day, 50/50 user-level randomized A/B test for a marketplace's search ranking change that favors nearby merchants. Daily DAU ≈ 200,000. Baseline conversion p0 = 0.12. Expected relative lift = +5% on conversion. Guardrails: cancellation rate must not increase by > 0.2 percentage points (pp), and average delivery time must not worsen by > 3 minutes.

Answer the following:

(a) Compute the minimum per-arm sample size for conversion using a two-sided z-test with α = 0.05 and power = 0.80. Show formulas and numeric steps assuming independent Bernoulli trials. Then discuss how user clustering and repeat sessions/orders would inflate variance and how to correct (e.g., variance inflation factor via empirical design effect or cluster-robust SEs).

(b) Specify primary, secondary, and guardrail metrics; pre-register hypotheses; and define the decision rule combining effect size and statistical significance (include MDE and non-inferiority thresholds for guardrails).

(c) Describe a CUPED or pre-period covariate adjustment using user-level 28-day pretest conversion propensity; provide the adjusted estimator and how you would validate variance reduction (A/A test, placebo checks).

(d) Outline a ramp plan (1% → 10% → 50% → 100%), novelty and learning effects monitoring, weekday/seasonality controls, and how to handle attribution and tracking delays (48h late events).

(e) If the test is geo-split (city-level) instead of user-split, propose a difference-in-differences setup with city fixed effects and calendar effects; list assumptions and how you’d test for pre-trend balance.

(f) Explain how you will monitor and correct for peeking and multiple metrics (alpha-spending, O’Brien–Fleming; FDR for many guardrails) and define a rollback plan.

Experiment Design: Proximity-Weighted Search Ranking A/B Test

Answer the following:

(d) Outline a ramp plan (1% → 10% → 50% → 100%), novelty and learning effects monitoring, weekday/seasonality controls, and how to handle attribution and tracking delays (48h late events).

(f) Explain how you will monitor and correct for peeking and multiple metrics (alpha-spending, O’Brien–Fleming; FDR for many guardrails) and define a rollback plan.

Design and analyze an A/B test

Quick Overview

Experiment Design: Proximity-Weighted Search Ranking A/B Test

Solution

Submit Your Answer to Earn 20XP

Design and analyze an A/B test

Quick Overview

Experiment Design: Proximity-Weighted Search Ranking A/B Test

Solution

Submit Your Answer to Earn 20XP