Reduce variance with covariate adjustment

Quick Overview

This question evaluates a data scientist's competency in experimental design, variance reduction techniques, and regression-based covariate adjustment (CUPED/ANCOVA) within randomized A/B tests, including understanding of statistical power and diagnostic checks.

Reduce variance with covariate adjustment

Company: Meta

Role: Data Scientist

Category: Analytics & Experimentation

Difficulty: medium

Interview Round: Onsite

A pre‑period covariate X explains 36% of the variance in outcome Y. a) Derive how CUPED/regression adjustment changes Var(Δ) and quantify the expected sample‑size savings. b) List assumptions that invalidate CUPED (e.g., post‑treatment leakage, mis‑timed covariates) and diagnostics you’d run. c) Would you prefer stratified randomization or covariate adjustment here, and why?

Quick Answer: This question evaluates a data scientist's competency in experimental design, variance reduction techniques, and regression-based covariate adjustment (CUPED/ANCOVA) within randomized A/B tests, including understanding of statistical power and diagnostic checks.

|Home/Analytics & Experimentation/Meta
Meta logo
Meta
Oct 13, 2025, 9:49 PM
mediumData ScientistOnsiteAnalytics & Experimentation
4
0

Experiment Design and CUPED/Regression Adjustment

You are running a randomized A/B test with outcome Y. You also have a pre-period covariate X (measured before any treatment exposure) that explains 36% of the variance in Y (i.e., R² = 0.36) when regressing Y on X.

Answer the following:

(a) Derive how CUPED (a.k.a. regression/ANCOVA adjustment using a pre-period covariate) changes the variance of the treatment effect estimator Δ, and quantify the expected sample-size savings and MDE change given R² = 0.36.

(b) List assumptions and failure modes that would invalidate or undermine CUPED (e.g., post‑treatment leakage, mis‑timed covariates), and diagnostics you would run to detect problems.

(c) For this setting, would you prefer stratified randomization or covariate adjustment, and why?

Loading comments...

Browse More Questions

More Analytics & Experimentation•More Meta•More Data Scientist•Meta Data Scientist•Meta Analytics & Experimentation•Data Scientist Analytics & Experimentation