You must ship a News Feed ranking change where content produced by treated users can be seen by control users, creating interference and within-user correlation across sessions. Only logged-in traffic is eligible. Constraints: max 10% concurrent treatment ramp; analysis window is 14 days; primary metric is sessions per user; guardrails include crash rate and time spent; sequential looks every 2 days are required by policy. Design and analysis questions: 1) Choose the experimental unit and randomization scheme to mitigate interference (e.g., graph/ego-network clustering, producer-side randomization, or two-stage randomized encouragement). Justify your choice given a 10% cap and highly skewed degree distribution. 2) Define precise estimands: direct effect on consumers, spillover effect, and total effect. Specify exposure conditions and an exposure model that makes your estimands identifiable. 3) Describe the estimator and variance: show how you would compute point estimates and 95% CIs with cluster-robust or randomization-inference SEs. State assumptions explicitly and how violations change interpretation. 4) Quantify design effect and sample size under clustering: derive the required N given average cluster size m = 120 and ICC ρ = 0.02; show n_eff = n / (1 + (m−1)ρ) and solve for n to achieve power 80% to detect a 0.6% relative lift when baseline mean is 4.0 sessions/user with SD 5.5. 5) Diagnostics: propose at least three concrete checks that detect leakage/interference (e.g., cut-edge exposure rates, treated-producer content share in control, placebo effects on non-exposed control users) and how you’d react to each. 6) Sequential testing: pick a method (e.g., alpha-spending O’Brien–Fleming or always-valid tests) and outline stopping/decision rules that control Type I error across interim looks. 7) If leadership mandates user-level 50/50 randomization (no clustering), propose post-hoc adjustments to bound bias and obtain conservative CIs (e.g., exposure-weighted IV, CUPED with pre-period, sensitivity bounds).

This question evaluates proficiency in experimental design and causal inference for online A/B testing under interference, covering competencies such as defining estimands and exposure models, handling clustered or dependent data, variance estimation and confidence intervals, power and sample-size calculations, sequential testing, and diagnostics for leakage. It is commonly asked in Analytics & Experimentation interviews because it probes practical application of statistical design and analysis in production-constrained settings, examines understanding of bias introduced by cross-group interference and operational constraints, and falls within the Analytics & Experimentation domain with a primary emphasis on practical application complemented by conceptual understanding.

How do I approach Analytics & Experimentation interview questions?

Analytics & Experimentation questions require understanding of core concepts and practice. PracHub provides solutions with explanations to help you master analytics & experimentation interviews.

What difficulty level is this interview question?

This is a Medium difficulty Analytics & Experimentation question, commonly asked during Technical Screen rounds at Meta.

What role is this question designed for?

This question is commonly asked for Data Scientist candidates at Meta during technical interviews.

Design and analyze A/B test with interference

You must ship a News Feed ranking change where content produced by treated users can be seen by control users, creating interference and within-user correlation across sessions. Only logged-in traffic is eligible. Constraints: max 10% concurrent treatment ramp; analysis window is 14 days; primary metric is sessions per user; guardrails include crash rate and time spent; sequential looks every 2 days are required by policy. Design and analysis questions:

Choose the experimental unit and randomization scheme to mitigate interference (e.g., graph/ego-network clustering, producer-side randomization, or two-stage randomized encouragement). Justify your choice given a 10% cap and highly skewed degree distribution.
Define precise estimands: direct effect on consumers, spillover effect, and total effect. Specify exposure conditions and an exposure model that makes your estimands identifiable.
Describe the estimator and variance: show how you would compute point estimates and 95% CIs with cluster-robust or randomization-inference SEs. State assumptions explicitly and how violations change interpretation.
Quantify design effect and sample size under clustering: derive the required N given average cluster size m = 120 and ICC ρ = 0.02; show n_eff = n / (1 + (m−1)ρ) and solve for n to achieve power 80% to detect a 0.6% relative lift when baseline mean is 4.0 sessions/user with SD 5.5.
Diagnostics: propose at least three concrete checks that detect leakage/interference (e.g., cut-edge exposure rates, treated-producer content share in control, placebo effects on non-exposed control users) and how you’d react to each.
Sequential testing: pick a method (e.g., alpha-spending O’Brien–Fleming or always-valid tests) and outline stopping/decision rules that control Type I error across interim looks.
If leadership mandates user-level 50/50 randomization (no clustering), propose post-hoc adjustments to bound bias and obtain conservative CIs (e.g., exposure-weighted IV, CUPED with pre-period, sensitivity bounds).

Choose the experimental unit and randomization scheme to mitigate interference (e.g., graph/ego-network clustering, producer-side randomization, or two-stage randomized encouragement). Justify your choice given a 10% cap and highly skewed degree distribution.
Define precise estimands: direct effect on consumers, spillover effect, and total effect. Specify exposure conditions and an exposure model that makes your estimands identifiable.
Describe the estimator and variance: show how you would compute point estimates and 95% CIs with cluster-robust or randomization-inference SEs. State assumptions explicitly and how violations change interpretation.
Quantify design effect and sample size under clustering: derive the required N given average cluster size m = 120 and ICC ρ = 0.02; show n_eff = n / (1 + (m−1)ρ) and solve for n to achieve power 80% to detect a 0.6% relative lift when baseline mean is 4.0 sessions/user with SD 5.5.
Diagnostics: propose at least three concrete checks that detect leakage/interference (e.g., cut-edge exposure rates, treated-producer content share in control, placebo effects on non-exposed control users) and how you’d react to each.
Sequential testing: pick a method (e.g., alpha-spending O’Brien–Fleming or always-valid tests) and outline stopping/decision rules that control Type I error across interim looks.
If leadership mandates user-level 50/50 randomization (no clustering), propose post-hoc adjustments to bound bias and obtain conservative CIs (e.g., exposure-weighted IV, CUPED with pre-period, sensitivity bounds).

Design and analyze A/B test with interference

Quick Overview

Submit Your Answer

Design and analyze A/B test with interference

Quick Overview

Submit Your Answer