Choose alternatives when randomization fails

Q: Choose alternatives when randomization fails

This question evaluates proficiency in causal inference and experimentation analytics for a Data Scientist role, focusing on selecting appropriate identification strategies, designing placebo and pre‑trend tests, and specifying robustness and sensitivity analyses for non‑randomized feature adoption.

Q: How do I approach Analytics & Experimentation interview questions?

Analytics & Experimentation questions require understanding of core concepts and practice. PracHub provides solutions with explanations to help you master analytics & experimentation interviews.

Q: What difficulty level is this interview question?

This is a hard difficulty Analytics & Experimentation question, commonly asked during Onsite rounds at Meta.

Q: What role is this question designed for?

This question is commonly asked for Data Scientist candidates at Meta during technical interviews.

Question

Causal Impact of an Autoloaded Feature Without Clean Randomization

Context

You need to estimate the causal effect of a new autoloaded feature that is only enabled for opted‑in power users. Because eligibility is restricted and adoption is not randomized, standard A/B testing is not feasible. You have user‑level panel data (pre/post outcomes, eligibility, adoption timing, and rich covariates like app version, device, region, prior engagement).

Tasks

(a) Choose one approach among Difference‑in‑Differences (DiD), Synthetic Control, Instrumental Variables (IV), or Propensity Score Matching (PSM). Justify the choice and clearly state its identifying assumptions.

(b) Design placebo and pre‑trend tests and specify robustness checks (e.g., varying windows, alternative control groups/covariates).

(c) Propose sensitivity analyses (e.g., negative controls, Rosenbaum bounds or related confounding robustness analyses).

(d) Explain the likely directional bias risks relative to a randomized A/B test, and why.

Choose alternatives when randomization fails

Quick Overview

Choose alternatives when randomization fails

Causal Impact of an Autoloaded Feature Without Clean Randomization

Context

Tasks

Write your answer