Observational Causal Inference For Meta Products

What's being tested

Interviewers are checking that you can convert messy product telemetry into credible causal claims without an experiment: diagnose confounding, pick an identification strategy, implement estimators, and justify robustness. For Meta products this matters because many interventions (recommendation tweaks, ranking changes) are rolled out observationally or require quick retrospective estimates to inform experiments. Expect to be probed on reasoning, diagnostics, and tradeoffs between bias and variance.

Core knowledge

Potential outcomes framework: define causal estimands like ATE and ATT with $Y(1),Y(0)$ and estimands $E[Y(1)-Y(0)]$ ; clarify sample vs population target and SUTVA assumptions.
Confounding & DAGs: draw a causal graph (DAG) to identify backdoor paths; use the backdoor criterion to list covariates to adjust for and expose colliders to avoid.
Ignorability / Unconfoundedness: if $Y(0),Y(1)\perp T \mid X$ , then conditional adjustment works; state when this is implausible for recommendation exposures.
Overlap / Positivity: check that $0<p(T=1\mid X)<1$ across covariate strata; lack of overlap breaks IPW and matching. Trim or coarsen when extreme.
Propensity scores: estimate with scikit-learn or XGBoost; use for matching, stratification, and inverse probability weighting (IPW). IPW estimator: $\hat{ATE}_{IPW}=\frac{1}{n}\sum_i\frac{T_iY_i}{\hat{e}(X_i)}-\frac{1}{n}\sum_i\frac{(1-T_i)Y_i}{1-\hat{e}(X_i)}.$
Doubly robust (DR) estimators: combine outcome model $\hat{m}(X)$ and $\hat{e}(X)$ for protection against one model misspecification; DR reduces bias if either is correct.
Matching & balance diagnostics: use nearest-neighbor or coarsened exact matching; report covariate standardized mean differences (SMD) and variance ratios; aim for SMD < 0.1 for key covariates.
Instrumental variables (IV): when unobserved confounding exists, propose an instrument $Z$ that affects treatment but not outcome except through treatment; state exclusion and monotonicity assumptions.
Difference-in-differences (DiD) & panel methods: use when you have pre/post periods and parallel trends plausibility; add unit fixed effects or synthetic control for better comparators.
Sensitivity analysis: quantify how large an unobserved confounder would need to be to explain away effect (e.g., Rosenbaum bounds) and report E-values for transparency.
Practical telemetry & implementation: pull covariates from Postgres or logging infra, seed models using python libs like econml/causalml/DoWhy, and bootstrap for CIs; watch latency and missingness patterns.
Variance & trimming: IPW suffers from high-variance weights; trim or stabilize weights, and report effective sample size and treatment group ESS after weighting.

Worked example

Question framing (typical): "Estimate the causal effect of a new recommendation sidebar on DAU using observational logs." First 30s: ask which users saw the sidebar (treatment definition), the rollout timing, available pre-treatment covariates (engagement history, device, locale), and any business rules that assign exposure. Skeleton of an answer: (1) draw a DAG to show plausible confounders (e.g., active users more likely to be targeted); (2) propose identification: if rich pre-treatment covariates exist, do propensity-score IPW + balance checks; (3) fit a doubly robust estimator (outcome model + propensity) and bootstrap CIs; (4) run sensitivity/negative-control analyses (e.g., outcome unaffected by sidebar like signup time). Key tradeoff: a complex ML propensity model improves overlap modeling but can overfit and produce extreme weights — plan to prune or stabilize weights and report ESS. Close by saying you'd run a complementary DiD on users with pre/post data and, if possible, recommend a small randomized pilot for validation.

A second angle

Consider instead estimating the effect of a backend ranking change where assignment depends on server-side business rules and unobserved query context. Here ignorability is less plausible, so pivot to IV or natural experiments: find an instrumental variable such as random hashing that routed some traffic to the new ranking for load balancing. In 4–6 sentences: articulate instrument validity (random routing independent of user intent), estimate local average treatment effect (LATE) via two-stage least squares, and discuss monotonicity (routing never makes someone less likely to receive treatment). Emphasize falsification checks: instrument shouldn’t predict pre-treatment outcomes and should be balanced across user segments.

Common pitfalls

Pitfall: Adjusting for post-treatment variables — conditioning on mediators or variables downstream of treatment induces bias; always separate pre-treatment covariates from post-treatment measures.

Pitfall: Over-reliance on a single estimator without diagnostics — reporting an IPW point estimate without balance tables, extreme weights, or ESS will look naive; always show diagnostics and robustness checks.

Pitfall: Claiming causal language prematurely — saying "X caused Y" without discussing unobserved confounding, instrument validity, or sensitivity bounds loses credibility; qualify claims and state assumptions explicitly.

Connections

Interviewers may pivot to adjacent topics such as A/B testing (when to run an experiment vs. observational study), uplift modeling (heterogeneous treatment effects and targeting), or causal discovery (using time-series or DAG search for variable selection). Be ready to switch between identification strategies and pragmatic experimental solutions.