Estimate impact without experiments and pick variant
Quick Overview
This question evaluates causal inference and experimental-analysis competencies in Analytics & Experimentation and Data Science, covering observational estimands, causal identification assumptions and biases, uncertainty quantification for A/B/C tests, multiple-comparisons reasoning, and post-launch forecasting and monitoring.
Part A — Measuring impact when you cannot run an experiment
You are a Staff Data Scientist working on a product change (feature/policy/model update). Stakeholders want to measure causal impact (incremental lift) of the change, but you cannot launch a randomized experiment (e.g., legal constraints, all users must receive the change, platform limitations, or risk).
Task:
- Propose an end-to-end approach to estimate the causal impact of the change using observational data (you may use an ML-based counterfactual if appropriate).
-
Clearly state:
- The estimand (e.g., ATE, ATT, incremental purchases per user/day).
- Key assumptions required for causal identification.
- Major biases/failure modes (confounding, selection bias, interference, data drift, novelty effects, etc.).
- How you would validate the approach (placebo tests, negative controls, sensitivity analysis, backtesting).
- Explain how you would reason about short-term vs. long-term impact , and what additional data or modeling you would need.
Part B — 3-variant experiment and forecasting post-launch conversion
You ran an online experiment with three variants (A/B/C). The goal is to maximize CTP (purchase rate), defined as:
Observed results (assume visits are independent Bernoulli trials; one purchase at most per visit):
- Variant A: 150 visits, 43 purchases
- Variant B: 200 visits, 48 purchases
- Variant C: 100 visits, 15 purchases
Questions:
-
Which variant is “winning”?
- Provide point estimates of CTP.
- Quantify uncertainty (e.g., confidence/credible intervals).
- Address multiple comparisons / decision criteria if needed.
-
Suppose you launch the chosen variant to 100% traffic. How would you
predict the future CTP
after launch?
- Describe a statistical approach to generate a forecast (and interval).
- List key factors that may cause post-launch CTP to differ from experiment CTP (traffic mix shift, seasonality, ramp-up effects, novelty, instrumentation changes, etc.).
- Mention how you would monitor/validate the forecast after launch (guardrails, alerting, recalibration).