How would you measure causal impact?

Answer the following two analytics interview prompts. 1. **Causal impact without an experiment** Describe a real or hypothetical product, model, or policy change where the business wants to measure impact, but a randomized experiment cannot be launched because of operational, legal, ethical, network-effect, or rollout constraints. Explain: - the treatment, unit of analysis, target population, and primary success metric(s) - why an experiment is infeasible - which causal inference approach you would use (for example: difference-in-differences, synthetic control, matching, inverse propensity weighting, doubly robust estimation, interrupted time series, instrumental variables, regression discontinuity, or an ML-based counterfactual model) - the assumptions required for identification - potential sources of bias or confounding - how you would validate the method and quantify uncertainty - how you would separate short-term impact from long-term impact - why you chose this approach instead of other seemingly simpler methods 2. **Three-variant experiment and forecasting future conversion** You run a 3-arm experiment to maximize **CTP (purchase rate = purchases / visits)**. The observed results are: - Variant A: 150 visits, 43 purchases - Variant B: 200 visits, 48 purchases - Variant C: 100 visits, 15 purchases Answer the following: - Which variant is currently winning? - Show a reasonable by-hand statistical analysis using confidence intervals or hypothesis tests. - How would your recommendation change if additional metrics also matter, such as revenue per visitor, average order value, refund rate, retention, or latency? - If one variant is launched, how would you predict its future CTP in production, accounting for uncertainty and possible traffic or seasonality shifts?