Answer the following two analytics interview prompts. ### Constraints & Assumptions - For causal impact, separate prediction from causal identification. - For the experiment, use by-hand calculations that are reasonable in an interview and state uncertainty. - Do not optimize one metric blindly if revenue, refunds, retention, latency, or long-term value matter. - Explain how you would forecast production performance after launch, including uncertainty and regression to the mean. ### Clarifying Questions to Ask - For the causal-impact case, what treatment, population, rollout timing, and data history are available? - Why exactly is a randomized experiment infeasible? - What is the decision that will be made from the causal estimate? - For the A/B/C test, was the metric pre-registered and are there guardrails? - Is the traffic mix in the experiment representative of production? ### Part 1 - Measure Causal Impact Without An Experiment Describe a real or hypothetical product, model, or policy change where the business wants to measure impact, but a randomized experiment cannot be launched. Explain the treatment, unit, population, metric, identification approach, assumptions, bias risks, validation, uncertainty, and short-term versus long-term impact. #### What This Part Should Cover - Clear estimand such as ATE or ATT. - Why randomization is infeasible. - Appropriate quasi-experimental method: difference-in-differences, synthetic control, matching, inverse propensity weighting, doubly robust estimation, interrupted time series, regression discontinuity, IV, or ML counterfactual. - Identification assumptions, bias/confounding risks, validation tests, uncertainty intervals, and long-term outcome tracking. ### Part 2 - Analyze A Three-Variant Experiment You run a 3-arm experiment to maximize CTP, where CTP equals purchases divided by visits: - Variant A: 150 visits, 43 purchases - Variant B: 200 visits, 48 purchases - Variant C: 100 visits, 15 purchases Which variant is currently winning? Show a by-hand statistical analysis using confidence intervals or hypothesis tests. #### What This Part Should Cover - Point estimates for A, B, and C. - Approximate confidence intervals for proportions. - Direct comparison between A and B, noting overlap and uncertainty. - Multiple-testing and power caveats. ### Part 3 - Include Additional Metrics How would your recommendation change if revenue per visitor, average order value, refund rate, retention, or latency also matter? #### What This Part Should Cover - Primary metric, business-value metric, and guardrails. - Expected value per visitor or profit per visitor when relevant. - Decision rule based on pre-registered priorities, not only highest CTP. ### Part 4 - Forecast Future Production CTP If one variant is launched, how would you predict its future CTP in production? #### What This Part Should Cover - Observed rate as a starting point. - Uncertainty from finite sample size, winner's curse, traffic mix, seasonality, novelty effects, and regression to the mean. - Bayesian beta-binomial, empirical Bayes shrinkage, holdout monitoring, and time-aware adjustment. ### What a Strong Answer Covers - Defines causal estimands and assumptions explicitly. - Performs reasonable proportion math and avoids overclaiming significance. - Balances business metrics and guardrails. - Forecasts production performance with uncertainty rather than point-estimate optimism. ### Follow-up Questions - What placebo test would you run for your causal design? - What if parallel trends fail? - Should C be stopped early? - How would you adjust for multiple comparisons? - What would make you choose B over A despite lower CTP?