How would you measure causal impact?

Quick Overview

Answer causal-impact and three-variant experiment prompts with clear estimands, quasi-experimental assumptions, validation, uncertainty, by-hand CTP intervals, multiple-comparison caveats, business guardrails, and Bayesian forecasting for production conversion.

How would you measure causal impact?

Company: Upstart

Role: Data Scientist

Category: Analytics & Experimentation

Difficulty: medium

Interview Round: Technical Screen

Answer the following two analytics interview prompts. ### Constraints & Assumptions - For causal impact, separate prediction from causal identification. - For the experiment, use by-hand calculations that are reasonable in an interview and state uncertainty. - Do not optimize one metric blindly if revenue, refunds, retention, latency, or long-term value matter. - Explain how you would forecast production performance after launch, including uncertainty and regression to the mean. ### Clarifying Questions to Ask - For the causal-impact case, what treatment, population, rollout timing, and data history are available? - Why exactly is a randomized experiment infeasible? - What is the decision that will be made from the causal estimate? - For the A/B/C test, was the metric pre-registered and are there guardrails? - Is the traffic mix in the experiment representative of production? ### Part 1 - Measure Causal Impact Without An Experiment Describe a real or hypothetical product, model, or policy change where the business wants to measure impact, but a randomized experiment cannot be launched. Explain the treatment, unit, population, metric, identification approach, assumptions, bias risks, validation, uncertainty, and short-term versus long-term impact. #### What This Part Should Cover - Clear estimand such as ATE or ATT. - Why randomization is infeasible. - Appropriate quasi-experimental method: difference-in-differences, synthetic control, matching, inverse propensity weighting, doubly robust estimation, interrupted time series, regression discontinuity, IV, or ML counterfactual. - Identification assumptions, bias/confounding risks, validation tests, uncertainty intervals, and long-term outcome tracking. ### Part 2 - Analyze A Three-Variant Experiment You run a 3-arm experiment to maximize CTP, where CTP equals purchases divided by visits: - Variant A: 150 visits, 43 purchases - Variant B: 200 visits, 48 purchases - Variant C: 100 visits, 15 purchases Which variant is currently winning? Show a by-hand statistical analysis using confidence intervals or hypothesis tests. #### What This Part Should Cover - Point estimates for A, B, and C. - Approximate confidence intervals for proportions. - Direct comparison between A and B, noting overlap and uncertainty. - Multiple-testing and power caveats. ### Part 3 - Include Additional Metrics How would your recommendation change if revenue per visitor, average order value, refund rate, retention, or latency also matter? #### What This Part Should Cover - Primary metric, business-value metric, and guardrails. - Expected value per visitor or profit per visitor when relevant. - Decision rule based on pre-registered priorities, not only highest CTP. ### Part 4 - Forecast Future Production CTP If one variant is launched, how would you predict its future CTP in production? #### What This Part Should Cover - Observed rate as a starting point. - Uncertainty from finite sample size, winner's curse, traffic mix, seasonality, novelty effects, and regression to the mean. - Bayesian beta-binomial, empirical Bayes shrinkage, holdout monitoring, and time-aware adjustment. ### What a Strong Answer Covers - Defines causal estimands and assumptions explicitly. - Performs reasonable proportion math and avoids overclaiming significance. - Balances business metrics and guardrails. - Forecasts production performance with uncertainty rather than point-estimate optimism. ### Follow-up Questions - What placebo test would you run for your causal design? - What if parallel trends fail? - Should C be stopped early? - How would you adjust for multiple comparisons? - What would make you choose B over A despite lower CTP?

Overview: Answer causal-impact and three-variant experiment prompts with clear estimands, quasi-experimental assumptions, validation, uncertainty, by-hand CTP intervals, multiple-comparison caveats, business guardrails, and Bayesian forecasting for production conversion.

Read the full Upstart Data Scientist interview experience this question came from

Upstart logo
Upstart
Dec 11, 2024
mediumData ScientistTechnical ScreenAnalytics & Experimentation
4
0

Answer the following two analytics interview prompts.

Constraints & Assumptions

  • For causal impact, separate prediction from causal identification.
  • For the experiment, use by-hand calculations that are reasonable in an interview and state uncertainty.
  • Do not optimize one metric blindly if revenue, refunds, retention, latency, or long-term value matter.
  • Explain how you would forecast production performance after launch, including uncertainty and regression to the mean.

Clarifying Questions to Ask Guidance

  • For the causal-impact case, what treatment, population, rollout timing, and data history are available?
  • Why exactly is a randomized experiment infeasible?
  • What is the decision that will be made from the causal estimate?
  • For the A/B/C test, was the metric pre-registered and are there guardrails?
  • Is the traffic mix in the experiment representative of production?

Part 1 - Measure Causal Impact Without An Experiment

Describe a real or hypothetical product, model, or policy change where the business wants to measure impact, but a randomized experiment cannot be launched. Explain the treatment, unit, population, metric, identification approach, assumptions, bias risks, validation, uncertainty, and short-term versus long-term impact.

What This Part Should Cover Guidance

  • Clear estimand such as ATE or ATT.
  • Why randomization is infeasible.
  • Appropriate quasi-experimental method: difference-in-differences, synthetic control, matching, inverse propensity weighting, doubly robust estimation, interrupted time series, regression discontinuity, IV, or ML counterfactual.
  • Identification assumptions, bias/confounding risks, validation tests, uncertainty intervals, and long-term outcome tracking.

Part 2 - Analyze A Three-Variant Experiment

You run a 3-arm experiment to maximize CTP, where CTP equals purchases divided by visits:

  • Variant A: 150 visits, 43 purchases
  • Variant B: 200 visits, 48 purchases
  • Variant C: 100 visits, 15 purchases

Which variant is currently winning? Show a by-hand statistical analysis using confidence intervals or hypothesis tests.

What This Part Should Cover Guidance

  • Point estimates for A, B, and C.
  • Approximate confidence intervals for proportions.
  • Direct comparison between A and B, noting overlap and uncertainty.
  • Multiple-testing and power caveats.

Part 3 - Include Additional Metrics

How would your recommendation change if revenue per visitor, average order value, refund rate, retention, or latency also matter?

What This Part Should Cover Guidance

  • Primary metric, business-value metric, and guardrails.
  • Expected value per visitor or profit per visitor when relevant.
  • Decision rule based on pre-registered priorities, not only highest CTP.

Part 4 - Forecast Future Production CTP

If one variant is launched, how would you predict its future CTP in production?

What This Part Should Cover Guidance

  • Observed rate as a starting point.
  • Uncertainty from finite sample size, winner's curse, traffic mix, seasonality, novelty effects, and regression to the mean.
  • Bayesian beta-binomial, empirical Bayes shrinkage, holdout monitoring, and time-aware adjustment.

What a Strong Answer Covers Guidance

  • Defines causal estimands and assumptions explicitly.
  • Performs reasonable proportion math and avoids overclaiming significance.
  • Balances business metrics and guardrails.
  • Forecasts production performance with uncertainty rather than point-estimate optimism.

Follow-up Questions Guidance

  • What placebo test would you run for your causal design?
  • What if parallel trends fail?
  • Should C be stopped early?
  • How would you adjust for multiple comparisons?
  • What would make you choose B over A despite lower CTP?