Evaluate impact without randomized experiments

Quick Overview

This question evaluates a data scientist's competence in causal inference and identification strategies for estimating promotion effects from observational longitudinal data, emphasizing challenges such as staggered rollout, interference (spillovers), seasonality, and uncertainty quantification.

Evaluate impact without randomized experiments

Company: Uber

Role: Data Scientist

Category: Analytics & Experimentation

Difficulty: hard

Interview Round: Technical Screen

If you cannot run an experiment, propose and justify at least three identification strategies to estimate the promotion’s causal effect, including assumptions, diagnostics, and failure modes: (1) Difference‑in‑Differences/event‑study with staggered rollout using modern estimators (avoid naive TWFE bias), parallel‑trends checks, and placebo tests; (2) Synthetic Control at the geo level with pre‑period fit metrics and sensitivity to donor pool/regularization; (3) Instrumental Variables exploiting exogenous variation (e.g., eligibility rules, random delivery outages), with relevance/exclusion checks and weak‑IV diagnostics; (4) Regression Discontinuity if eligibility thresholds exist (e.g., tenure ≥ 30 days), with bandwidth/functional‑form sensitivity and McCrary density tests; (5) Propensity‑score or doubly‑robust learners (IPW/AIPW, causal forests) with overlap checks and covariate balance. Explain how you would report ATE/ATT with uncertainty, account for interference/network effects, and control for seasonality and concurrent campaigns.

Quick Answer: This question evaluates a data scientist's competence in causal inference and identification strategies for estimating promotion effects from observational longitudinal data, emphasizing challenges such as staggered rollout, interference (spillovers), seasonality, and uncertainty quantification.

|Home/Analytics & Experimentation/Uber
Uber logo
Uber
Oct 13, 2025, 9:49 PM
hardData ScientistTechnical ScreenAnalytics & Experimentation
5
0

Estimating a Promotion's Causal Effect Without an Experiment

Context

You need to estimate the causal impact of a marketing promotion on engagement (e.g., trips, bookings, revenue) when a randomized experiment is not feasible. You have longitudinal data at user/geo × time granularity, with information on promotion eligibility/exposure, outcomes, covariates, and rollout timing that may be staggered across units.

Task

Propose and justify at least three distinct identification strategies to estimate the promotion’s causal effect. For each strategy, clearly state:

  • Identification setup and estimator(s)
  • Key assumptions
  • Diagnostics and robustness checks
  • Failure modes and how you would detect/mitigate them

You may choose from (or add to) the following:

  1. Difference-in-Differences (DiD) / Event Study with staggered rollout using modern estimators (avoid naive TWFE bias), including parallel-trends checks and placebo tests.
  2. Synthetic Control at the geo level with pre-period fit metrics and sensitivity to donor pool/regularization.
  3. Instrumental Variables exploiting exogenous variation (e.g., eligibility rules, random delivery outages), with relevance/exclusion checks and weak-IV diagnostics.
  4. Regression Discontinuity if eligibility thresholds exist (e.g., tenure ≥ 30 days), with bandwidth/functional-form sensitivity and McCrary density tests.
  5. Propensity-score or doubly-robust learners (IPW/AIPW, causal forests) with overlap checks and covariate balance.

Additionally, explain how you would:

  • Report ATE/ATT with uncertainty
  • Account for interference/network effects (spillovers)
  • Control for seasonality and concurrent campaigns
Loading comments...