Evaluate impact without randomized experiments

Q: Evaluate impact without randomized experiments

This question evaluates a data scientist's competence in causal inference and identification strategies for estimating promotion effects from observational longitudinal data, emphasizing challenges such as staggered rollout, interference (spillovers), seasonality, and uncertainty quantification.

Q: How do I approach Analytics & Experimentation interview questions?

Analytics & Experimentation questions require understanding of core concepts and practice. PracHub provides solutions with explanations to help you master analytics & experimentation interviews.

Q: What difficulty level is this interview question?

This is a hard difficulty Analytics & Experimentation question, commonly asked during Technical Screen rounds at Uber.

Q: What role is this question designed for?

This question is commonly asked for Data Scientist candidates at Uber during technical interviews.

Question

Estimating a Promotion's Causal Effect Without an Experiment

Context

You need to estimate the causal impact of a marketing promotion on engagement (e.g., trips, bookings, revenue) when a randomized experiment is not feasible. You have longitudinal data at user/geo × time granularity, with information on promotion eligibility/exposure, outcomes, covariates, and rollout timing that may be staggered across units.

Task

Propose and justify at least three distinct identification strategies to estimate the promotion’s causal effect. For each strategy, clearly state:

Identification setup and estimator(s)
Key assumptions
Diagnostics and robustness checks
Failure modes and how you would detect/mitigate them

You may choose from (or add to) the following:

Difference-in-Differences (DiD) / Event Study with staggered rollout using modern estimators (avoid naive TWFE bias), including parallel-trends checks and placebo tests.
Synthetic Control at the geo level with pre-period fit metrics and sensitivity to donor pool/regularization.
Instrumental Variables exploiting exogenous variation (e.g., eligibility rules, random delivery outages), with relevance/exclusion checks and weak-IV diagnostics.
Regression Discontinuity if eligibility thresholds exist (e.g., tenure ≥ 30 days), with bandwidth/functional-form sensitivity and McCrary density tests.
Propensity-score or doubly-robust learners (IPW/AIPW, causal forests) with overlap checks and covariate balance.

Additionally, explain how you would:

Report ATE/ATT with uncertainty
Account for interference/network effects (spillovers)
Control for seasonality and concurrent campaigns

Evaluate impact without randomized experiments

Quick Overview

Estimating a Promotion's Causal Effect Without an Experiment

Context

Task

Solution

Comments (0)