Measure feature impact with switchback, PSM, and CACE
Overview
This question evaluates proficiency in causal inference and experimental design for product metrics, specifically testing knowledge of switchback experiments, time-series adjustments (seasonality and autocorrelation), propensity score methods for observational comparisons, and complier-average causal effect (LATE/CACE) estimation; it belongs to the Analytics & Experimentation and Data Science domain. It is commonly asked because it probes handling of real-world complications—time and city heterogeneity, non-compliance, and derived-metric inference—and tests both conceptual understanding of causal assumptions and practical application of statistical adjustment and inference.
You work at a ridesharing company and want to measure the impact of a new membership feature on rides-per-user (RPU).
Part A — Switchback experimentation
You run a switchback experiment with randomization at (day × city) granularity.
- Propose a practical switchback design (assignment scheme, duration, and what units are randomized).
- Explain why you generally should not take the raw aggregated results and run a vanilla two-sample t-test .
-
Describe an analysis approach that accounts for:
- time trends / seasonality
- city-level heterogeneity
- autocorrelation induced by switchbacks
- covariate adjustment (if relevant)
Part B — No A/B test available (observational measurement)
Assume the membership feature was launched without an A/B test. Also assume supply is unlimited (so supply constraints do not confound the outcome via availability).
- How would you estimate the causal impact of membership on RPU using propensity score matching (PSM) (or a closely related propensity-score method)?
- How would you assess whether your matching/weighting is “good enough” to trust the estimate?
Part C — If an A/B test exists but with non-compliance
Now assume you did run a user-level randomized experiment, but not everyone assigned to treatment actually takes up membership (non-compliance).
- How would you estimate the causal effect on RPU using Complier Average Causal Effect (CACE/LATE) ?
-
How would you compute a
confidence interval
for:
- the impact on RPU
- the impact on profit-per-user (PPU) (a derived metric, e.g., revenue minus cost per user)
State key assumptions, likely pitfalls, and at least one robustness/sensitivity check for each measurement approach.
Solution
Login required