How do I approach Analytics & Experimentation interview questions?

Analytics & Experimentation questions require understanding of core concepts and practice. PracHub provides solutions with explanations to help you master analytics & experimentation interviews.

What difficulty level is this interview question?

This is a easy difficulty Analytics & Experimentation question, commonly asked during Technical Screen rounds at Uber.

What role is this question designed for?

This question is commonly asked for Data Scientist candidates at Uber during technical interviews.

Measure feature impact with switchback, PSM, and CACE

Q: Measure feature impact with switchback, PSM, and CACE

This question evaluates proficiency in causal inference and experimental design for product metrics, specifically testing knowledge of switchback experiments, time-series adjustments (seasonality and autocorrelation), propensity score methods for observational comparisons, and complier-average causal effect (LATE/CACE) estimation; it belongs to the Analytics & Experimentation and Data Science domain. It is commonly asked because it probes handling of real-world complications—time and city heterogeneity, non-compliance, and derived-metric inference—and tests both conceptual understanding of causal assumptions and practical application of statistical adjustment and inference.

You work at a ridesharing company and want to measure the impact of a new membership feature on rides-per-user (RPU).

Part A — Switchback experimentation

You run a switchback experiment with randomization at (day × city) granularity.

Propose a practical switchback design (assignment scheme, duration, and what units are randomized).
Explain why you generally should not take the raw aggregated results and run a vanilla two-sample t-test .
Describe an analysis approach that accounts for:
- time trends / seasonality
- city-level heterogeneity
- autocorrelation induced by switchbacks
- covariate adjustment (if relevant)

Part B — No A/B test available (observational measurement)

Assume the membership feature was launched without an A/B test. Also assume supply is unlimited (so supply constraints do not confound the outcome via availability).

How would you estimate the causal impact of membership on RPU using propensity score matching (PSM) (or a closely related propensity-score method)?
How would you assess whether your matching/weighting is “good enough” to trust the estimate?

Part C — If an A/B test exists but with non-compliance

Now assume you did run a user-level randomized experiment, but not everyone assigned to treatment actually takes up membership (non-compliance).

How would you estimate the causal effect on RPU using Complier Average Causal Effect (CACE/LATE) ?
How would you compute a confidence interval for:
- the impact on RPU
- the impact on profit-per-user (PPU) (a derived metric, e.g., revenue minus cost per user)

State key assumptions, likely pitfalls, and at least one robustness/sensitivity check for each measurement approach.

You work at a ridesharing company and want to measure the impact of a new membership feature on rides-per-user (RPU).

Part A — Switchback experimentation

You run a switchback experiment with randomization at (day × city) granularity.

Propose a practical switchback design (assignment scheme, duration, and what units are randomized).
Explain why you generally should not take the raw aggregated results and run a vanilla two-sample t-test .
Describe an analysis approach that accounts for:
- time trends / seasonality
- city-level heterogeneity
- autocorrelation induced by switchbacks
- covariate adjustment (if relevant)

Part B — No A/B test available (observational measurement)

Assume the membership feature was launched without an A/B test. Also assume supply is unlimited (so supply constraints do not confound the outcome via availability).

How would you estimate the causal impact of membership on RPU using propensity score matching (PSM) (or a closely related propensity-score method)?
How would you assess whether your matching/weighting is “good enough” to trust the estimate?

Part C — If an A/B test exists but with non-compliance

Now assume you did run a user-level randomized experiment, but not everyone assigned to treatment actually takes up membership (non-compliance).

How would you estimate the causal effect on RPU using Complier Average Causal Effect (CACE/LATE) ?
How would you compute a confidence interval for:
- the impact on RPU
- the impact on profit-per-user (PPU) (a derived metric, e.g., revenue minus cost per user)

State key assumptions, likely pitfalls, and at least one robustness/sensitivity check for each measurement approach.

Measure feature impact with switchback, PSM, and CACE

Quick Overview

Part A — Switchback experimentation

Part B — No A/B test available (observational measurement)

Part C — If an A/B test exists but with non-compliance

Solution

Comments (0)

Measure feature impact with switchback, PSM, and CACE

Quick Overview

Part A — Switchback experimentation

Part B — No A/B test available (observational measurement)

Part C — If an A/B test exists but with non-compliance

Solution

Comments (0)