Uber Senior+ Data Scientist Interview Experience — Diff-in-Diff, Switchback Design, and a Causal Inference Deep Dive

Sharing an interview. I'm guessing I failed.

Uber Staff Scientist.

1. Coding + experimentation

The first part was data manipulation. I was given a pandas DataFrame and had to convert a column's type from string to int. There was a missing value that needed to be imputed using a simple average. After those two steps, I had to compute a diff-in-diff.

The experimentation part was pretty relaxed. He asked if I had experience with anything other than a standard A/B test, and I said I'd worked with switchback tests. Then he asked a few casual follow-ups:

• Design a switchback experiment at the [day x city] granularity
• Can you just take the results and run a t-test directly? No — you need covariate adjustment and some time series adjustment.

Strongly recommend reading Uber's blog post on experimentation — it explains this stuff really clearly.

2. Causal inference & measurement

If Uber launched a "membership" feature but didn't run an A/B test — assuming supply is unlimited — how would you measure the impact on rides-per-user? Propensity score matching.

How do you estimate whether the propensity score matching is good enough? From here on I honestly wasn't sure what the "correct" answer was.

If you did run a user-level A/B test, how would you measure the impact on rides-per-user? Complier average causal effect.

How do you compute a confidence interval for the impact above? ...
Then compute a profit-per-user impact too — how do you compute a confidence interval for that metric? ...

This blog post summarizes the causal inference methods Uber commonly uses, though honestly I haven't actually used most of them myself, T_T.