Overall the experience was good. The interviewer was very nice, and communication was smooth. The 45 minutes covered product sense, experiment design, experiment analysis, and SQL.
Question 1
How to investigate a metric decline. Pretty standard question. Seasonality, product changes, user segments, external competition — as long as it's reasonable, that's fine.
Question 2
If you want to launch a new pricing model incentivizing shoppers to pick up more orders during rush hours, how would you design the experiment. Because network effects affect supply and demand, you can't use an A/B test. You need a geo-based experiment to find lookalike markets, check covariates for potential bias, etc.
Question 3
Suppose you've already run an experiment, and the north star metric is profit per order. The results show average order volume increased, but profit actually decreased instead of increasing — should you roll it out or not. I initially said you could investigate further for segment-level results, because the volume increase was promising blah blah blah, but the interviewer kindly reminded me that profit per order is the NSM. The suggestion should be not to launch.
Question 4
Two SQL questions, on a Google Doc. The first was a left join + group by, the second was a window function, lag/lead for MoM change. Neither was hard.
Question 5
Interpret a chart. A D14 retention metric dropped a lot in the past week, how would you find the cause. It's clearly that the metric hasn't matured yet, so this is a false alarm.
No especially tricky questions, but you do need to understand marketplace experiment practices. Good luck everyone!
Discussion
Loading comments…