Overall the experience was good. The interviewer was very nice and the conversation flowed smoothly. In 45 minutes we covered product sense, experiment design, experiment analysis, and SQL.
Question 1
How to investigate a metric decline. Pretty standard question. Seasonality, product changes, user segments, external competition — anything reasonable works.
Question 2
If you were launching a new pricing model to incentivize shoppers to pick up more orders during rush hours, how would you design the experiment? Because network effects affect supply and demand, you can't use an A/B test. You need a geo-based experiment to find lookalike markets, check covariates for potential bias, and so on.
Question 3
Suppose you've already run an experiment where the north star metric is profit per order. The results show average order volume went up, but profit went down instead of up — should you roll it out? I initially said you could investigate further at the segment level, since the volume increase seemed pretty promising blah blah blah, but the interviewer kindly reminded me that profit per order is the NSM. The right recommendation is not to launch.
Question 4
Two SQL questions, in a Google Doc. The first was a left join plus group by, the second was a window function — lag/lead for month-over-month change. Neither was hard.
Question 5
Look at a chart and explain it: a D14 retention metric dropped a lot in the most recent week — how would you find the cause? The answer is that the metric just hasn't matured yet, so this is a false alarm.
No particularly tricky questions, but you do need some familiarity with how marketplace companies run experiments. Good luck everyone!
Discussion
Loading comments…