It started with a detailed walk through my own experience. The follow-ups were pretty detailed: why I did it that way, how I measured the effect, that kind of thing. It's best to get the full story of your own projects straight beforehand.
Then there was an experimentation case. The scenario was launching a change to search recommendation, and the question was how to design the experiment. The line of thinking is how to choose the randomization unit, how to set the primary metric and the guardrails, and whether to run an A/A test before launch.
They asked how to estimate power. It basically comes down to alpha, power, MDE, and the variance of the metric.
Follow-up: if the baseline metric changes, how does the required sample size change?
They asked whether the experiment can be ended early. You can answer from the angle of when it's okay to stop (for example, a guardrail clearly gets worse) and when it shouldn't be stopped.
Following from that, they asked whether peeking is okay and how to handle it.
They asked what to do when some metrics go up and some go down.
Last, they asked about the novelty effect: how to identify it and how to handle it. You can think about it from the angle of how the effect changes over time.
Overall it felt like they put a lot of weight on experimentation fundamentals, and the follow-ups dig down layer by layer, so when preparing, just make sure you've thought through all of these areas.
Discussion
Loading comments…