Overall, this round was heavily focused on ad measurement, experiment design, and statistics fundamentals. The interviewer gave me an ad split test scenario and then dug deep into the randomization unit, treatment/control assignment, pre-treatment balance, statistical power, MDE, and how to tell whether an observed lift is a real effect or a bias caused by the experiment setup.
Two of the more specific questions:
An experiment was planned with a 90/10 split between treatment and control, but the actual traffic ended up at 92/8. How would you decide whether this is a problem? Besides checking for SRM, how else would you verify that the two groups were balanced before the experiment?
If the experiment result is directionally positive but not statistically significant, and you find the test was underpowered, how would you explain the result? What could you do before the experiment starts, and what could you do after it ends?
My overall impression is that the interviewer would not settle for answers like "do a power analysis, check balance." They kept pushing on how exactly you would check, which metrics you would use, and how these issues would affect the final business conclusion.
Discussion
Loading comments…