Onsite Project Review: Analyze and present an A/B test
Before the onsite, you completed a take-home project analyzing an A/B test (you can assume typical product experimentation data: assignment, exposure, user events, and outcome metrics).
During the onsite, you must present slides and answer deep-dive questions.
What you should prepare
-
Summarize the experiment goal, design, and key assumptions.
-
Validate experiment integrity and data quality (what checks do you run?).
-
Estimate the treatment effect on pre-specified metrics.
-
Discuss interpretation and limitations (confounding risks, interference, multiple testing, seasonality).
-
Provide a clear ship/no-ship recommendation and next steps.
Interviewer follow-ups to expect
-
What would you do if you see a sample ratio mismatch?
-
How do you pick primary vs guardrail metrics?
-
How do you handle many metrics or repeated looks at the data?
-
What if average impact is neutral but a segment improves a lot?