Present an A/B test project review

Q: Present an A/B test project review

This is a Analytics & Experimentation interview question from PayPal for Data Scientist roles. View the full question and solution on PracHub.

Q: How do I approach Analytics & Experimentation interview questions?

Analytics & Experimentation questions require understanding of core concepts and practice. PracHub provides solutions with explanations to help you master analytics & experimentation interviews.

Question

Onsite Project Review: Analyze and present an A/B test

Before the onsite, you completed a take-home project analyzing an A/B test (you can assume typical product experimentation data: assignment, exposure, user events, and outcome metrics).

During the onsite, you must present slides and answer deep-dive questions.

What you should prepare

Summarize the experiment goal, design, and key assumptions.
Validate experiment integrity and data quality (what checks do you run?).
Estimate the treatment effect on pre-specified metrics.
Discuss interpretation and limitations (confounding risks, interference, multiple testing, seasonality).
Provide a clear ship/no-ship recommendation and next steps.

Interviewer follow-ups to expect

What would you do if you see a sample ratio mismatch?
How do you pick primary vs guardrail metrics?
How do you handle many metrics or repeated looks at the data?
What if average impact is neutral but a segment improves a lot?

Present an A/B test project review

Onsite Project Review: Analyze and present an A/B test

What you should prepare

Interviewer follow-ups to expect

Solution

Comments (0)