Interpret p-values and common pitfalls

Q: Interpret p-values and common pitfalls

Evaluates interpretation of p-values, hypothesis testing, experimental design and statistical pitfalls in a fraud/experimentation context—including concepts like effect size, statistical power, multiple comparisons, delayed/rare outcomes, and imperfect randomization; category/domain: Statistics & Math.

Q: How do I approach Statistics & Math interview questions?

Statistics & Math questions require understanding of core concepts and practice. PracHub provides solutions with explanations to help you master statistics & math interviews.

Question

In a Fraud Data Science interview, you are asked “some p-value questions.”

Answer the following in a fraud/experimentation context:

Define a p-value precisely. What does it mean, and what does it NOT mean?
If you run an A/B test on a new friction step (e.g., extra OTP) and get p=0.03, what conclusions can you draw? What additional information do you need (effect size, power, business impact)?
Describe at least 4 common pitfalls when using p-values in practice (multiple testing, p-hacking, peeking, nonstationarity, selection bias).
Explain how you would adjust your analysis if:
- You are testing many regions/segments.
- Outcomes are rare and delayed (e.g., chargebacks arriving weeks later).
- Randomization is imperfect or there is interference/spillover.

Provide at least one concrete method for each scenario (e.g., Bonferroni/FDR, sequential testing, CUPED, Bayesian, cluster-robust SEs).

Interpret p-values and common pitfalls

Quick Overview

Solution

Comments (0)