Explain P-Value and Errors in A/B Testing

Q: Explain P-Value and Errors in A/B Testing

This question evaluates a Data Scientist's competency in statistical inference, hypothesis testing (p-values and Type I/II errors), causal inference methods, Simpson’s Paradox awareness, and selection of primary/secondary/guard-rail metrics within online experiment design, falling under the Analytics & Experimentation domain.

Q: How do I approach Analytics & Experimentation interview questions?

Analytics & Experimentation questions require understanding of core concepts and practice. PracHub provides solutions with explanations to help you master analytics & experimentation interviews.

Question

A/B Test Design and Analysis: Core Concepts

Scenario

You are advising on the design and analysis of an A/B test for a new product feature (e.g., a checkout or payments flow change). Assume standard online experimentation: users are randomly assigned to control (A) or treatment (B), and we observe conversion and risk outcomes.

Questions

What does a p-value represent in the context of an experiment?
Define Type-I and Type-II errors and give business-relevant examples.
Describe Simpson’s Paradox and why it is dangerous in experiment readouts.
How would you select primary, secondary, and guard-rail metrics for this experiment?
If randomization were impossible, name and briefly describe two causal-inference methods you would use.

Explain P-Value and Errors in A/B Testing

A/B Test Design and Analysis: Core Concepts

Scenario

Questions

Solution

Comments (0)