Master A/B Testing: Key Concepts and Methodologies Explained
Overview
This question evaluates a data scientist's practical mastery of A/B testing and causal inference, covering statistical concepts (p-values, Type I/II errors and power), experimental design (sample size, segmentation, variance, metrics) and nonrandomized causal methods.
PayPal
Jul 12, 2025, 6:59 PM
Data Scientist
Onsite
Analytics & Experimentation
64
0
A/B Testing and Causal Inference: Core Concepts and Practice
Scenario
You are a data scientist interviewing for a role working on an online product. You are asked to demonstrate practical A/B testing and causal inference knowledge.
Task
Provide concise, accurate explanations and practical guidance for the following:
- P-value: What it represents and common misinterpretations.
- Errors: Define Type I and Type II errors; relate to power.
- Experimentation Workflow: Outline an end-to-end process from hypothesis to decision, including sample size, segmentation, and variance considerations.
- Simpson's Paradox: Define, give a practical example, and explain how to detect and handle it.
- Metrics: Propose primary and secondary/guardrail metrics for an online product experiment.
- Causal Inference Without Randomization: Name two useful methods and when you would apply each.
Hints to address: hypothesis clarity, sample-size (power) calculation, segmentation, lift vs variance, DAGs or matching, and practical examples.
Solution
Login required
Show