Explain P-Value and Errors in A/B Testing

Quick Overview

This interview question evaluates metric design, causal reasoning, experiment setup, diagnostics, SQL/statistical checks, and recommendations in a realistic interview setting. A strong answer for Explain P-Value and Errors in A/B Testing states assumptions, handles edge cases, explains trade-offs, and shows how to validate the result clearly.

Explain P-Value and Errors in A/B Testing

Company: PayPal

Role: Data Scientist

Category: Analytics & Experimentation

Difficulty: medium

Interview Round: Onsite

##### Scenario You are advising on the design and analysis of an A/B test for a new feature. ##### Question Explain what a p-value represents in an experiment context. Define Type-I and Type-II errors with business examples. Describe Simpson’s Paradox and its danger in experiment readouts. How would you select primary, secondary, and guard-rail metrics? Name and briefly describe two causal-inference methods you would use if randomization were impossible. ##### Hints Cover hypothesis testing framework, practical significance, stratification, diff-in-diff, propensity scores.

Quick Answer: This interview question evaluates metric design, causal reasoning, experiment setup, diagnostics, SQL/statistical checks, and recommendations in a realistic interview setting. A strong answer for Explain P-Value and Errors in A/B Testing states assumptions, handles edge cases, explains trade-offs, and shows how to validate the result clearly.

|Home/Analytics & Experimentation/PayPal
PayPal logo
PayPal
Aug 4, 2025, 10:55 AM
mediumData ScientistOnsiteAnalytics & Experimentation
5
0

Explain P-Value and Errors in A/B Testing

A/B Test Design and Analysis: Core Concepts

Scenario

You are advising on the design and analysis of an A/B test for a new product feature (e.g., a checkout or payments flow change). Assume standard online experimentation: users are randomly assigned to control (A) or treatment (B), and we observe conversion and risk outcomes.

Questions

  1. What does a p-value represent in the context of an experiment?
  2. Define Type-I and Type-II errors and give business-relevant examples.
  3. Describe Simpson’s Paradox and why it is dangerous in experiment readouts.
  4. How would you select primary, secondary, and guard-rail metrics for this experiment?
  5. If randomization were impossible, name and briefly describe two causal-inference methods you would use.

Clarifying Questions to Ask Guidance

  • Clarify the business objective, unit of analysis, time window, exposure definition, and primary metric.
  • State assumptions about instrumentation, randomization, sample size, and data quality.
  • Separate descriptive analysis from causal claims.

What a Strong Answer Covers Guidance

  • A metric framework with primary, guardrail, and diagnostic metrics.
  • A credible analysis or experiment design with clear assumptions and bias checks.
  • SQL/statistical logic for segmentation, variance, confidence, and data validation where relevant.
  • An actionable recommendation that explains trade-offs and next steps.

Follow-up Questions Guidance

  • What sanity checks would you run before trusting the result?
  • How would you handle novelty effects, seasonality, or selection bias?
  • What decision would you make if metrics disagree?
Loading comments...

Browse More Questions

More Analytics & Experimentation•More PayPal•More Data Scientist•PayPal Data Scientist•PayPal Analytics & Experimentation•Data Scientist Analytics & Experimentation