Differentiate and control Type I/II errors

Quick Overview

This question evaluates understanding of hypothesis testing fundamentals such as Type I/Type II errors, statistical power and sample size calculation for two-proportion tests, family-wise error rate control (e.g., Bonferroni), and sequential analysis/alpha-spending schemes.

Differentiate and control Type I/II errors

Company: OneMain Financial

Role: Data Scientist

Category: Statistics & Math

Difficulty: medium

Interview Round: Technical Screen

You are planning a two-arm A/B test on sign-up conversion. Baseline conversion p0 = 0.10. You want to detect an absolute uplift of +0.01 (to 0.11) with a two-sided α = 0.05 and power 1−β = 0.80 using a z-test for two proportions. 1) Derive the required sample size per arm and show your formula and numbers. 2) If you will also monitor 10 secondary metrics and must control the family-wise error rate with Bonferroni, what is the per-metric α and how does this change your sample size? 3) If you plan to peek daily over 14 days, describe a valid sequential scheme (e.g., O’Brien–Fleming or Pocock alpha-spending) and how it alters the stopping boundary. 4) Explain the practical cost of Type II error in this test and give one concrete way to reduce β without increasing α (quantify the trade-off).

Quick Answer: This question evaluates understanding of hypothesis testing fundamentals such as Type I/Type II errors, statistical power and sample size calculation for two-proportion tests, family-wise error rate control (e.g., Bonferroni), and sequential analysis/alpha-spending schemes.

|Home/Statistics & Math/OneMain Financial
OneMain Financial logo
OneMain Financial
Oct 13, 2025, 9:49 PM
mediumData ScientistTechnical ScreenStatistics & Math
7
0

A/B Test Powering and Error Control (Two-Proportion Z-Test)

Context: You are planning a two-arm A/B test on sign-up conversion. The current baseline conversion rate is p0 = 0.10. You want to detect an absolute uplift of +0.01 (to p1 = 0.11) using a two-sided test.

Assumptions:

  • Equal allocation to A and B (n per arm).
  • Two-sided significance level α = 0.05, desired power 1 − β = 0.80.
  • Normal approximation (z-test) for two independent proportions.

Tasks

  1. Derive the required sample size per arm to detect a 1 pp uplift (0.10 → 0.11). Show the formula and plug in the numbers.
  2. You will also monitor 10 secondary metrics and must control the family-wise error rate (FWER) using Bonferroni.
    • What is the per-metric α?
    • How does this change your required sample size for detecting the same 1 pp uplift, under reasonable interpretations of FWER control?
  3. If you plan to peek daily over 14 days, describe a valid sequential scheme (e.g., O’Brien–Fleming or Pocock alpha-spending) and how it alters the stopping boundaries relative to a fixed-sample analysis.
  4. Explain the practical cost of a Type II error (false negative) in this test. Provide one concrete way to reduce β without increasing α, and quantify the trade-off.
Loading comments...