Test payment-accuracy lift with p-value and power

Quick Overview

This question evaluates proficiency in hypothesis testing for proportions, interpretation of p-values and confidence intervals, and power/sample-size calculations within the Statistics & Math domain for a Data Scientist role.

Test payment-accuracy lift with p-value and power

Company: CVS Health

Role: Data Scientist

Category: Statistics & Math

Difficulty: medium

Interview Round: Technical Screen

A payment-accuracy team launched a rule intended to reduce incorrect payments. In a prior period (control), 10,000 transactions yielded 9,520 correct and 480 incorrect. In the post-launch period (treatment), 10,000 transactions yielded 9,640 correct and 360 incorrect. 1) Choose an appropriate hypothesis test and state H0 and H1 formally. 2) Compute the test statistic and two-sided p-value for the difference in accuracy (proportions). Show the pooled vs unpooled choices and justify which you use. 3) Construct a 95% confidence interval for the difference in proportions (treatment − control). 4) Interpret the results for a non-technical stakeholder in one sentence, including practical significance. 5) Power planning: If baseline accuracy is 96.0%, what equal per-group sample size is needed to detect an absolute improvement of 0.2 percentage points (to 96.2%) with 80% power at α = 0.05 (two-sided), using the normal approximation? Show the formula you use and the numeric inputs (Z-scores).

Quick Answer: This question evaluates proficiency in hypothesis testing for proportions, interpretation of p-values and confidence intervals, and power/sample-size calculations within the Statistics & Math domain for a Data Scientist role.

|Home/Statistics & Math/CVS Health
CVS Health logo
CVS Health
Oct 13, 2025
mediumData ScientistTechnical ScreenStatistics & Math
5
0

AB Test of Payment Accuracy (Two Proportions)

Assume transactions in the two periods are independent and large enough for normal approximations. Define "success" as a correct payment (accuracy).

  • Control (pre-launch): n = 10,000, correct = 9,520, incorrect = 480
  • Treatment (post-launch): n = 10,000, correct = 9,640, incorrect = 360

Tasks

  1. Choose an appropriate hypothesis test and state H0 and H1 formally.
  2. Compute the test statistic and two-sided p-value for the difference in accuracy (proportions). Show pooled vs unpooled standard error choices and justify which you use for the test.
  3. Construct a 95% confidence interval (CI) for the difference in proportions (treatment − control).
  4. Interpret the results for a non-technical stakeholder in one sentence, including practical significance.
  5. Power planning: If baseline accuracy is 96.0%, what equal per-group sample size is needed to detect an absolute improvement of 0.2 percentage points (to 96.2%) with 80% power at α = 0.05 (two-sided), using the normal approximation? Show the formula and numeric inputs (Z-scores).
Loading comments...