Calculate A/B sample size, CI, decision rules

Q: Calculate A/B sample size, CI, decision rules

This question evaluates skills in experimental design and statistical inference, including sample size calculation, hypothesis testing, confidence interval interpretation, multiple comparison control, variance reduction techniques, and operational considerations for A/B testing.

Q: How do I approach Statistics & Math interview questions?

Statistics & Math questions require understanding of core concepts and practice. PracHub provides solutions with explanations to help you master statistics & math interviews.

Question

You are designing and analyzing a two-arm A/B test for a signup funnel. Assume 1:1 traffic split and independent user observations.

Given:

Baseline conversion p0 = 0.040
Minimum detectable effect (absolute) = 0.004 → target p1 = 0.044
Two-sided α = 0.05, power = 0.80

Tasks

Sample Size
- Compute the required sample size per arm for a two-proportion z-test. Show the formula and the numeric result.
CI and p-value
- After launch you observe: Control nC = 50,000, xC = 2,150 (4.30%) and Treatment nT = 49,500, xT = 2,090 (4.22%).
- Compute the p-value for the difference in proportions using both pooled and unpooled standard errors.
- Compute a 95% confidence interval for (pT − pC) and interpret practically.
Design Choices
- Specify primary vs. guardrail metrics.
- If you track cancellation rate and latency as guardrails, explain how you would correct for multiple comparisons.
Power / Variance Reduction
- Explain how to use CUPED or stratification to reduce variance and how that changes required sample size.
Operational Risks
- Discuss sequential peeking and how to control type-I error (e.g., alpha-spending).
- What if traffic is heterogeneous and the global average obscures segment-level reversals?

Calculate A/B sample size, CI, decision rules

Solution

Comments (0)

Calculate A/B sample size, CI, decision rules

Overview

A/B Test Design and Analysis: Signup Funnel

Solution

Comments (0)