Design and analyze pricing-page A/B test
Overview
This question evaluates data-science competencies in online experiment design and analysis, including statistical power and sample-size estimation, variance-reduction methods, guardrail and SRM monitoring, randomization and contamination handling, sequential monitoring and stopping rules, heterogeneity/subgroup analysis, and result communication within the analytics & experimentation domain. It is commonly asked to assess practical application of statistical inference and experimental rigor in product and business contexts, primarily testing practical application while requiring conceptual understanding of underlying statistical principles and operational trade-offs.
AB Test Plan: New Pricing-Page Layout
Context: You will run a 2-arm online experiment on a pricing page. The primary metric is user-level paid conversion within 7 days of a user's first pricing-page view. Baseline conversion is 6.0%, the minimum detectable effect (MDE) is a +5% relative lift (to 6.3%), with α = 0.05 (two-sided) and power = 0.80. Daily traffic is ~200,000 unique users (60% new, 40% returning; 10% tablet, 30% desktop, 60% mobile). Some users switch devices; a bot filter is applied; a portion of traffic is from paid ads with different intent.
Tasks
-
Sample size and duration
- Compute per-variant sample size and the minimum test duration with and without CUPED (assume CUPED yields 20% variance reduction).
-
Guardrails and SRM
- Define guardrail metrics (e.g., SRM, bounce rate, revenue/user) and how to detect SRM in near real time.
-
Randomization and contamination
- Specify the randomization unit (cookie vs user vs account), how to handle cross-device identity, and how to prevent contamination.
-
Sequential monitoring and stopping
- Plan sequential looks that control Type I error (e.g., alpha-spending) and describe a pre-registered stopping rule.
-
Heterogeneity analysis
- Outline subgroup analysis across device and traffic source, and how to proceed if the overall effect is neutral but mobile shows a significant lift.
-
Analysis and decisioning
- Provide a concrete plan for analyzing and communicating results, including effect sizes with 95% CIs and a decision framework for ship/iterate/halt.
Solution
Login required