Interpret a Checkout Redesign A/B Test: Conversion Up, Order Value Down, Revenue Flat

A commerce platform tested a redesigned one-page checkout (treatment) against its current multi-step checkout (control). Visitors who reached checkout were randomized 50/50 at the visitor level for 14 days. A slide shows the readout below, and you are asked to interpret it. The report says only that candidates were asked how to interpret A/B test results; the experiment and the numbers below are illustrative. | Metric | Control | Treatment | Difference (treatment − control) | 95% CI of the difference | p-value | | --- | --- | --- | --- | --- | --- | | Visitors randomized | 50,120 | 49,880 | | | | | Orders | 31,074 | 31,325 | | | | | Checkout conversion (orders ÷ visitors) | 62.0% | 62.8% | +0.8 pp (+1.3% relative) | +0.2 to +1.4 pp | 0.009 | | Average order value (revenue ÷ orders) | \$84.10 | \$82.90 | −\$1.20 (−1.4%) | −\$2.60 to +\$0.20 | 0.09 | | Revenue per visitor (revenue ÷ visitors) | \$52.14 | \$52.06 | −\$0.08 (−0.2%) | −\$1.08 to +\$0.92 | 0.88 | "pp" means percentage points. Each randomized visitor counts once in the visitor totals, whether or not they ordered. ### Clarifying Questions - Which metric was declared primary before the test started, and was a minimum effect worth acting on agreed in advance? - Is the goal more buyers, or more sales for merchants? - Were guardrail metrics such as payment failures, refunds or page load time tracked? - Does the one-page design change anything besides layout, such as which add-on or upsell offers appear? - Are orders counted when placed, before any later cancellations or refunds? ### Part 1 — Check the readout before trusting it What would you check before interpreting the differences? Use the numbers on the slide where you can. ```hint Start with the split Compare the visitor counts with what a 50/50 split should produce, and check that the three metrics agree with each other. ``` #### What This Part Should Cover - A sample-ratio check computed from the visitor counts - Consistency among the three metrics and their definitions - Design questions the slide leaves open, such as duration, novelty and the analysis plan ### Part 2 — Interpret each metric For each metric, state what the result supports and what it does not. ```hint Who is in the denominator Ask which metrics are averaged over randomized visitors and which over a group whose membership the treatment itself changes. ``` #### What This Part Should Cover - Statistical and practical significance of the conversion change - Why the order-value comparison is not a clean randomized comparison - What the revenue interval rules in and rules out ### Part 3 — Recommend a decision Would you launch the one-page checkout? Give your recommendation and say what would change it. ```hint Tie it to the goal The call depends on whether the team optimizes for conversion or for revenue; say which you assume and how the decision would flip under the other. ``` #### What This Part Should Cover - A clear recommendation that follows from the primary metric and the guardrails - Next steps such as segment cuts, checks on the mechanism, a follow-up test or more power - The risks of launching and of not launching ### What a Strong Answer Covers - Validity checks performed before any interpretation - Correct reading of confidence intervals and p-values, including what a non-significant result does and does not show - Use of the identity revenue per visitor = conversion × average order value, and what it implies here - A decision rule tied to the pre-registered primary metric and guardrails - Concrete next steps with a sense of how much more data a revenue answer would need ### Follow-up Questions - Suppose the split had been 52,000 control visitors and 48,000 treatment visitors. What would you conclude, and what would you do? - How would you test whether customers who would have bought anyway now spend less, as opposed to the extra buyers simply placing smaller orders? - Roughly how much longer would the test need to run to detect a 1% change in revenue per visitor with 80% power?