Experiment Design to Measure Whether Voice Ordering Lifts Amazon Purchase Frequency

You are a product manager at Amazon. A new feature that lets customers shop by voice is being tested. Design an A/B test that would show whether the feature makes customers purchase more often. ```hint Decide which orders count A new way to buy creates orders through that channel almost by definition. Think about what the comparison must count so that the result shows extra buying rather than buying that simply moved from one channel to another. ``` ### Clarifying Questions - Where does the feature run: on Alexa-enabled devices such as Echo speakers, as voice input inside the Amazon shopping app, or both? - Which customers can use it: everyone with a supported device, or only accounts with a payment method set up and voice purchasing turned on? - Does purchase frequency mean orders per customer across all of Amazon over a fixed window, or only orders in certain categories, such as reorders of household items? - What is the smallest increase that would justify investing further in the feature, and when is a decision needed? ### What a Strong Answer Covers - A stated hypothesis and a primary metric that counts orders across every channel over a fixed window, not voice orders alone - A unit of randomization and an eligibility rule that fit shared household accounts and devices, with an intent-to-treat analysis - Sample size and duration suited to a low-frequency, skewed purchase metric, including variance reduction - Guardrails on revenue, order value, accidental orders, returns and cancellations, and customer contacts - A decision rule written in advance, segment reads, and threats to validity such as novelty, sample-ratio mismatch and spillover ### Follow-up Questions - Orders per customer rise in treatment, but average order value falls enough that revenue per customer is flat. What do you recommend? - Almost all of the lift comes from customers reordering the same household items. How does that change your view of the feature and the next test? - Leadership asks for a read after one week. What can you honestly say at that point, and what can't you?