Experiment design for emails introducing a new product to existing customers

You are a product manager at Capital One, and the company is about to email customers to announce a new product. How would you A/B test those announcement emails? ```hint Decide what success means first Ask what the email exists to achieve, then check whether the metric that is easiest to move is the one that reflects it. ``` ### Clarifying Questions - Which product is being announced, and what should a reader do next: apply, turn on a feature, or simply learn that it exists? - Who receives the email: all customers, or only those who have opted in to marketing email and are eligible for the product? - Is the test about the content (subject line, message, call to action), about timing and targeting, or about whether sending the email helps at all? - Is this a one-time send, or a template that future product launches will reuse? ### What a Strong Answer Covers - A hypothesis and a primary metric tied to product adoption, with opens and clicks treated as diagnostics - Randomization at the customer level, variants that change one element each, and a no-email holdout - A sample-size estimate for the chosen metric and a fixed measurement window - Guardrails such as unsubscribes, spam and phishing reports, plus compliance review of every variant - Analysis and decision rules, including unreliable open tracking and the risk of false winners across several variants ### Follow-up Questions - The variant with the best open rate produced the fewest sign-ups. Which one do you ship, and why? - Your eligible list is too small to detect a change in sign-ups. What would you do instead? - How would you test ten subject lines at once without being fooled by a false winner?