Design fundraising experiment and guardrails

Quick Overview

This question evaluates experimental design, statistical power and sample-size calculation, selection of primary and guardrail metrics, segmentation with multiple-testing control or hierarchical modeling, stopping rules for sequential analysis, and revenue-impact calculations for A/B testing of email solicitations.

Design fundraising experiment and guardrails

Company: Capital One

Role: Data Scientist

Category: Analytics & Experimentation

Difficulty: medium

Interview Round: Onsite

You plan to A/B test two online solicitation variants for the nonprofit's email campaign (subject line + suggested amount). Assume baseline conversion is 5.0% and Variant B is expected to improve conversion by 10% relative. Per-person reach cost is $1, and the average donation among converters is $80 (assume unchanged by variant). There is no capacity constraint online. Tasks: 1) Define the primary success metric and at least two guardrail metrics that protect long-term health (e.g., unsubscribe rate, complaint rate). Justify each. 2) Calculate the minimum sample size per arm to detect the expected lift (two-sided test, 95% confidence, 80% power). Show the formula and numeric result; state any approximations. 3) Describe how you would segment results by donor tier (H vs L) without inflating false positives. Include your multiple-testing or hierarchical modeling approach. 4) Outline a stopping rule and the risk of peeking. How would you handle uneven email deliverability across segments? 5) If Variant B raises conversion but lowers average gift by 5%, show how you would recompute the decision using net revenue per reached recipient.

Quick Answer: This question evaluates experimental design, statistical power and sample-size calculation, selection of primary and guardrail metrics, segmentation with multiple-testing control or hierarchical modeling, stopping rules for sequential analysis, and revenue-impact calculations for A/B testing of email solicitations.

|Home/Analytics & Experimentation/Capital One
Capital One logo
Capital One
Oct 13, 2025, 9:49 PM
mediumData ScientistOnsiteAnalytics & Experimentation
4
0

You plan to A/B test two online solicitation variants for the nonprofit's email campaign (subject line + suggested amount). Assume baseline conversion is 5.0% and Variant B is expected to improve conversion by 10% relative. Per-person reach cost is 1,andtheaveragedonationamongconvertersis1, and the average donation among converters is 80 (assume unchanged by variant). There is no capacity constraint online. Tasks:

  1. Define the primary success metric and at least two guardrail metrics that protect long-term health (e.g., unsubscribe rate, complaint rate). Justify each.
  2. Calculate the minimum sample size per arm to detect the expected lift (two-sided test, 95% confidence, 80% power). Show the formula and numeric result; state any approximations.
  3. Describe how you would segment results by donor tier (H vs L) without inflating false positives. Include your multiple-testing or hierarchical modeling approach.
  4. Outline a stopping rule and the risk of peeking. How would you handle uneven email deliverability across segments?
  5. If Variant B raises conversion but lowers average gift by 5%, show how you would recompute the decision using net revenue per reached recipient.
Loading comments...