Design A/B Test for Marketing Campaign Impact Evaluation
Quick Overview
This interview question evaluates metric design, causal reasoning, experiment setup, diagnostics, SQL/statistical checks, and recommendations in a realistic interview setting. A strong answer for Design A/B Test for Marketing Campaign Impact Evaluation states assumptions, handles edge cases, explains trade-offs, and shows how to validate the result clearly.
Design A/B Test for Marketing Campaign Impact Evaluation
Company: Capital One
Role: Data Scientist
Category: Analytics & Experimentation
Difficulty: medium
Interview Round: Onsite
##### Scenario
Analyst case study – evaluate the business impact of a marketing campaign waiving first-year annual fees.
##### Question
Design an A/B test to measure incremental card acquisitions and spend. Specify primary metrics, guardrail metrics, sample-size calculation, and post-experiment analysis steps.
##### Hints
Conversion rate, revenue lift, CUPED or stratification to increase power.
Quick Answer: This interview question evaluates metric design, causal reasoning, experiment setup, diagnostics, SQL/statistical checks, and recommendations in a realistic interview setting. A strong answer for Design A/B Test for Marketing Campaign Impact Evaluation states assumptions, handles edge cases, explains trade-offs, and shows how to validate the result clearly.
Design A/B Test for Marketing Campaign Impact Evaluation
Scenario
A credit card issuer is testing a marketing campaign that waives the first-year annual fee. You need to design an A/B test to measure incremental card acquisitions and cardholder spend while ensuring customer experience and risk do not degrade.
Assume the campaign can run across web and email to prospect audiences and that you can randomize at the person level consistently across channels. The standard offer (control) charges the first-year annual fee; the variant (treatment) waives it.
Task
Design the experiment and analysis plan. Include:
Experiment design and assignment.
Primary outcome metrics and how they are computed.
Guardrail metrics (with rationale).
Sample-size calculations for acquisition and spend (state assumptions and show formulas with a small numeric example).
Post-experiment analysis steps, including variance reduction (e.g., CUPED or stratification).
Hints
Focus on conversion rate and revenue/spend lift.
Consider CUPED or stratified randomization to increase power.
Address heavy-tailed spend distributions and operational/risk guardrails.
Constraints & Assumptions
Preserve the scope, facts, inputs, and requested outputs from the prompt above.
If the prompt leaves a detail unspecified, state a reasonable assumption before relying on it.
Keep the answer interview-ready: concise enough to present, but concrete enough to implement or evaluate.
Clarifying Questions to Ask Guidance
Clarify the business objective, unit of analysis, time window, exposure definition, and primary metric.
State assumptions about instrumentation, randomization, sample size, and data quality.
Separate descriptive analysis from causal claims.
What a Strong Answer Covers Guidance
A metric framework with primary, guardrail, and diagnostic metrics.
A credible analysis or experiment design with clear assumptions and bias checks.
SQL/statistical logic for segmentation, variance, confidence, and data validation where relevant.
An actionable recommendation that explains trade-offs and next steps.
Follow-up Questions Guidance
What sanity checks would you run before trusting the result?
How would you handle novelty effects, seasonality, or selection bias?
