Design A/B Test for Marketing Campaign Impact Evaluation

Q: Design A/B Test for Marketing Campaign Impact Evaluation

This question evaluates experimental design and causal inference competencies for measuring incremental card acquisitions and cardholder spend, covering metric definition, assignment strategy, guardrail identification, and risk-aware analysis within the Analytics & Experimentation domain.

Q: How do I approach Analytics & Experimentation interview questions?

Analytics & Experimentation questions require understanding of core concepts and practice. PracHub provides solutions with explanations to help you master analytics & experimentation interviews.

Q: What difficulty level is this interview question?

This is a medium difficulty Analytics & Experimentation question, commonly asked during Onsite rounds at Capital One.

Q: What role is this question designed for?

This question is commonly asked for Data Scientist candidates at Capital One during technical interviews.

Question

Scenario

A credit card issuer is testing a marketing campaign that waives the first-year annual fee. You need to design an A/B test to measure incremental card acquisitions and cardholder spend while ensuring customer experience and risk do not degrade.

Assume the campaign can run across web and email to prospect audiences and that you can randomize at the person level consistently across channels. The standard offer (control) charges the first-year annual fee; the variant (treatment) waives it.

Task

Design the experiment and analysis plan. Include:

Experiment design and assignment.
Primary outcome metrics and how they are computed.
Guardrail metrics (with rationale).
Sample-size calculations for acquisition and spend (state assumptions and show formulas with a small numeric example).
Post-experiment analysis steps, including variance reduction (e.g., CUPED or stratification).

Hints

Focus on conversion rate and revenue/spend lift.
Consider CUPED or stratified randomization to increase power.
Address heavy-tailed spend distributions and operational/risk guardrails.

Design A/B Test for Marketing Campaign Impact Evaluation

Quick Overview

Scenario

Task

Hints

Solution

Comments (0)