Design A/B test for credit card offer

Quick Overview

This question evaluates a data scientist's competency in experimental design, causal inference, metric definition and measurement windows, power analysis, bias mitigation, identity resolution, and regulatory compliance for financial-product A/B testing.

Design A/B test for credit card offer

Company: Capital One

Role: Data Scientist

Category: Analytics & Experimentation

Difficulty: hard

Interview Round: Technical Screen

You’re launching a new credit-card acquisition flow with a revised APR disclosure and signup bonus. Design an end-to-end A/B test: define the unit of randomization, eligibility/exclusions (e.g., existing customers, fraud), primary and guardrail metrics (e.g., approved accounts, activation rate, delinquency/default), minimal detectable effect, statistical power, sample size, and expected test duration. Address selection bias (pre-approval, underwriting), cross-channel interference, peeking/early stopping, and regulatory constraints (e.g., fair lending). How would you analyze heterogeneous effects by segment while controlling false discovery?

Quick Answer: This question evaluates a data scientist's competency in experimental design, causal inference, metric definition and measurement windows, power analysis, bias mitigation, identity resolution, and regulatory compliance for financial-product A/B testing.

|Home/Analytics & Experimentation/Capital One
Capital One logo
Capital One
Oct 13, 2025, 9:49 PM
hardData ScientistTechnical ScreenAnalytics & Experimentation
11
0

A/B Test Design: New Credit-Card Acquisition Flow (Revised APR Disclosure + Signup Bonus)

Context

You are launching a new credit-card acquisition flow that changes the APR disclosure and introduces a revised signup bonus. Design an end-to-end A/B test that measures business impact while maintaining risk and regulatory compliance.

Tasks

  1. Experimental design
    • Define the unit of randomization, exposure stickiness, and allocation.
    • Describe how you will ensure assignment consistency across channels/devices.
  2. Eligibility and exclusions
    • Specify who is eligible and who is excluded (e.g., existing customers, suspected fraud, do-not-solicit, restricted geographies).
  3. Metrics
    • Define the primary success metric.
    • Define secondary and guardrail metrics (e.g., application completion, approval rate, activation rate, early delinquency/default or risk score proxies).
    • Provide clear metric definitions and measurement windows.
  4. Powering the test
    • Set the minimal detectable effect (MDE) and statistical power.
    • Compute the required sample size and expected test duration given daily traffic assumptions.
  5. Bias and interference
    • Address selection bias (e.g., pre-approved lists, underwriting filters).
    • Address cross-channel interference and identity resolution.
  6. Decision integrity
    • Describe how to prevent peeking/early stopping and how you will handle sequential looks if needed.
  7. Regulatory and compliance constraints
    • Identify relevant constraints (e.g., fair lending) and how the test will comply.
  8. Heterogeneous effects
    • Explain how you would analyze heterogeneous treatment effects by segment while controlling false discovery.
Loading comments...