Design and analyze an SBA mini case experiment

Quick Overview

This question evaluates expertise in experimental design and causal inference, including sample-size and power calculations, ITT/TOT analysis, variance reduction techniques, heterogeneity measurement, data quality and fraud handling, guardrail monitoring, and rollout decision-making for product experiments.

Design and analyze an SBA mini case experiment

Company: Capital One

Role: Data Scientist

Category: Analytics & Experimentation

Difficulty: hard

Interview Round: Technical Screen

A small‑business acquisition team plans a 2‑week pilot: new accounts receive a $100 credit after identity verification. Primary KPI: 30‑day activation rate = % of new accounts that achieve ≥$50 net ad spend within 30 days of signup. Guardrails: refund rate, support tickets per 1,000 accounts, average first‑30‑day spend, and CAC. Baseline activation is 12%. Budget allows 10,000 treatment accounts over 14 days; daily traffic and mix vary by region and industry. Design a rigorous experiment: choose randomization unit and stratification variables; handle day‑of‑week and regional seasonality; plan for non‑compliance and fraud. Compute the required sample size for 80% power and α=0.05 to detect a +2.0 percentage‑point lift; state all assumptions and show your calculation method. Specify the exact analysis plan: ITT vs. TOT definitions, variance reduction (e.g., CUPED with pre‑verification activity), guardrail monitoring with kill‑switch thresholds, and a precise decision rule (e.g., one‑sided test with a superiority margin and multiplicity control). Describe how you will check data quality (event lag, duplicate accounts, bots), measure heterogeneity of treatment effects (industry, region, spend propensity), and ensure results generalize beyond the 2‑week window (novelty and seasonality adjustments). Finally, outline how you would recommend rollout if the observed lift is +1.6pp with a 95% CI of [+0.2pp, +3.0pp] while support tickets increase by 12%.

Quick Answer: This question evaluates expertise in experimental design and causal inference, including sample-size and power calculations, ITT/TOT analysis, variance reduction techniques, heterogeneity measurement, data quality and fraud handling, guardrail monitoring, and rollout decision-making for product experiments.

|Home/Analytics & Experimentation/Capital One
Capital One logo
Capital One
Oct 13, 2025, 9:49 PM
hardData ScientistTechnical ScreenAnalytics & Experimentation
4
0

Design a 2‑Week Experiment: $100 Credit After ID Verification

You are designing a 2‑week pilot in which new accounts receive a 100creditafteridentityverification.TheprimaryKPIisthe30dayactivationrate:thepercentageofnewaccountsthatachieveatleast100 credit after identity verification. The primary KPI is the 30‑day activation rate: the percentage of new accounts that achieve at least 50 net ad spend within 30 days of signup.

Baseline activation rate is 12%. The budget allows up to 10,000 treatment accounts over 14 days. Daily traffic and the mix vary by region and industry.

Tasks

  1. Experimental design
    • Choose the randomization unit.
    • Choose stratification/blocking variables.
    • Explain how you will handle day‑of‑week and regional seasonality.
    • Plan for non‑compliance and fraud.
  2. Sample size and power
    • Compute the required sample size for 80% power and α = 0.05 to detect a +2.0 percentage‑point lift in activation (12% → 14%).
    • State assumptions and show the calculation method.
  3. Analysis plan
    • Define ITT and TOT; specify how you will estimate each.
    • Describe variance reduction (e.g., CUPED using pre‑treatment covariates).
    • Define guardrail monitoring and pre‑set kill‑switch thresholds for: refund rate, support tickets per 1,000 accounts, average first‑30‑day spend, and CAC.
    • Provide a precise decision rule (e.g., one‑sided test with a superiority margin) and multiplicity control.
  4. Data quality
    • Describe how you will check event lag, duplicate accounts, and bots.
  5. Heterogeneity of treatment effects (HTE)
    • Describe how you will measure HTE across industry, region, and spend propensity, and how you’ll control for multiple comparisons.
  6. External validity
    • Explain how you will ensure results generalize beyond the 2‑week window, including novelty and seasonality adjustments.
  7. Rollout decision (scenario)
    • Recommend a rollout plan if the observed lift is +1.6pp with a 95% CI of [+0.2pp, +3.0pp] while support tickets increase by 12%.
Loading comments...