Experimentation And A/B Testing

What's being tested

Interviewers are probing your practical command of experimentation as a product lever: how you frame decisions, pick and defend metrics, design (and stop) tests responsibly, and translate results into rollout decisions. Capital One cares because experiments are the core way PMs trade off revenue, risk, and customer experience while keeping regulatory and trust constraints in mind. Expect them to evaluate your ability to balance statistical reasoning, product tradeoffs, stakeholder communication, and operational guardrails.

Core knowledge

• A/B test basics: randomize users into treatment/control, define exposure precisely, and measure the average treatment effect (ATE) on your primary metric with pre-specified analysis.

• Primary metric selection: choose one metric that directly ties to the business decision (e.g., conversion_rate, ARPU, churn_rate); tie secondary metrics and guardrails to safety, legal, or latency concerns.

• Minimum Detectable Effect (MDE) and sample sizing: compute sample size from desired power and alpha using $n \approx 2\left(\frac{Z_{1-\alpha/2}+Z_{1-\beta}}{\delta/\sigma}\right)^2$ where $\delta$ is MDE and $\sigma$ is metric SD.

• Power (1−β) and Type I error (α) tradeoffs: common PM defaults are power = 80% and α = 5%, but lower α may be required for high-risk launches (financial, legal).

• Sequential testing & peeking: ad-hoc interim looks inflate false positives; use pre-registration, alpha-spending or proper sequential frameworks, or adopt Bayesian methods and state them up-front.

• Guardrails and SLA metrics: simultaneously monitor error_rate, latency, and customer experience proxies (e.g., NPS, retention) for rollback criteria and business safety—predefine alert thresholds.

• Heterogeneous treatment effects (HTE) and segmentation: pre-specify segment analyses (by device, cohort, geography) to avoid post-hoc fishing; control for multiple comparisons using Bonferroni or FDR when needed.

• Instrumention & data quality: validate randomization, sample integrity, event loss, and duplication; a PM should own the question "Can we trust conversion events in Snowflake/Looker/Amplitude?" before deciding.

• SUTVA / interference: watch for spillovers (social features, referrals) that violate independent assignment; prefer cluster-randomization when users influence each other.

• Short-term vs long-term effects: measure immediate metrics and plan for long-horizon tracking (lifetime value, retention) using holdouts or longitudinal experiments to detect transient novelty effects.

• Reversibility & rollout strategy: always decide if a change is reversible; use staged rollouts (e.g., 1% → 10% → 50%) with objective stop criteria, and consider a permanent holdout for long-term impact estimation.

• Communication & decision framing: pre-specify success criteria, write a clear experiment brief, and communicate uncertainty with confidence intervals and practical impact (e.g., "this increases weekly spend by $0.75 ±$0.30 per user").

Worked example — How do you make data-driven decisions?

First 30s: clarify the decision (e.g., launch feature X globally?), the audience (ops/finance), and the time horizon (short-term conversions vs long-term retention). Skeleton answer pillars: (1) define the primary metric and its business-alignment, (2) decide experiment vs observational approach and compute MDE/sample size, (3) enumerate diagnostics and guardrails to detect regressions, (4) plan segmentation and rollout, (5) define communication and stop rules. Flag a tradeoff: picking a single primary metric simplifies hypothesis testing but can miss important side effects—so pick one primary plus explicit guardrails. Close by stating next steps: run a pilot with instrumentation checks, preregister analysis, run to required power, then ramp if safe; add “if I had more time, I’d set up a 90-day holdout to measure retention and LTV.”

A second angle — Improve Capital One Shopping

Here the experimentation frame shifts to commercial partnerships and long-term value. Treat merchant payouts and coupon availability as business constraints: define both user-centric metrics (conversion, coupon-usage trust) and revenue/cost-weighted metrics (net take-rate, partner lifetime value). Email A/B tests require lifecycle-aware metrics (open → click → purchase), and personalization tests need offline slices to avoid heavy multiple-testing penalties. Tradeoffs include short-term revenue vs user trust (over-personalization reduces coupon diversity), and regulatory constraints around disclosures. Pre-specify tests on a per-partner basis to avoid biased monetization results.

Common pitfalls

Pitfall: Picking many "interesting" metrics as primaries. Multiple primary metrics creates ambiguous success criteria and inflates false positives; pick one business-aligned primary and pre-declare secondaries.

Pitfall: Peeking and stopping when the result looks good. Running unplanned interim analyses without correction leads to false discoveries; always pre-register stopping rules or use proper sequential methods.

Pitfall: Ignoring instrumentation drift and exposure definition. Trusting events before verifying randomization, event completeness, and user-scope leads to garbage conclusions—run A/A tests and data QA first.

Connections

Interviewers often pivot to adjacent topics: product prioritization (how experiments inform roadmap tradeoffs), causal inference (when experiments aren't possible), or analytics instrumentation (ensuring events and cohort definitions are reliable). Be ready to discuss tradeoffs between experimentation and observational analyses.

Further reading

• Kohavi et al., “Online Controlled Experiments at Large Scale” (MSR) — practical engineering and statistical lessons from large-scale experimentation.

• Evan Miller, “A/B Testing” guide — concise tutorial on statistics, power, and pitfalls for practitioners.