Design an A/B for ATO rule

Quick Overview

This question evaluates a data scientist's experiment-design and statistical-analysis competencies, including cluster-aware randomization, power/sample-size calculation, sequential monitoring, and translating detectable effects into net business impact for a real-time account-takeover prevention rule on a payment platform.

Design an A/B for ATO rule

Company: PayPal

Role: Data Scientist

Category: Analytics & Experimentation

Difficulty: hard

Interview Round: Technical Screen

Experiment design case: PayPal/Venmo wants to launch a new real-time ATO rule that blocks high-risk transfers. Design and analyze an online experiment to estimate net business impact. Constraints and inputs: (1) Randomization unit must avoid cross-over: choose user_id vs. transaction-level; justify with interference risks (recipients can be common). (2) Baselines: weekly fraud base rate on transfers p0 = 0.0012, average fraudulent loss per incident L_f = $200, expected relative reduction = 20%; legitimate block cost C_fp = $1.50 per blocked legit transfer; expected block rate under treatment = 1.0% of legit transfers. (3) Traffic: 10M transfers/week, average 5 transfers per active user/week; ICC (cluster at user) = 0.02, average cluster size m = 5. (4) Guardrails: auth success rate, dispute rate within 7 days, P95 time-to-pay. (5) Stats: two-sided alpha 0.05, power 0.80; allow sequential monitoring (daily) with alpha spending; require pre-registration and an A/A test. Tasks: A) Choose randomization unit and explain spillover/contamination mitigations (e.g., recipient or graph clusters). B) Compute the minimum per-arm sample size (transfers) for detecting a 20% relative drop in fraud rate using a two-sample proportion Z-test; then inflate by the design effect DE = 1 + (m-1)*ICC. Show formulas and numeric results. C) Convert the detectable effect into expected weekly net dollars using: Net = (Fraud prevented * L_f) - (Incremental legit blocks * C_fp); state any additional assumptions you need and bound the estimate. D) Define primary, secondary, and guardrail metrics with precise denominators; specify slicing (e.g., by device novelty, account age). E) Propose a ramp plan and stopping rules under group sequential design (e.g., Pocock or O'Brien–Fleming), and how you’ll monitor production for post-launch drift.

Quick Answer: This question evaluates a data scientist's experiment-design and statistical-analysis competencies, including cluster-aware randomization, power/sample-size calculation, sequential monitoring, and translating detectable effects into net business impact for a real-time account-takeover prevention rule on a payment platform.

|Home/Analytics & Experimentation/PayPal
PayPal logo
PayPal
Oct 13, 2025, 9:49 PM
hardData ScientistTechnical ScreenAnalytics & Experimentation
4
0

Experiment Design Case: Real-time ATO Rule for PayPal/Venmo

Context: You are designing and analyzing an online experiment to estimate the net business impact of a new real-time ATO rule that blocks high-risk transfers. The rule reduces successful ATO fraud at the potential cost of blocking some legitimate transfers.

Inputs and constraints:

  • Randomization unit must avoid cross-over/interference. Choose user_id vs transaction-level; justify with interference risks (recipients can be common).
  • Baselines:
    • Weekly fraud base rate on transfers p0 = 0.0012
    • Average fraudulent loss per incident L_f = $200
    • Expected relative fraud reduction under treatment = 20%
    • Legitimate block cost C_fp = $1.50 per blocked legitimate transfer
    • Expected block rate under treatment = 1.0% of legitimate transfers
  • Traffic and clustering:
    • 10M transfers/week
    • Average 5 transfers per active user/week
    • ICC (clustered at user) = 0.02
    • Average cluster size m = 5
  • Guardrails: authentication success rate, dispute rate within 7 days, P95 time-to-pay
  • Stats: two-sided alpha = 0.05, power = 0.80; allow daily sequential monitoring with alpha spending; require pre-registration and an A/A test

Tasks: A) Choose the randomization unit and explain spillover/contamination mitigations (e.g., recipient or graph clusters).

B) Compute the minimum per-arm sample size (in transfers) to detect a 20% relative drop in fraud rate using a two-sample proportion Z-test; then inflate by the design effect DE = 1 + (m−1)·ICC. Show formulas and numeric results.

C) Convert the detectable effect into expected weekly net dollars using: Net = (Fraud prevented × L_f) − (Incremental legitimate blocks × C_fp). State additional assumptions and bound the estimate.

D) Define primary, secondary, and guardrail metrics with precise denominators; specify key slicing (e.g., by device novelty, account age).

E) Propose a ramp plan and stopping rules under a group sequential design (e.g., Pocock or O'Brien–Fleming), and how you’ll monitor production for post-launch drift.

Loading comments...