Design KYC experiment amid crypto volatility
Overview
The prompt evaluates proficiency in experiment design and causal inference under external market shocks, covering competencies in metric definition and guardrails, sample size and sequential monitoring planning, and managing compliance and operational heterogeneity for an Analytics & Experimentation Data Scientist role.
A/B Test Design: Improve Mobile KYC Completion During High Market Volatility
Context: You are analyzing a mobile onboarding funnel where the KYC (Know Your Customer) completion rate declined by 8% week-over-week during a period of high BTC price volatility. Design a rigorous A/B test to increase KYC completion without increasing fraud risk, accounting for market shocks and operational constraints.
Specify the following:
(a) Unit of randomization (user/session/geo) and why.
(b) Primary metric and at least two guardrails (e.g., fraud approval rate, time-to-approve, support tickets per 1k signups).
(c) How to neutralize exogenous market shocks (e.g., include BTC/ETH price and realized volatility as covariates with CUPED or run a Difference‑in‑Differences against a stable geo holdout).
(d) Sample size, expected duration, and a sequential monitoring plan that controls Type I error (e.g., alpha‑spending or group‑sequential boundaries).
(e) How to handle country‑specific compliance step heterogeneity and app store review delays.
(f) A precise pre‑registration: hypotheses, data exclusions, stopping rules, and analysis plan.
Solution
Login required