Design KYC experiment amid crypto volatility

Quick Overview

The prompt evaluates proficiency in experiment design and causal inference under external market shocks, covering competencies in metric definition and guardrails, sample size and sequential monitoring planning, and managing compliance and operational heterogeneity for an Analytics & Experimentation Data Scientist role.

Design KYC experiment amid crypto volatility

Company: Coinbase

Role: Data Scientist

Category: Analytics & Experimentation

Difficulty: hard

Interview Round: Technical Screen

You're the analyst for a crypto exchange’s mobile onboarding funnel. KYC completion rate dropped 8% WoW during a period of high BTC volatility. Design an A/B test to improve completion without increasing fraud. Specify: a) unit of randomization (user/session/geo) and why, b) primary metric and at least two guardrails (e.g., fraud approval rate, time-to-approve, support tickets per 1k signups), c) how to neutralize exogenous market shocks (e.g., include BTC/ETH price and realized volatility as covariates with CUPED or run a Difference‑in‑Differences against a stable geo holdout), d) sample size, expected duration, and a sequential monitoring plan that controls Type I error (e.g., alpha‑spending or group‑sequential boundaries), e) country‑specific compliance step heterogeneity and app store review delays, and f) a precise pre‑registration (hypotheses, data exclusions, stopping rules, analysis plan).

Quick Answer: The prompt evaluates proficiency in experiment design and causal inference under external market shocks, covering competencies in metric definition and guardrails, sample size and sequential monitoring planning, and managing compliance and operational heterogeneity for an Analytics & Experimentation Data Scientist role.

|Home/Analytics & Experimentation/Coinbase
Coinbase logo
Coinbase
Oct 13, 2025, 9:49 PM
hardData ScientistTechnical ScreenAnalytics & Experimentation
4
0

A/B Test Design: Improve Mobile KYC Completion During High Market Volatility

Context: You are analyzing a mobile onboarding funnel where the KYC (Know Your Customer) completion rate declined by 8% week-over-week during a period of high BTC price volatility. Design a rigorous A/B test to increase KYC completion without increasing fraud risk, accounting for market shocks and operational constraints.

Specify the following:

(a) Unit of randomization (user/session/geo) and why.

(b) Primary metric and at least two guardrails (e.g., fraud approval rate, time-to-approve, support tickets per 1k signups).

(c) How to neutralize exogenous market shocks (e.g., include BTC/ETH price and realized volatility as covariates with CUPED or run a Difference‑in‑Differences against a stable geo holdout).

(d) Sample size, expected duration, and a sequential monitoring plan that controls Type I error (e.g., alpha‑spending or group‑sequential boundaries).

(e) How to handle country‑specific compliance step heterogeneity and app store review delays.

(f) A precise pre‑registration: hypotheses, data exclusions, stopping rules, and analysis plan.

Loading comments...