How would you A/B test first trade rate?
Company: Citi
Role: Data Scientist
Category: Analytics & Experimentation
Difficulty: medium
Interview Round: Technical Screen
Coinbase wants to increase **new user first trade rate**.
Design an experiment (A/B test) to evaluate a product change intended to increase the probability that a newly signed-up user places their **first trade**.
In your answer, cover:
1. **Hypothesis** and what user behavior you are trying to change.
2. **Primary metric** (pick a clear definition, e.g., “first trade within 7 days of signup”) and at least 2 **secondary/guardrail metrics** (risk, compliance, user harm, revenue quality).
3. **Experiment unit & randomization** (user/account/device), eligibility rules, and when a user is considered “exposed”.
4. Key threats to validity (selection bias, learning/novelty effects, interference, instrumentation issues, sample ratio mismatch).
5. **Power/MDE** approach (how you would size the test, what baseline inputs you need).
6. What you would do if the result is **not statistically significant** but directional, or if key guardrails regress.
Quick Answer: This question evaluates A/B testing and experimentation design skills—metric definition, unit and randomization choice, threat-to-validity identification, and power/MDE calculation—for a Data Scientist in the Analytics & Experimentation domain.