Design Identity-Trust A/B Test
Company: Coinbase
Role: Data Scientist
Category: Analytics & Experimentation
Difficulty: medium
Interview Round: Onsite
You are interviewing for a Data Scientist role on an Identity & Trust team at a consumer product company. The team wants to launch a feature that strengthens identity verification and adds trust signals such as a verified badge, additional account checks, or warnings on suspicious profiles.
How would you design an A/B test to evaluate this launch? Discuss:
- the unit of randomization and whether user-level randomization is valid,
- primary success metrics and guardrail metrics,
- how to measure trust when adverse events are rare,
- interference or network effects if treated and control users interact,
- likely sources of bias or confounding,
- how you would size the test and interpret results if fraud decreases but engagement or conversion also declines.
Quick Answer: This question evaluates a candidate's competency in experimental design, causal inference, metric selection, sample sizing, bias and confounding identification, interference/network effects, and measurement of low-base-rate adverse events within A/B testing for identity and trust features for a Data Scientist role.