Design an Identity Trust Experiment
Company: Coinbase
Role: Data Scientist
Category: Analytics & Experimentation
Difficulty: medium
Interview Round: Onsite
You are joining an Identity & Trust team at a consumer marketplace. The team wants to launch a new identity-verification badge that appears on seller profiles and listing pages. The product goal is to increase buyer trust and reduce fraud, but the verification flow may create extra friction for sellers and could reduce listing supply.
Design an A/B test to evaluate this feature. In your answer, address the following:
- What is the main product hypothesis?
- What should be the unit of randomization: seller, buyer, session, or something else?
- What are the primary success metrics, secondary metrics, and guardrail metrics?
- How would you handle interference or spillover effects, since buyers interact with sellers and trust signals may affect both sides of the marketplace?
- How would you segment the analysis across new vs. existing sellers, high-risk vs. low-risk geographies, and heavy vs. light buyers?
- How would you estimate sample size or minimum detectable effect, and when would CUPED or triggered analysis be useful?
- What sources of bias or invalid inference would you watch for, such as selection bias, Simpson's paradox, sample ratio mismatch, delayed fraud labels, or novelty effects?
- How would you interpret results if fraud-related metrics improve but conversion or revenue declines?
Quick Answer: This question evaluates a candidate's abilities in experimental design, causal inference, metric definition, interference mitigation, sample size estimation, and bias detection as applied to marketplace identity and trust features.