Analyze Success Metrics and Diagnose Crypto Feature Issues
Post-Launch Evaluation: Crypto Trading Feature
Context
You are a Data Scientist evaluating the post-launch performance of a crypto-trading feature integrated into an existing payments app. The goal is to grow sustainable trading usage and revenue while maintaining trust, compliance, and reliability.
Tasks
-
Define the key success metrics to track after launch. Include acquisition/activation, engagement, monetization, risk/compliance, reliability, and customer satisfaction, plus a clear north-star metric and guardrails.
-
Transaction volume drops after release. Outline a structured root-cause diagnosis plan: what to look at, how to segment, which analyses to run, and how to isolate causality vs. correlation.
-
Propose two concrete, data-driven product improvements for the crypto feature. For each, state the hypothesis, the change, success metrics, and how you would test it (e.g., A/B or staged rollout).
Constraints & Assumptions
-
Preserve the scope, facts, inputs, and requested outputs from the prompt above.
-
If the prompt leaves a detail unspecified, state a reasonable assumption before relying on it.
-
Keep the answer interview-ready: concise enough to present, but concrete enough to implement or evaluate.
Clarifying Questions to Ask
-
Clarify the business objective, unit of analysis, time window, exposure definition, and primary metric.
-
State assumptions about instrumentation, randomization, sample size, and data quality.
-
Separate descriptive analysis from causal claims.
What a Strong Answer Covers
-
A metric framework with primary, guardrail, and diagnostic metrics.
-
A credible analysis or experiment design with clear assumptions and bias checks.
-
SQL/statistical logic for segmentation, variance, confidence, and data validation where relevant.
-
An actionable recommendation that explains trade-offs and next steps.
Follow-up Questions
-
What sanity checks would you run before trusting the result?
-
How would you handle novelty effects, seasonality, or selection bias?
-
What decision would you make if metrics disagree?