Evaluate Core Metrics for New Product Feature Launch
Scenario
You are a data scientist evaluating the health of a newly launched product feature in a consumer-facing app (e.g., investing/finance). The goal is to define what to monitor, diagnose issues if a key metric drops, and design an experiment to improve performance.
Tasks
-
Core Metrics and Computation
-
List the core metrics you would monitor post-launch and how each is computed.
-
Investigation of a 5% Drop
-
A key metric suddenly drops by 5% (assume a relative drop unless stated otherwise). List plausible root causes and specify the exact data you would pull to validate or rule out each cause.
-
Experiment Design for a UI Change
-
Design an experiment to test a UI change intended to improve the key metric. Specify:
-
Exposure unit and eligibility
-
Primary/secondary metrics and clear hypotheses
-
Randomization/assignment strategy
-
Duration and power/MDE assumptions
-
Success criteria and guardrail metrics
Hints: Think DAU/WAU and stickiness, conversion funnel definitions and denominators, exposure units, power calculations, and guardrail metrics (e.g., stability, latency, error rates, revenue risk).
Constraints & Assumptions
-
Preserve the scope, facts, inputs, and requested outputs from the prompt above.
-
If the prompt leaves a detail unspecified, state a reasonable assumption before relying on it.
-
Keep the answer interview-ready: concise enough to present, but concrete enough to implement or evaluate.
Clarifying Questions to Ask
-
Clarify the business objective, unit of analysis, time window, exposure definition, and primary metric.
-
State assumptions about instrumentation, randomization, sample size, and data quality.
-
Separate descriptive analysis from causal claims.
What a Strong Answer Covers
-
A metric framework with primary, guardrail, and diagnostic metrics.
-
A credible analysis or experiment design with clear assumptions and bias checks.
-
SQL/statistical logic for segmentation, variance, confidence, and data validation where relevant.
-
An actionable recommendation that explains trade-offs and next steps.
Follow-up Questions
-
What sanity checks would you run before trusting the result?
-
How would you handle novelty effects, seasonality, or selection bias?
-
What decision would you make if metrics disagree?