Define and validate product metrics
End-to-End Analytics Design for a New Product Feature
Context: You are the data engineer partnering with product, engineering, and data science to launch a new user-facing feature in a large-scale consumer app. You must define success, design instrumentation, produce reliable KPIs, and operate the metrics in both streaming and batch.
Tasks:
-
Define the core business metrics:
-
North-star metric(s) that represent durable user/business value.
-
Guardrail metrics to protect reliability, performance, quality, and adjacent business outcomes.
-
Instrumentation and tracking plan:
-
Specify events, schemas, identifiers, and when each event fires.
-
Include experiment fields, versions, deduplication, and sampling choices.
-
From raw events to KPIs:
-
Describe transformations/modeling steps to derive daily/weekly KPIs from raw logs.
-
Metric quality and operations:
-
How to sanity-check correctness pre/post launch.
-
How to set thresholds and detect/investigate anomalies.
-
Batch vs. streaming views:
-
How definitions and numbers can differ; watermarks, late data, deduplication, approximations.
-
Communication:
-
How to document caveats, data latency, and metric maturity to stakeholders.
Constraints & Assumptions
-
Preserve the scope, facts, inputs, and requested outputs from the prompt above.
-
If the prompt leaves a detail unspecified, state a reasonable assumption before relying on it.
-
Keep the answer interview-ready: concise enough to present, but concrete enough to implement or evaluate.
Clarifying Questions to Ask
-
Clarify the business objective, unit of analysis, time window, exposure definition, and primary metric.
-
State assumptions about instrumentation, randomization, sample size, and data quality.
-
Separate descriptive analysis from causal claims.
What a Strong Answer Covers
-
A metric framework with primary, guardrail, and diagnostic metrics.
-
A credible analysis or experiment design with clear assumptions and bias checks.
-
SQL/statistical logic for segmentation, variance, confidence, and data validation where relevant.
-
An actionable recommendation that explains trade-offs and next steps.
Follow-up Questions
-
What sanity checks would you run before trusting the result?
-
How would you handle novelty effects, seasonality, or selection bias?
-
What decision would you make if metrics disagree?