Define and validate product metrics
Overview
This question evaluates a data engineer's competency in defining and validating product metrics, encompassing instrumentation design, event schemas and identifiers, transformations from raw events to KPIs, metric quality checks, and operational differences between streaming and batch pipelines.
Meta
Jul 15, 2025, 12:00 AM
Data Engineer
Onsite
Analytics & Experimentation
0
0
End-to-End Analytics Design for a New Product Feature
Context: You are the data engineer partnering with product, engineering, and data science to launch a new user-facing feature in a large-scale consumer app. You must define success, design instrumentation, produce reliable KPIs, and operate the metrics in both streaming and batch.
Tasks:
-
Define the core business metrics:
- North-star metric(s) that represent durable user/business value.
- Guardrail metrics to protect reliability, performance, quality, and adjacent business outcomes.
-
Instrumentation and tracking plan:
- Specify events, schemas, identifiers, and when each event fires.
- Include experiment fields, versions, deduplication, and sampling choices.
-
From raw events to KPIs:
- Describe transformations/modeling steps to derive daily/weekly KPIs from raw logs.
-
Metric quality and operations:
- How to sanity-check correctness pre/post launch.
- How to set thresholds and detect/investigate anomalies.
-
Batch vs. streaming views:
- How definitions and numbers can differ; watermarks, late data, deduplication, approximations.
-
Communication:
- How to document caveats, data latency, and metric maturity to stakeholders.
Solution
Login required
Show