North Star Metrics & Experiment Design
Overview
This question evaluates a product manager's competency in product sense, metric definition, analytics instrumentation, and controlled experiment design by requiring a North Star metric, supporting leading indicators with SQL-level data considerations, and a randomized experiment to improve the metric.
Microsoft
Jul 4, 2025, 8:28 PM
Product Manager
HR Screen
Product / Decision Making
11
0
Product Sense: Define a North Star Metric and Experiment (Microsoft)
Context
- Choose any Microsoft product you know (e.g., Teams, Outlook, Azure AI).
- Assume you have access to anonymized, event-level telemetry (users, teams/tenants, events, subscriptions) and can run controlled experiments.
Tasks
- Pick one Microsoft product and briefly state the customer job-to-be-done.
-
Define a single north star metric (NSM):
- Name it and give a precise, measurable definition (window, qualifying actions/thresholds).
- Explain why it reflects long-term customer value.
-
List two leading indicators that support/drive the NSM:
- Define each metric clearly.
- Specify the SQL-level data you would track (tables/columns) and, if helpful, short SQL sketches.
-
Describe one experiment to improve the NSM:
- Hypothesis and treatment.
- Unit of randomization and eligibility.
- Success criteria: primary metric, supporting/guardrail metrics, sample size/MDE assumptions, and duration.
Solution
Login required
Show