Practice defining a North Star Metric and experiment for a Microsoft product using Teams as the example. The solution defines Weekly Actively Collaborating Users, leading indicators with SQL sketches, telemetry requirements, randomization unit, experiment metrics, guardrails, power, and analysis.
##### Question
Choose a Microsoft product you admire (e.g., Teams, Outlook, Azure AI) and:
Define its north star metric and explain why it captures long-term customer value.
List two supporting leading indicators and the SQL-level data you would track.
Describe an experiment you would run to improve the north star metric, including success criteria.
Quick Answer: Practice defining a North Star Metric and experiment for a Microsoft product using Teams as the example. The solution defines Weekly Actively Collaborating Users, leading indicators with SQL sketches, telemetry requirements, randomization unit, experiment metrics, guardrails, power, and analysis.
mediumProduct ManagerHR ScreenProduct / Decision Making
17
0
Product Metrics Prompt: North Star Metric and Experiment Design
Choose a Microsoft product you know, such as Teams, Outlook, Azure AI, OneDrive, or Copilot. Assume you have anonymized event-level telemetry and can run controlled experiments.
Constraints & Assumptions
Pick one product and clearly state the customer job to be done.
Define one precise North Star Metric with a measurement window and qualifying actions.
Include leading indicators with SQL-level data requirements.
Design one experiment to improve the North Star Metric with randomization, metrics, guardrails, power, and duration.
Clarifying Questions to Ask Guidance
Which Microsoft product should I use?
Is the target user an individual, team, tenant, developer, or enterprise admin?
What business goal matters most: engagement, retention, productivity, revenue, or expansion?
What telemetry tables and privacy constraints are available?
Can we randomize by user, tenant, team, or workspace?
Part 1 - Product and North Star Metric
Pick a product, state the customer job to be done, and define a single North Star Metric.
What This Part Should Cover Guidance
Customer job, target user, and why the product creates long-term value.
Metric name, formula, time window, inclusion criteria, exclusions, and threshold.
Why the metric is not vanity usage and why it correlates with customer value.
Potential failure modes or gaming risks.
Part 2 - Leading Indicators and SQL-Level Data
List two leading indicators that support the North Star Metric. Define each metric and specify the tables or columns you would track, with SQL sketches if useful.
What This Part Should Cover Guidance
Input metrics that teams can move before the North Star changes.
Event names, user IDs, tenant IDs, timestamps, entity IDs, and action thresholds.
SQL-level logic for distinct users, qualifying actions, collaborators, sessions, or activation.
Data quality concerns such as bots, duplicate events, timezone, and tenant-level aggregation.
Part 3 - Experiment Design
Describe one experiment to improve the North Star Metric.
What This Part Should Cover Guidance
Hypothesis and treatment.
Unit of randomization and eligibility.
Primary metric, supporting metrics, and guardrails.
Sample size, minimum detectable effect, duration, and novelty effects.
Analysis plan and decision rule.
What a Strong Answer Covers Guidance
A strong answer defines a measurable North Star tied to durable customer value, identifies leading indicators with implementable telemetry, and designs an experiment that respects the product's collaboration or enterprise structure.
Follow-up Questions Guidance
Why not use DAU or time spent as the North Star?
How would your metric handle enterprise tenants of very different sizes?
What would you do if the experiment improves usage but hurts satisfaction?
How would you prevent bots or system events from inflating the metric?
What if user-level randomization causes spillovers?