Product Metrics Prompt: North Star Metric and Experiment Design
Choose a Microsoft product you know, such as Teams, Outlook, Azure AI, OneDrive, or Copilot. Assume you have anonymized event-level telemetry and can run controlled experiments.
Constraints & Assumptions
-
Pick one product and clearly state the customer job to be done.
-
Define one precise North Star Metric with a measurement window and qualifying actions.
-
Include leading indicators with SQL-level data requirements.
-
Design one experiment to improve the North Star Metric with randomization, metrics, guardrails, power, and duration.
Clarifying Questions to Ask
-
Which Microsoft product should I use?
-
Is the target user an individual, team, tenant, developer, or enterprise admin?
-
What business goal matters most: engagement, retention, productivity, revenue, or expansion?
-
What telemetry tables and privacy constraints are available?
-
Can we randomize by user, tenant, team, or workspace?
Part 1 - Product and North Star Metric
Pick a product, state the customer job to be done, and define a single North Star Metric.
What This Part Should Cover
-
Customer job, target user, and why the product creates long-term value.
-
Metric name, formula, time window, inclusion criteria, exclusions, and threshold.
-
Why the metric is not vanity usage and why it correlates with customer value.
-
Potential failure modes or gaming risks.
Part 2 - Leading Indicators and SQL-Level Data
List two leading indicators that support the North Star Metric. Define each metric and specify the tables or columns you would track, with SQL sketches if useful.
What This Part Should Cover
-
Input metrics that teams can move before the North Star changes.
-
Event names, user IDs, tenant IDs, timestamps, entity IDs, and action thresholds.
-
SQL-level logic for distinct users, qualifying actions, collaborators, sessions, or activation.
-
Data quality concerns such as bots, duplicate events, timezone, and tenant-level aggregation.
Part 3 - Experiment Design
Describe one experiment to improve the North Star Metric.
What This Part Should Cover
-
Hypothesis and treatment.
-
Unit of randomization and eligibility.
-
Primary metric, supporting metrics, and guardrails.
-
Sample size, minimum detectable effect, duration, and novelty effects.
-
Analysis plan and decision rule.
What a Strong Answer Covers
A strong answer defines a measurable North Star tied to durable customer value, identifies leading indicators with implementable telemetry, and designs an experiment that respects the product's collaboration or enterprise structure.
Follow-up Questions
-
Why not use DAU or time spent as the North Star?
-
How would your metric handle enterprise tenants of very different sizes?
-
What would you do if the experiment improves usage but hurts satisfaction?
-
How would you prevent bots or system events from inflating the metric?
-
What if user-level randomization causes spillovers?