Product Metrics and Debugging Scenarios
You are a PM candidate evaluating data, metrics, and operational plans for large-scale consumer products. Answer three prompts that test metrics design, data reasoning, and debugging under ambiguity.
Constraints & Assumptions
-
Assume a modern mobile app ecosystem with global users and standard privacy constraints.
-
Be explicit about metric definitions, baselines, segments, and guardrails.
-
Use examples and formulas where helpful, but do not over-index on math at the expense of product judgment.
-
For debugging, start by verifying measurement before proposing product fixes.
Clarifying Questions to Ask
-
Which user segment, geography, or platform should I focus on?
-
Is the interviewer looking for a PM-level framework or deep data-science detail?
-
Should I assume first-party telemetry is available and privacy-consented?
-
What is the exact engagement metric or business goal being optimized?
Part 1 - Real-Time Traffic Data for Maps
What data should a maps product collect to power and improve real-time traffic?
What This Part Should Cover
-
Probe telemetry, road graph data, historical speed profiles, incidents, weather, sensors, and ground truth.
-
Map matching, aggregation windows, latency requirements, sparse-data handling, and privacy protections.
-
Success metrics such as ETA error, incident detection quality, reroute quality, and user satisfaction.
Part 2 - Facebook Live Metrics
Which north-star and supporting metrics would you track for Facebook Live?
What This Part Should Cover
-
A north-star metric that balances watch time, meaningful engagement, and quality.
-
Viewer funnel, creator supply, quality of experience, safety, retention, and monetization metrics.
-
Guardrails for notification fatigue, policy violations, rebuffering, crashes, and creator concentration.
Part 3 - Messenger Engagement Drop
Messenger engagement suddenly drops. Detail a step-by-step debugging plan.
What This Part Should Cover
-
Measurement validation, scope definition, segmentation, release correlation, incident review, and funnel analysis.
-
Checks for client health, server errors, push notifications, ranking, spam rules, permissions, and external events.
-
Mitigation, communication, rollout, monitoring, and postmortem mechanisms.
What a Strong Answer Covers
-
Precise metric definitions and a clear causal debugging process.
-
Trade-offs between engagement, quality, safety, and privacy.
-
Practical use of segmentation, guardrails, experiments, and operational response.
Follow-up Questions
-
How would you distinguish a real product decline from an instrumentation issue?
-
What privacy constraints matter for live traffic data?
-
When would watch time be a bad north-star metric?
-
What rollback would you try first for the Messenger drop?
-
How would you communicate uncertainty to leadership during the incident?