Select and prioritize metrics with guardrails
Overview
This question evaluates a candidate's ability to design a robust metrics framework for product experiments, covering selection and normalization of success metrics, feature-only sizing and ecosystem metrics, explicit cannibalization reads, guardrails, multiple-testing control, and back-of-the-envelope LTV/ARPDAU impact estimation.
Design a Metrics Framework for a New Groups Stories Feature
Context
You are evaluating a new Groups Stories feature whose goal is to increase meaningful engagement while avoiding harm to the broader product ecosystem. Propose a metrics framework for experimentation and rollout gating, including success metrics, guardrails, and ecosystem checks.
Tasks
- Define three top-line goal (success) metrics and specify the normalization base (per DAU, per viewer, or per poster) and why.
- List feature-only sizing metrics (e.g., treatment-only CTR, creation rate) to estimate volumes if launched.
- List ecosystem metrics to catch second-order effects (e.g., sessions, retention, revenue).
- Specify explicit cannibalization reads on other surfaces (e.g., Feed, Marketplace, Pages).
- Define guardrails (e.g., latency, crash, report rate, cancellations) and sanity checks.
- Specify a multiple-testing control method (e.g., Holm–Bonferroni vs Benjamini–Hochberg) and when you’d prefer each.
- Show a back-of-the-envelope calculation for LTV and ARPDAU impact from a +1% DAU movement.
Finally, pick a minimal gating set (three metrics that must be green to roll out to 100%) and explain why the other metrics are secondary.
Solution
Login required