Select and prioritize metrics with guardrails

Q: Select and prioritize metrics with guardrails

This question evaluates a candidate's ability to design a robust metrics framework for product experiments, covering selection and normalization of success metrics, feature-only sizing and ecosystem metrics, explicit cannibalization reads, guardrails, multiple-testing control, and back-of-the-envelope LTV/ARPDAU impact estimation.

Q: How do I approach Analytics & Experimentation interview questions?

Analytics & Experimentation questions require understanding of core concepts and practice. PracHub provides solutions with explanations to help you master analytics & experimentation interviews.

Q: What difficulty level is this interview question?

This is a medium difficulty Analytics & Experimentation question, commonly asked during Technical Screen rounds at Meta.

Q: What role is this question designed for?

This question is commonly asked for Data Scientist candidates at Meta during technical interviews.

Question

Design a Metrics Framework for a New Groups Stories Feature

Context

You are evaluating a new Groups Stories feature whose goal is to increase meaningful engagement while avoiding harm to the broader product ecosystem. Propose a metrics framework for experimentation and rollout gating, including success metrics, guardrails, and ecosystem checks.

Tasks

Define three top-line goal (success) metrics and specify the normalization base (per DAU, per viewer, or per poster) and why.
List feature-only sizing metrics (e.g., treatment-only CTR, creation rate) to estimate volumes if launched.
List ecosystem metrics to catch second-order effects (e.g., sessions, retention, revenue).
Specify explicit cannibalization reads on other surfaces (e.g., Feed, Marketplace, Pages).
Define guardrails (e.g., latency, crash, report rate, cancellations) and sanity checks.
Specify a multiple-testing control method (e.g., Holm–Bonferroni vs Benjamini–Hochberg) and when you’d prefer each.
Show a back-of-the-envelope calculation for LTV and ARPDAU impact from a +1% DAU movement.

Finally, pick a minimal gating set (three metrics that must be green to roll out to 100%) and explain why the other metrics are secondary.

Select and prioritize metrics with guardrails

Quick Overview

Select and prioritize metrics with guardrails

Design a Metrics Framework for a New Groups Stories Feature

Context

Tasks

Write your answer