Define metrics and design experiments for notifications
Overview
This question evaluates a candidate's competency in analytics and experimentation, including metric specification (North Star, drivers, guardrails), causal inference and bias correction, power and MDE calculations, A/B and cluster-randomized experiment design, attribution and instrumentation, and decision-framework construction for notification systems. It is in the Analytics & Experimentation domain and is commonly asked because it probes both conceptual understanding and practical application of rigorous measurement, pre-launch sizing, investment justification, interference detection, and trade-off management in real-world product experiments.
Analytics/Experimentation Case: "Your friend is attending a local event—join them?"
You are evaluating a proposed notification: "Your friend is attending a local event—join them?" Define what “high quality” means for this notification system and design a rigorous plan to estimate impact, justify investment, and test pre/post launch.
Answer all parts:
- Metrics and Definitions
-
Specify one North Star metric, three driver metrics, and three guardrail metrics with exact, unit-consistent formulas. Include:
- Denominator definitions.
- Attribution windows.
- Whether each metric is a leading or lagging indicator.
- Pre-build Adoption and Opportunity Sizing
-
Using historical event data, estimate adoption. The naive estimator averages, across past events, the percent of sign-ups who arrived with ≥1 friend.
- List at least four biases (e.g., event-type confounding, seasonality, selection on observables, friendship-inference error).
- Propose a corrected estimator (e.g., stratified weighting or matched cohorts).
- Show how you would compute a 95% confidence interval (CI) for expected daily notification opens.
- Investment Justification
- Derive a break-even formula linking expected incremental opens, sessions, and retention to dev-weeks and infra cost.
- State the minimum detectable effect (MDE) required to proceed.
- Pre-launch Evidence Plan
- Describe how to use concept tests/surveys and analog-notification benchmarks to predict CTR uplift, and how to adjust for response bias and population mismatch.
- A/B Experiment Design
- Define the unit of randomization, primary and guardrail metrics, sample sizing, and when cluster randomization (e.g., by event or geography) is required due to network effects.
- Specify how to detect interference and what to do if contamination is observed.
- Decision Framework
- If an experiment shows time-on-site increases but CTR decreases, outline a metric hierarchy or utility function to make a go/no-go call, including constraints on unsubscribe/complaint rates and notification volume.
Solution
Login required