Design and Measurement: Metrics, Instrumentation, and Experiment Plan

Context (added for clarity) You are shipping "Freshness Boost," a change to the main short‑form video feed ranking that upweights newly uploaded videos. The goal is to increase engagement without harming reliability, content quality, or creator fairness.

Tasks

1. Define primary outcome metrics. For each, give a precise definition including events, time windows, and denominators.
2. Define secondary metrics (with precise definitions).
3. Define guardrail metrics (with precise definitions).
4. Outline an instrumentation and data‑quality plan: event schema, identifiers, sessionization, clocks/time zones, deduplication, and QA checks.
5. Design an experiment or observational evaluation: unit of randomization, sampling and ramp, exposure definition, duration, statistical power and expected effect size, how you will handle seasonality and heterogeneity.
6. Discuss pitfalls (metric gaming, selection bias, Simpson's paradox) and mitigations.
7. Explain how you would monitor and alert on regressions in both product metrics and data quality.