Design and Measurement: Metrics, Instrumentation, and Experiment Plan
Context (added for clarity)
You are shipping "Freshness Boost," a change to the main short‑form video feed ranking that upweights newly uploaded videos. The goal is to increase engagement without harming reliability, content quality, or creator fairness.
Tasks
-
Define primary outcome metrics. For each, give a precise definition including events, time windows, and denominators.
-
Define secondary metrics (with precise definitions).
-
Define guardrail metrics (with precise definitions).
-
Outline an instrumentation and data‑quality plan: event schema, identifiers, sessionization, clocks/time zones, deduplication, and QA checks.
-
Design an experiment or observational evaluation: unit of randomization, sampling and ramp, exposure definition, duration, statistical power and expected effect size, how you will handle seasonality and heterogeneity.
-
Discuss pitfalls (metric gaming, selection bias, Simpson's paradox) and mitigations.
-
Explain how you would monitor and alert on regressions in both product metrics and data quality.