Define and measure project metrics

Quick Overview

This question evaluates a software engineer's competency in defining outcome, secondary and guardrail metrics, designing instrumentation and data-quality plans, and structuring valid experiments or observational analyses within the Analytics & Experimentation domain.

Design and Measurement: Metrics, Instrumentation, and Experiment Plan

Context (added for clarity) You are shipping "Freshness Boost," a change to the main short‑form video feed ranking that upweights newly uploaded videos. The goal is to increase engagement without harming reliability, content quality, or creator fairness.

Tasks

Define primary outcome metrics. For each, give a precise definition including events, time windows, and denominators.
Define secondary metrics (with precise definitions).
Define guardrail metrics (with precise definitions).
Outline an instrumentation and data‑quality plan: event schema, identifiers, sessionization, clocks/time zones, deduplication, and QA checks.
Design an experiment or observational evaluation: unit of randomization, sampling and ramp, exposure definition, duration, statistical power and expected effect size, how you will handle seasonality and heterogeneity.
Discuss pitfalls (metric gaming, selection bias, Simpson's paradox) and mitigations.
Explain how you would monitor and alert on regressions in both product metrics and data quality.

Define and measure project metrics

Quick Overview

Define and measure project metrics

Design and Measurement: Metrics, Instrumentation, and Experiment Plan

Write your answer