Define and measure project metrics

Quick Overview

This question evaluates a software engineer's competency in defining outcome, secondary and guardrail metrics, designing instrumentation and data-quality plans, and structuring valid experiments or observational analyses within the Analytics & Experimentation domain.

Define and measure project metrics

Company: TikTok

Role: Software Engineer

Category: Analytics & Experimentation

Difficulty: hard

Interview Round: Technical Screen

Define and measure project metrics: For a project of your choice, specify primary outcome metrics, secondary metrics, and guardrails. Precisely define each metric (events, windows, denominators), outline an instrumentation and data-quality plan, and design an experiment or observational evaluation covering sampling, statistical power, expected effect size, seasonality, and heterogeneity. Discuss pitfalls such as metric gaming, selection bias, and Simpson's paradox, and explain how you would monitor and alert on regressions.

Quick Answer: This question evaluates a software engineer's competency in defining outcome, secondary and guardrail metrics, designing instrumentation and data-quality plans, and structuring valid experiments or observational analyses within the Analytics & Experimentation domain.

|Home/Analytics & Experimentation/TikTok
TikTok logo
TikTok
Sep 6, 2025, 12:00 AM
hardSoftware EngineerTechnical ScreenAnalytics & Experimentation
4
0

Design and Measurement: Metrics, Instrumentation, and Experiment Plan

Context (added for clarity) You are shipping "Freshness Boost," a change to the main short‑form video feed ranking that upweights newly uploaded videos. The goal is to increase engagement without harming reliability, content quality, or creator fairness.

Tasks

  1. Define primary outcome metrics. For each, give a precise definition including events, time windows, and denominators.
  2. Define secondary metrics (with precise definitions).
  3. Define guardrail metrics (with precise definitions).
  4. Outline an instrumentation and data‑quality plan: event schema, identifiers, sessionization, clocks/time zones, deduplication, and QA checks.
  5. Design an experiment or observational evaluation: unit of randomization, sampling and ramp, exposure definition, duration, statistical power and expected effect size, how you will handle seasonality and heterogeneity.
  6. Discuss pitfalls (metric gaming, selection bias, Simpson's paradox) and mitigations.
  7. Explain how you would monitor and alert on regressions in both product metrics and data quality.
Loading comments...