Design tests to measure latency impact

You are a Data Scientist supporting a large consumer product (e.g., YouTube). A team ships a change intended to reduce client-side latency.

Part A — Experiment design (latency → business impact)

1. Propose an A/B test to estimate the causal impact of reduced latency on user engagement.
2. Define:
   • Primary metric(s) (at least one), and why they are sensitive to latency.
   • Diagnostic metrics to understand where changes come from.
   • Guardrail metrics (quality/revenue/reliability) to avoid shipping regressions.
3. Describe:
   • Unit of randomization (user/device/session) and why.
   • Power/MDE approach and what variance drivers you’d account for.
   • Key threats to validity (e.g., novelty, network effects, logging changes, missing data).

Part B — Variance reduction

What techniques would you use to reduce variance (or improve sensitivity) in this experiment? Explain when each is appropriate (e.g., CUPED, stratification, winsorization, triggering, clustered standard errors).

Part C — Diagnosing a ratio metric change

Suppose leadership cares about a ratio metric like CTR = clicks / impressions, or conversion rate = purchases / sessions.

1. If the ratio moved by +0.3%, outline a structured approach to diagnose why it changed.
2. Explain how you would decompose the change into numerator/denominator effects and guard against misleading interpretations (e.g., Simpson’s paradox).

Part D — When randomization is not possible (propensity score matching)

Assume you cannot randomize the latency change (e.g., it rolled out selectively due to infra constraints). You only observe that some users experienced lower latency than others.

1. Describe how you would use propensity score matching (PSM) to estimate the impact of latency on engagement.
2. List the assumptions required for PSM to be credible and how you would validate/sensitivity-test them.

Assumptions

• You can define a pre-period to compute baselines/covariates.
• Logging is available for latency, exposure, and key engagement outcomes.
• Timezone: use a consistent reporting timezone (e.g., UTC) for daily metrics to avoid boundary artifacts.