##### Question You are a Data Scientist supporting a large consumer product (e.g., YouTube). Engineering ships a change intended to reduce **client-side / video-start latency** by roughly 100 ms for some users, but the new code path **might increase the error rate and change buffering behavior**. Design and analyze the measurement program for this change. 1. **Experiment design (latency → business impact).** Propose an **A/B test** to estimate the causal impact of reduced latency on user engagement. Define: - A **hypothesis** and a **primary metric** (one decision metric), explaining why it is sensitive to latency and the sensitivity-vs-business-relevance tradeoff. - **Diagnostic metrics** to localize where any change comes from (e.g., funnel steps, latency percentiles). - **Guardrail metrics** (quality / reliability / revenue) to avoid shipping a regression — in particular playback error rate and rebuffering, since the new stack may be unstable. 2. **Unit of randomization & interference.** Choose user vs. device vs. session/request-level randomization and justify it. Explain how you avoid contamination and interference (sticky bucketing, CDN/cache routing, cross-device spillover). 3. **Power / MDE / duration.** Describe how you would estimate sample size and runtime: what inputs you need, how you set the MDE, and how you handle **heavy-tailed watch-time**. Note relevant variance drivers (user heterogeneity, seasonality, outliers). 4. **Variance reduction.** Give at least two techniques to reduce variance / improve sensitivity and explain when each is appropriate (e.g., CUPED, stratification, triggering, winsorization, clustered standard errors). 5. **Analysis plan & conflicting movements.** Specify the estimator, how you handle multiple metrics and heterogeneity (e.g., WiFi vs. cellular), and how you decide when results conflict — e.g., watch time is up but the error guardrail is also up. 6. **Diagnosing a ratio-metric change.** Suppose leadership tracks a **ratio metric** such as **CTR = clicks / impressions** or **conversion rate = purchases / sessions**, and it moved by +0.3%. Outline a structured approach to diagnose *why* it changed, decompose the move into numerator vs. denominator effects, and guard against misleading interpretations such as **Simpson’s paradox**. 7. **When randomization is not possible (propensity score matching).** Now assume you **cannot randomize** the latency change (e.g., it rolled out selectively due to infra constraints) and you only observe that some users experienced lower latency than others. Describe how you would use **propensity score matching (PSM)** to estimate the impact, list the assumptions PSM requires, and explain how you would validate / sensitivity-test them. **Assumptions** - Users are global; traffic varies by time-of-day and day-of-week, and latency effects may be heterogeneous by network type (WiFi vs. cellular). - You can define a pre-period to compute baselines / covariates. - Logging is available for latency, exposure, errors, buffering, and key engagement outcomes. - Use a consistent reporting timezone (e.g., UTC) for daily metrics to avoid boundary artifacts.