##### Question You are a Data Scientist supporting a large consumer product (e.g., YouTube). Engineering ships a change intended to reduce **client-side / video-start latency** by roughly 100 ms for some users, but the new code path **might increase the error rate and change buffering behavior**. Design and analyze the measurement program for this change. 1. **Experiment design (latency → business impact).** Propose an **A/B test** to estimate the causal impact of reduced latency on user engagement. Define: - A **hypothesis** and a **primary metric** (one decision metric), explaining why it is sensitive to latency and the sensitivity-vs-business-relevance tradeoff. - **Diagnostic metrics** to localize where any change comes from (e.g., funnel steps, latency percentiles). - **Guardrail metrics** (quality / reliability / revenue) to avoid shipping a regression — in particular playback error rate and rebuffering, since the new stack may be unstable. 2. **Unit of randomization & interference.** Choose user vs. device vs. session/request-level randomization and justify it. Explain how you avoid contamination and interference (sticky bucketing, CDN/cache routing, cross-device spillover). 3. **Power / MDE / duration.** Describe how you would estimate sample size and runtime: what inputs you need, how you set the MDE, and how you handle **heavy-tailed watch-time**. Note relevant variance drivers (user heterogeneity, seasonality, outliers). 4. **Variance reduction.** Give at least two techniques to reduce variance / improve sensitivity and explain when each is appropriate (e.g., CUPED, stratification, triggering, winsorization, clustered standard errors). 5. **Analysis plan & conflicting movements.** Specify the estimator, how you handle multiple metrics and heterogeneity (e.g., WiFi vs. cellular), and how you decide when results conflict — e.g., watch time is up but the error guardrail is also up. 6. **Diagnosing a ratio-metric change.** Suppose leadership tracks a **ratio metric** such as **CTR = clicks / impressions** or **conversion rate = purchases / sessions**, and it moved by +0.3%. Outline a structured approach to diagnose *why* it changed, decompose the move into numerator vs. denominator effects, and guard against misleading interpretations such as **Simpson’s paradox**. 7. **When randomization is not possible (propensity score matching).** Now assume you **cannot randomize** the latency change (e.g., it rolled out selectively due to infra constraints) and you only observe that some users experienced lower latency than others. Describe how you would use **propensity score matching (PSM)** to estimate the impact, list the assumptions PSM requires, and explain how you would validate / sensitivity-test them. **Assumptions** - Users are global; traffic varies by time-of-day and day-of-week, and latency effects may be heterogeneous by network type (WiFi vs. cellular). - You can define a pre-period to compute baselines / covariates. - Logging is available for latency, exposure, errors, buffering, and key engagement outcomes. - Use a consistent reporting timezone (e.g., UTC) for daily metrics to avoid boundary artifacts.

A Google Data Scientist onsite case on measuring the causal impact of reduced YouTube video-start latency on engagement. It spans A/B design (primary metric, diagnostics, error/buffering guardrails), randomization unit and interference, power/MDE for heavy-tailed watch time, variance reduction (CUPED, stratification), analysis of conflicting movements, ratio-metric decomposition with Simpson’s paradox, and propensity score matching when randomization isn’t possible.

How do I approach Analytics & Experimentation interview questions?

Analytics & Experimentation questions require understanding of core concepts and practice. PracHub provides solutions with explanations to help you master analytics & experimentation interviews.

What difficulty level is this interview question?

This is a easy difficulty Analytics & Experimentation question, commonly asked during Onsite rounds at Google.

What role is this question designed for?

This question is commonly asked for Data Scientist candidates at Google during technical interviews.

Design tests to measure latency impact | Google Interview Question

Question

You are a Data Scientist supporting a large consumer product (e.g., YouTube). Engineering ships a change intended to reduce client-side / video-start latency by roughly 100 ms for some users, but the new code path might increase the error rate and change buffering behavior. Design and analyze the measurement program for this change.

Experiment design (latency → business impact). Propose an A/B test to estimate the causal impact of reduced latency on user engagement. Define:
- A hypothesis and a primary metric (one decision metric), explaining why it is sensitive to latency and the sensitivity-vs-business-relevance tradeoff.
- Diagnostic metrics to localize where any change comes from (e.g., funnel steps, latency percentiles).
- Guardrail metrics (quality / reliability / revenue) to avoid shipping a regression — in particular playback error rate and rebuffering, since the new stack may be unstable.
Unit of randomization & interference. Choose user vs. device vs. session/request-level randomization and justify it. Explain how you avoid contamination and interference (sticky bucketing, CDN/cache routing, cross-device spillover).
Power / MDE / duration. Describe how you would estimate sample size and runtime: what inputs you need, how you set the MDE, and how you handle heavy-tailed watch-time . Note relevant variance drivers (user heterogeneity, seasonality, outliers).
Variance reduction. Give at least two techniques to reduce variance / improve sensitivity and explain when each is appropriate (e.g., CUPED, stratification, triggering, winsorization, clustered standard errors).
Analysis plan & conflicting movements. Specify the estimator, how you handle multiple metrics and heterogeneity (e.g., WiFi vs. cellular), and how you decide when results conflict — e.g., watch time is up but the error guardrail is also up.
Diagnosing a ratio-metric change. Suppose leadership tracks a ratio metric such as CTR = clicks / impressions or conversion rate = purchases / sessions , and it moved by +0.3%. Outline a structured approach to diagnose why it changed, decompose the move into numerator vs. denominator effects, and guard against misleading interpretations such as Simpson’s paradox .
When randomization is not possible (propensity score matching). Now assume you cannot randomize the latency change (e.g., it rolled out selectively due to infra constraints) and you only observe that some users experienced lower latency than others. Describe how you would use propensity score matching (PSM) to estimate the impact, list the assumptions PSM requires, and explain how you would validate / sensitivity-test them.

Assumptions

Users are global; traffic varies by time-of-day and day-of-week, and latency effects may be heterogeneous by network type (WiFi vs. cellular).
You can define a pre-period to compute baselines / covariates.
Logging is available for latency, exposure, errors, buffering, and key engagement outcomes.
Use a consistent reporting timezone (e.g., UTC) for daily metrics to avoid boundary artifacts.

Question

Experiment design (latency → business impact). Propose an A/B test to estimate the causal impact of reduced latency on user engagement. Define:
- A hypothesis and a primary metric (one decision metric), explaining why it is sensitive to latency and the sensitivity-vs-business-relevance tradeoff.
- Diagnostic metrics to localize where any change comes from (e.g., funnel steps, latency percentiles).
- Guardrail metrics (quality / reliability / revenue) to avoid shipping a regression — in particular playback error rate and rebuffering, since the new stack may be unstable.
Unit of randomization & interference. Choose user vs. device vs. session/request-level randomization and justify it. Explain how you avoid contamination and interference (sticky bucketing, CDN/cache routing, cross-device spillover).
Power / MDE / duration. Describe how you would estimate sample size and runtime: what inputs you need, how you set the MDE, and how you handle heavy-tailed watch-time . Note relevant variance drivers (user heterogeneity, seasonality, outliers).
Variance reduction. Give at least two techniques to reduce variance / improve sensitivity and explain when each is appropriate (e.g., CUPED, stratification, triggering, winsorization, clustered standard errors).
Analysis plan & conflicting movements. Specify the estimator, how you handle multiple metrics and heterogeneity (e.g., WiFi vs. cellular), and how you decide when results conflict — e.g., watch time is up but the error guardrail is also up.
Diagnosing a ratio-metric change. Suppose leadership tracks a ratio metric such as CTR = clicks / impressions or conversion rate = purchases / sessions , and it moved by +0.3%. Outline a structured approach to diagnose why it changed, decompose the move into numerator vs. denominator effects, and guard against misleading interpretations such as Simpson’s paradox .
When randomization is not possible (propensity score matching). Now assume you cannot randomize the latency change (e.g., it rolled out selectively due to infra constraints) and you only observe that some users experienced lower latency than others. Describe how you would use propensity score matching (PSM) to estimate the impact, list the assumptions PSM requires, and explain how you would validate / sensitivity-test them.

Assumptions

Users are global; traffic varies by time-of-day and day-of-week, and latency effects may be heterogeneous by network type (WiFi vs. cellular).
You can define a pre-period to compute baselines / covariates.
Logging is available for latency, exposure, errors, buffering, and key engagement outcomes.
Use a consistent reporting timezone (e.g., UTC) for daily metrics to avoid boundary artifacts.

Design tests to measure latency impact

Quick Overview

Design tests to measure latency impact

Question

Write your answer

Design tests to measure latency impact

Quick Overview

Design tests to measure latency impact

Question

Write your answer