Infer causal impact without an A/B test

Quick Overview

This question evaluates causal inference and observational study design skills, including time-series and panel methods, identification strategies (e.g., ITS, DiD, synthetic control, RD), statistical modeling and power/sample-size estimation, confounder adjustment, and communication of uncertainty.

Infer causal impact without an A/B test

Company: Google

Role: Data Scientist

Category: Statistics & Math

Difficulty: hard

Interview Round: Technical Screen

Engineering shipped a new version intended to reduce disconnections, but no A/B holdout exists. Rigorously evaluate effectiveness: choose among interrupted time series with seasonality, difference-in-differences using untreated regions/devices, synthetic control from donor pools, or regression discontinuity if rollout timing is sharp; state identification assumptions, pre-trend checks, placebo tests, and robustness to staggered rollout; specify outcome and model family (e.g., binomial for drop rate, Poisson/negative binomial for drop counts), variance estimation (cluster-robust SEs by account/region), and variance reduction via CUPED; define the primary metric (drops per 1k minutes) and guardrails, then compute confidence intervals and minimal detectable change for 80% power at α=0.05 given a baseline of 3.0 drops/1k minutes over 10M daily minutes and realistic autocorrelation; explain how you’ll adjust for confounders such as network mix shifts, seasonality, and changing user composition, and how you’ll communicate uncertainty to stakeholders.

Quick Answer: This question evaluates causal inference and observational study design skills, including time-series and panel methods, identification strategies (e.g., ITS, DiD, synthetic control, RD), statistical modeling and power/sample-size estimation, confounder adjustment, and communication of uncertainty.

|Home/Statistics & Math/Google
Google logo
Google
Oct 13, 2025, 9:49 PM
hardData ScientistTechnical ScreenStatistics & Math
17
0

Evaluate Impact of a Shipped Version on Disconnections (No A/B Holdout)

Context

A new client version was shipped system-wide with the goal of reducing disconnections. There is no explicit A/B holdout. You must design and execute a rigorous observational evaluation, quantify uncertainty, and communicate results.

Tasks

  1. Design choices
    • Choose a primary identification strategy among:
      • Interrupted time series (ITS) with seasonality
      • Difference-in-differences (DiD) using any untreated regions/devices
      • Synthetic control from donor pools
      • Regression discontinuity (RD) if rollout timing is sharp and exogenous
    • Explain when you would use each method and why.
  2. Identification and validation
    • State identification assumptions for the chosen method.
    • Describe pre-trend/parallel-trend checks, placebo tests, and robustness to staggered rollout.
  3. Metrics and models
    • Define the primary outcome and guardrail metrics.
    • Specify an appropriate model family (e.g., binomial/Poisson/negative binomial) and exposure.
    • Specify variance estimation (e.g., cluster-robust SEs, HAC/Newey–West) and variance reduction (e.g., CUPED).
  4. Power and estimation
    • Primary metric: drops per 1,000 minutes.
    • Baseline: 3.0 drops/1,000 minutes; exposure: 10,000,000 minutes per day.
    • Compute:
      • A 95% confidence interval for the daily drop rate estimate under reasonable dispersion.
      • Minimal detectable change (MDC) for a pre/post comparison with 80% power at α = 0.05, assuming realistic autocorrelation.
      • Show assumptions and formulas. Provide MDC examples for 7-, 14-, and 28-day post windows.
  5. Confounding and communication
    • Explain how to adjust for confounders (network mix shifts, seasonality, user composition), including weighting or covariate adjustment.
    • Explain how you would communicate uncertainty and limitations to stakeholders.
Loading comments...