Analyze a geo rollout and interpret charts

Quick Overview

This question evaluates causal inference and product analytics competencies — specifically specification of causal estimands and difference-in-differences, segmented (interrupted) time-series regression, power and duration calculations, spillover and donor-pool diagnostics, and selection of guardrail metrics.

Analyze a geo rollout and interpret charts

Company: Pinterest

Role: Data Scientist

Category: Analytics & Experimentation

Difficulty: hard

Interview Round: Onsite

You launch a new onboarding flow on 2025-07-15 only in Texas and Florida. A week later, execs notice a 3% nationwide DAU dip on a line chart. Design an analysis and answer precisely: 1) Causal question: Did the feature cause a DAU change in treated states? Specify estimand and write the difference-in-differences (DiD) formula explicitly. 2) Using the following summary (daily averages, pre = 2025-06-15..2025-07-14, post = 2025-07-15..2025-08-14): Texas pre=500k, post=515k; Florida pre=300k, post=303k; Control pool (other states) pre=2,000k, post=2,060k. Compute DiD for each treated state and combined (population-weighted). Interpret sign and magnitude. 3) The chart also shows a weekend trough pattern and a visible break on 2025-07-20. Outline a segmented regression on the time series to quantify immediate level change and slope change for treated vs control. Include the regression equation and how you’d cluster standard errors. 4) Guardrail metrics: propose at least three (e.g., crash rate, latency p95, payment decline rate). Define decision thresholds and which are one-sided vs two-sided. 5) Power and duration: with average daily DAU in treated = 815k combined, MDE = 0.5% relative on DAU, alpha=0.05, power=0.8, estimate required days under a parallel-trends DiD. State assumptions and whether CUPED or synthetic controls would reduce required duration. 6) The chart is noisy around 2025-07-27 after a marketing campaign in California. Explain how you’d validate the parallel trends assumption and choose a donor pool or weights to mitigate spillovers. 7) Provide a brief go/no-go recommendation and the exact additional data you’d request to de-risk the decision.

Quick Answer: This question evaluates causal inference and product analytics competencies — specifically specification of causal estimands and difference-in-differences, segmented (interrupted) time-series regression, power and duration calculations, spillover and donor-pool diagnostics, and selection of guardrail metrics.

|Home/Analytics & Experimentation/Pinterest
Pinterest logo
Pinterest
Oct 13, 2025, 9:49 PM
hardData ScientistOnsiteAnalytics & Experimentation
19
0

Causal Impact of a New Onboarding Flow Launched in Texas and Florida

Context: A new onboarding flow was launched on 2025-07-15 only in Texas (TX) and Florida (FL). A line chart shows a 3% nationwide DAU dip about a week later. You are asked to design a defensible causal analysis and provide precise computations and recommendations.

Assumptions for clarity:

  • Outcome is daily active users (DAU) at the state level.
  • Pre period: 2025-06-15 to 2025-07-14. Post period: 2025-07-15 to 2025-08-14.
  • Control pool is all other US states (excluding TX and FL).

Tasks

  1. Causal question and estimand
    • State the causal question: Did the onboarding feature cause a change in DAU in the treated states (TX, FL)?
    • Specify the estimand clearly and write the difference-in-differences (DiD) formula explicitly (in both level and relative/log terms).
  2. Compute DiD using provided daily averages
    • Summary (daily averages):
      • Texas: pre = 500k, post = 515k
      • Florida: pre = 300k, post = 303k
      • Control pool (other states): pre = 2,000k, post = 2,060k
    • Compute DiD for each treated state and for the combined treated group (population-weighted by pre-period DAU). Interpret the sign and magnitude.
  3. Segmented (interrupted) time-series regression
    • The chart shows weekend troughs and a visible break on 2025-07-20.
    • Outline a segmented regression design to estimate immediate level change and slope change for treated vs. control.
    • Provide the regression equation and describe how you would cluster standard errors.
  4. Guardrail metrics
    • Propose at least three guardrails (e.g., crash rate, p95 latency, payment decline rate), define decision thresholds, and specify which are one-sided vs. two-sided.
  5. Power and duration for DiD
    • Given: average daily DAU in treated combined ≈ 815k, minimum detectable effect (MDE) = 0.5% relative on DAU, alpha = 0.05, power = 0.8.
    • Estimate required days under a parallel-trends DiD. State assumptions and whether CUPED or synthetic controls would reduce duration.
  6. Parallel trends and spillovers
    • Around 2025-07-27, there was a marketing campaign in California causing noise. Explain how to validate the parallel trends assumption and how to choose a donor pool or weights to mitigate spillovers.
  7. Recommendation and additional data
    • Provide a brief go/no-go recommendation and list the exact additional data you would request to de-risk the decision.
Loading comments...