PracHub
QuestionsPremiumLearningGuidesCheatsheetNEWCoaches
|Home/Analytics & Experimentation/Airbnb

Estimate impact of global launch without holdout

Last updated: Mar 29, 2026

Quick Overview

This question evaluates a candidate's competence in causal inference, observational study design, and statistical uncertainty quantification for product analytics, specifically estimating lift after a global launch without a holdout while addressing confounding, interference, and operational data-quality issues.

  • hard
  • Airbnb
  • Analytics & Experimentation
  • Data Scientist

Estimate impact of global launch without holdout

Company: Airbnb

Role: Data Scientist

Category: Analytics & Experimentation

Difficulty: hard

Interview Round: Technical Screen

A product feature was launched globally on 2025-05-10 with no control or holdout. Design a plan to estimate its causal lift on weekly revenue and 7-day retention. Propose at least two independent identification strategies (e.g., interrupted time series with covariates, synthetic control across markets, difference-in-differences using unaffected cohorts, regression discontinuity at the launch timestamp, IV/front-door using eligibility or latency shocks). For each, specify: (a) identification assumptions and threats; (b) unit of analysis, data needs, and pre-period length; (c) diagnostics and falsification tests (placebo dates, negative/positive control outcomes, pre-trend checks); (d) uncertainty quantification and 95% CIs (delta method vs bootstrap, cluster level); (e) how you will bound effects if assumptions partially fail (e.g., amplification/robustness curves, sensitivity to unobserved confounding); (f) how you will handle interference/spillovers, seasonality/holidays, concurrent marketing, logging/schema changes, backfilled events, selection into exposure, and migration; (g) a decision framework to reconcile conflicting estimates and produce a single recommendation (ship, rollback, or iterate), including acceptable risk thresholds.

Quick Answer: This question evaluates a candidate's competence in causal inference, observational study design, and statistical uncertainty quantification for product analytics, specifically estimating lift after a global launch without a holdout while addressing confounding, interference, and operational data-quality issues.

Related Interview Questions

  • Design and Analyze Airbnb Locker Experiment - Airbnb (medium)
  • Design a network-aware Wi‑Fi badge experiment - Airbnb (Medium)
  • Design an A/B test with causal inference - Airbnb (hard)
  • Design robust primary and guardrail metrics - Airbnb (hard)
  • Analyze A/B test with rigorous diagnostics - Airbnb (hard)
Airbnb logo
Airbnb
Oct 13, 2025, 9:49 PM
Data Scientist
Technical Screen
Analytics & Experimentation
14
0

Causal Lift Plan After a Global Launch Without a Holdout

Background

A new product feature was launched globally on 2025-05-10, with no control or holdout. You need to estimate the causal impact (lift) on:

  • Weekly revenue
  • 7-day retention

Assume standard product analytics instrumentation and access to geo-, device-, and cohort-level data.

Task

Design a plan to estimate causal lift using at least two independent identification strategies. For each strategy, specify:

(a) Identification assumptions and key threats

(b) Unit of analysis, data needs, and pre-period length

(c) Diagnostics and falsification tests (e.g., placebo dates, negative/positive control outcomes, pre-trend checks)

(d) Uncertainty quantification and 95% CIs (delta method vs bootstrap; clustering level)

(e) How you will bound effects if assumptions partially fail (e.g., robustness/amplification curves, sensitivity to unobserved confounding)

(f) How you will handle interference/spillovers, seasonality/holidays, concurrent marketing, logging/schema changes, backfilled events, selection into exposure, and migration

Also include:

(g) A decision framework to reconcile conflicting estimates and produce a single recommendation (ship, rollback, or iterate), including acceptable risk thresholds.

Solution

Show

Comments (0)

Sign in to leave a comment

Loading comments...

Browse More Questions

More Analytics & Experimentation•More Airbnb•More Data Scientist•Airbnb Data Scientist•Airbnb Analytics & Experimentation•Data Scientist Analytics & Experimentation
PracHub

Master your tech interviews with 7,500+ real questions from top companies.

Product

  • Questions
  • Learning Tracks
  • Interview Guides
  • Resources
  • Premium
  • For Universities
  • Student Access

Browse

  • By Company
  • By Role
  • By Category
  • Topic Hubs
  • SQL Questions
  • Compare Platforms
  • Discord Community

Support

  • support@prachub.com
  • (916) 541-4762

Legal

  • Privacy Policy
  • Terms of Service
  • About Us

© 2026 PracHub. All rights reserved.