Your team shipped a notification feature on 2025-06-01 but randomization was infeasible due to marketing constraints. Design a causal study to estimate its impact on 7-day retention. Compare and justify two approaches—propensity score matching (PSM) and difference-in-differences (DiD). Specify: (a) treatment/control definitions and unit of analysis; (b) covariates needed for conditional ignorability and why; (c) how you would test/enforce common support; (d) pre-trend/placebo checks; (e) handling staggered adoption and seasonality; (f) standard error computation and clustering strategy; (g) sensitivity analysis for unmeasured confounding; (h) decision criteria for preferring PSM vs DiD; (i) how you would communicate the causal estimate and uncertainty to leadership.

This question evaluates a data scientist's competency in causal inference and observational study design, focusing on propensity score matching and difference-in-differences to estimate a notification feature's impact on seven-day retention.

How do I approach Analytics & Experimentation interview questions?

Analytics & Experimentation questions require understanding of core concepts and practice. PracHub provides solutions with explanations to help you master analytics & experimentation interviews.

What difficulty level is this interview question?

This is a hard difficulty Analytics & Experimentation question, commonly asked during Technical Screen rounds at TikTok.

What role is this question designed for?

This question is commonly asked for Data Scientist candidates at TikTok during technical interviews.

Design causal measurement without randomization

Causal Study Design: Notification Feature Impact on 7-Day Retention

Context: A new notification feature shipped on 2025-06-01. Randomized rollout was infeasible due to marketing constraints. You need to estimate the causal impact of this feature on 7-day retention and compare two observational methods: propensity score matching (PSM) and difference-in-differences (DiD).

State your assumptions explicitly if needed and design the study. Address each item for both PSM and DiD:

(a) Treatment/control definitions and unit of analysis.

(b) Covariates required for conditional ignorability (or to support DiD’s identifying assumptions) and why they matter.

(d) Pre-trend and placebo checks.

(e) Handling staggered adoption and seasonality.

(f) Standard error computation and clustering strategy.

(g) Sensitivity analysis for unmeasured confounding.

(h) Decision criteria for preferring PSM vs DiD in this setting.

(i) How you would communicate the causal estimate and its uncertainty to leadership.

Causal Study Design: Notification Feature Impact on 7-Day Retention

State your assumptions explicitly if needed and design the study. Address each item for both PSM and DiD:

(a) Treatment/control definitions and unit of analysis.

(b) Covariates required for conditional ignorability (or to support DiD’s identifying assumptions) and why they matter.

(d) Pre-trend and placebo checks.

(e) Handling staggered adoption and seasonality.

(f) Standard error computation and clustering strategy.

(g) Sensitivity analysis for unmeasured confounding.

(h) Decision criteria for preferring PSM vs DiD in this setting.

(i) How you would communicate the causal estimate and its uncertainty to leadership.

Design causal measurement without randomization

Quick Overview

Causal Study Design: Notification Feature Impact on 7-Day Retention

Solution

Comments (0)

Design causal measurement without randomization

Quick Overview

Causal Study Design: Notification Feature Impact on 7-Day Retention

Solution

Comments (0)