Design causal measurement without randomization

Q: Design causal measurement without randomization

This question evaluates a data scientist's competency in causal inference and observational study design, focusing on propensity score matching and difference-in-differences to estimate a notification feature's impact on seven-day retention.

Q: How do I approach Analytics & Experimentation interview questions?

Analytics & Experimentation questions require understanding of core concepts and practice. PracHub provides solutions with explanations to help you master analytics & experimentation interviews.

Question

Causal Study Design: Notification Feature Impact on 7-Day Retention

Context: A new notification feature shipped on 2025-06-01. Randomized rollout was infeasible due to marketing constraints. You need to estimate the causal impact of this feature on 7-day retention and compare two observational methods: propensity score matching (PSM) and difference-in-differences (DiD).

State your assumptions explicitly if needed and design the study. Address each item for both PSM and DiD:

(a) Treatment/control definitions and unit of analysis.

(b) Covariates required for conditional ignorability (or to support DiD’s identifying assumptions) and why they matter.

(c) How you would test and enforce common support/overlap.

(d) Pre-trend and placebo checks.

(e) Handling staggered adoption and seasonality.

(f) Standard error computation and clustering strategy.

(g) Sensitivity analysis for unmeasured confounding.

(h) Decision criteria for preferring PSM vs DiD in this setting.

(i) How you would communicate the causal estimate and its uncertainty to leadership.

Design causal measurement without randomization

Causal Study Design: Notification Feature Impact on 7-Day Retention

Solution

Comments (0)

Design causal measurement without randomization

Overview

Causal Study Design: Notification Feature Impact on 7-Day Retention

Solution

Comments (0)