Evaluate concession gift-card policy with DID
Overview
This question evaluates a data scientist's competency in causal inference and experimentation design, encompassing difference-in-differences with staggered adoption and event-study diagnostics, data inventory and unit-of-analysis specification, clustering choices, heterogeneity and guardrail analyses, and cost–benefit and optional randomized experiment design. Commonly asked in Analytics & Experimentation interviews because it gauges the ability to justify identification strategies, diagnose violations of parallel trends and propose alternatives, and translate empirical results into business-impact metrics, it sits at the intersection of conceptual understanding and practical application within the Data Science domain.
Evaluate a Gift-Card Concession Pilot (Causal Impact with Staggered Adoption)
Context
Several regions piloted a policy: when a shipment is lost or damaged, affected customers receive a gift card instead of only a refund or replacement. You are asked to evaluate the program’s impact on customer experience and business outcomes.
Tasks
-
Clarify Objectives and Outcomes
- State the primary decision objective and identify primary/secondary outcomes (e.g., star ratings, complaint rates, repeat purchase/churn, CSAT/NPS, time-to-resolution, cost per incident).
-
Data Inventory and Unit of Analysis
- List required datasets and variables; specify unit(s) of analysis (incident-level, customer-level, city/region-week), inclusion/exclusion criteria, and whether pre-pilot data exist (and for how long).
-
DID with Staggered Adoption
- Write the regression for a difference-in-differences with staggered adoption, including treatment indicator × post, unit and time fixed effects, and covariates.
- State your clustering choice and why.
- Write an event-study specification for pre-trend checks and dynamic effects.
-
If Parallel Trends Fails
- Propose remedies (e.g., cohort-specific trends), and selection-on-observables alternatives such as PSM–DID with common support and balance checks, or synthetic control at the city level. Justify when each is appropriate.
-
Guardrails and Heterogeneity
- Define guardrail metrics (e.g., refund costs, abuse/fraud, customer service load).
- Outline heterogeneity analyses (e.g., by incident severity, item price/category, customer tenure).
-
Cost–Benefit
- Describe how you would compute ROI/business impact, including gift-card redemption/breakage, margins, and operational costs.
-
Optional: RCT Design
- If you could run a randomized experiment, propose the design (unit, randomization/stratification, sample size/power, spillover controls) and define stop/go decision rules.
Solution
Login required