Diagnose a failing campaign
Overview
This question evaluates competency in experimental diagnostics, causal inference, data-quality auditing, and analytics-driven troubleshooting for randomized CRM campaigns, emphasizing exposure verification, contamination detection, logging/ETL integrity, outcome validity, heterogeneity analysis, and re-test design.
Vaccination-Lift Experiment Triage Plan
Context: You ran a randomized CRM experiment (e.g., email/SMS/push) to increase vaccination uptake. The overall intention-to-treat (ITT) lift is near zero, with an unexpected negative effect among seniors (65+). Create a structured triage plan to diagnose, validate, and act. For each area below, specify concrete checks, example queries/plots, and the minimal additional data you would request.
Cover the following:
(a) Exposure verification: deliverability, inbox placement, opens/clicks under mail privacy protections.
(b) Targeting leakage/crossover and contamination checks.
(c) Logging/ETL audits for timestamp/timezone/dedup errors.
(d) Outcome validity: claims/EHR coverage gaps and shot-to-claim lag.
(e) Lift suppressors: fatigue, frequency-capping bugs, creative rendering issues.
(f) Heterogeneity and calendar interactions.
(g) Re-test design: A/A, split-by-seed, staggered rollouts; plus quick corrective experiments.
(h) Decision criteria to pause, pivot, or proceed.
Include:
- Example SQL-like queries or calculations.
- Diagnostic plots you would produce.
- The minimal additional data you would request to complete the triage.
Solution
Login required