Diagnose a failing campaign

Quick Overview

This question evaluates competency in experimental diagnostics, causal inference, data-quality auditing, and analytics-driven troubleshooting for randomized CRM campaigns, emphasizing exposure verification, contamination detection, logging/ETL integrity, outcome validity, heterogeneity analysis, and re-test design.

Diagnose a failing campaign

Company: CVS Health

Role: Data Scientist

Category: Analytics & Experimentation

Difficulty: hard

Interview Round: Technical Screen

Your vaccination-lift experiment shows no overall effect and a negative effect in seniors. Lay out a triage plan: (a) exposure verification (deliverability, inbox placement, opens/clicks under mail privacy protections), (b) targeting leakage/crossover and contamination checks, (c) logging/ETL audits for timestamp/timezone/dedup errors, (d) outcome validity (claims/EHR coverage gaps and lag from shot to claim), (e) lift suppressors (fatigue, frequency-capping bugs, creative rendering issues), (f) heterogeneity and calendar interactions, (g) re-test design (A/A, split-by-seed, staggered rollouts) and quick corrective experiments, and (h) decision criteria to pause, pivot, or proceed. Specify example queries/plots you would run and the minimal additional data you would request.

Quick Answer: This question evaluates competency in experimental diagnostics, causal inference, data-quality auditing, and analytics-driven troubleshooting for randomized CRM campaigns, emphasizing exposure verification, contamination detection, logging/ETL integrity, outcome validity, heterogeneity analysis, and re-test design.

|Home/Analytics & Experimentation/CVS Health
CVS Health logo
CVS Health
Oct 13, 2025
hardData ScientistTechnical ScreenAnalytics & Experimentation
5
0

Vaccination-Lift Experiment Triage Plan

Context: You ran a randomized CRM experiment (e.g., email/SMS/push) to increase vaccination uptake. The overall intention-to-treat (ITT) lift is near zero, with an unexpected negative effect among seniors (65+). Create a structured triage plan to diagnose, validate, and act. For each area below, specify concrete checks, example queries/plots, and the minimal additional data you would request.

Cover the following:

(a) Exposure verification: deliverability, inbox placement, opens/clicks under mail privacy protections.

(b) Targeting leakage/crossover and contamination checks.

(c) Logging/ETL audits for timestamp/timezone/dedup errors.

(d) Outcome validity: claims/EHR coverage gaps and shot-to-claim lag.

(e) Lift suppressors: fatigue, frequency-capping bugs, creative rendering issues.

(f) Heterogeneity and calendar interactions.

(g) Re-test design: A/A, split-by-seed, staggered rollouts; plus quick corrective experiments.

(h) Decision criteria to pause, pivot, or proceed.

Include:

  • Example SQL-like queries or calculations.
  • Diagnostic plots you would produce.
  • The minimal additional data you would request to complete the triage.
Loading comments...