How do I approach Analytics & Experimentation interview questions?

Analytics & Experimentation questions require understanding of core concepts and practice. PracHub provides solutions with explanations to help you master analytics & experimentation interviews.

What difficulty level is this interview question?

This is a hard difficulty Analytics & Experimentation question, commonly asked during Technical Screen rounds at CVS Health.

What role is this question designed for?

This question is commonly asked for Data Scientist candidates at CVS Health during technical interviews.

Diagnose a failing campaign | CVS Health Interview Question

Quick Overview

This question evaluates competency in experimental diagnostics, causal inference, data-quality auditing, and analytics-driven troubleshooting for randomized CRM campaigns, emphasizing exposure verification, contamination detection, logging/ETL integrity, outcome validity, heterogeneity analysis, and re-test design.

Vaccination-Lift Experiment Triage Plan

Context: You ran a randomized CRM experiment (e.g., email/SMS/push) to increase vaccination uptake. The overall intention-to-treat (ITT) lift is near zero, with an unexpected negative effect among seniors (65+). Create a structured triage plan to diagnose, validate, and act. For each area below, specify concrete checks, example queries/plots, and the minimal additional data you would request.

Cover the following:

(a) Exposure verification: deliverability, inbox placement, opens/clicks under mail privacy protections.

(b) Targeting leakage/crossover and contamination checks.

(d) Outcome validity: claims/EHR coverage gaps and shot-to-claim lag.

(e) Lift suppressors: fatigue, frequency-capping bugs, creative rendering issues.

(f) Heterogeneity and calendar interactions.

(g) Re-test design: A/A, split-by-seed, staggered rollouts; plus quick corrective experiments.

(h) Decision criteria to pause, pivot, or proceed.

Include:

Example SQL-like queries or calculations.
Diagnostic plots you would produce.
The minimal additional data you would request to complete the triage.

Quick Overview

Vaccination-Lift Experiment Triage Plan

Cover the following:

(a) Exposure verification: deliverability, inbox placement, opens/clicks under mail privacy protections.

(b) Targeting leakage/crossover and contamination checks.

(d) Outcome validity: claims/EHR coverage gaps and shot-to-claim lag.

(e) Lift suppressors: fatigue, frequency-capping bugs, creative rendering issues.

(f) Heterogeneity and calendar interactions.

(g) Re-test design: A/A, split-by-seed, staggered rollouts; plus quick corrective experiments.

(h) Decision criteria to pause, pivot, or proceed.

Include:

Example SQL-like queries or calculations.

Diagnostic plots you would produce.

The minimal additional data you would request to complete the triage.

Diagnose a failing campaign

Quick Overview

Vaccination-Lift Experiment Triage Plan

Solution

Comments (0)

Diagnose a failing campaign

Quick Overview

Vaccination-Lift Experiment Triage Plan

Solution

Comments (0)