How do I approach Analytics & Experimentation interview questions?

Analytics & Experimentation questions require understanding of core concepts and practice. PracHub provides solutions with explanations to help you master analytics & experimentation interviews.

What difficulty level is this interview question?

This is a hard difficulty Analytics & Experimentation question, commonly asked during Technical Screen rounds at CVS Health.

What role is this question designed for?

This question is commonly asked for Data Scientist candidates at CVS Health during technical interviews.

Design an email flu-shot experiment | CVS Health Interview Question

Quick Overview

This question evaluates end-to-end experiment design and causal inference skills for a Data Scientist, including A/B testing and randomization choices, interference control, metric definition and power/sample-size calculation, outcome sourcing from claims/EHR/pharmacy feeds, cadence/MVT design, seasonality controls, and compliance/privacy considerations. It is commonly asked in analytics and experimentation interviews because it tests both practical application and conceptual understanding of field experiment planning, real-world data challenges (missing outcomes, claim lag), and analysis decisions (ITT vs per-protocol, multiple testing) within the Analytics & Experimentation domain.

End-to-End Experiment Design: Email Campaign to Increase Verified Flu Vaccinations

Context: You are designing a 30-day email campaign to increase verified flu vaccinations among eligible health-plan/pharmacy members. Assume eligibility is: age ≥18, no verified flu shot this season, valid email and marketing opt-in, and reachable within the service area. The goal is to measure verified vaccinations (claims/EHR/pharmacy) within 30 days of first exposure.

Specify the following:

Targeting and Exclusions
- Define who is in-scope and who must be excluded (e.g., recent vaccination, bounced email, legal/marketing opt-outs, high-contact risk).
Randomization Unit and Interference
- Choose the randomization unit (individual vs household vs clinic). Explain why, and how you will limit spillover/interference across units.
Holdout Structure and Seeding
- Choose A/A, A/B, or multi-arm. Describe any pre-launch A/A checks, roll-out seeding/ramping, and how you will keep assignments stable.
Metrics and Outcome Sourcing
- Define the primary success metric and guardrail metrics. Explain how you will obtain ground truth (claims/EHR, pharmacy feeds) and handle missing outcomes for shots outside the network and claim lag.
Power and Sample Size
- Baseline 30-day vaccination rate: 2.0%; expected relative uplift: 20% (i.e., 2.4% in treatment). Use two-sided alpha=0.05 and power=0.80. Provide required sample size (equal allocation) and discuss an option for unequal allocation.
Cadence, Frequency Capping, and MVT
- Propose send cadence and frequency caps. Describe how to run multivariate subject-line/creative tests without contaminating the main treatment-control comparison.
Seasonality and Overlapping Campaigns
- Describe controls for holidays/seasonality and how you will handle other overlapping member campaigns.
Compliance/Privacy
- Address compliance constraints (opt-outs, HIPAA/PHI, content restrictions) and analytical implications.
Analysis Plan
- Specify intention-to-treat (ITT) vs per-protocol, heterogeneity reads (pre-specified subgroups), multiple testing controls, and a backstop causal method if randomization is partially broken.

Quick Overview

End-to-End Experiment Design: Email Campaign to Increase Verified Flu Vaccinations

Specify the following:

Targeting and Exclusions

Define who is in-scope and who must be excluded (e.g., recent vaccination, bounced email, legal/marketing opt-outs, high-contact risk).

Randomization Unit and Interference

Choose the randomization unit (individual vs household vs clinic). Explain why, and how you will limit spillover/interference across units.

Holdout Structure and Seeding

Choose A/A, A/B, or multi-arm. Describe any pre-launch A/A checks, roll-out seeding/ramping, and how you will keep assignments stable.

Metrics and Outcome Sourcing

Define the primary success metric and guardrail metrics. Explain how you will obtain ground truth (claims/EHR, pharmacy feeds) and handle missing outcomes for shots outside the network and claim lag.

Power and Sample Size

Baseline 30-day vaccination rate: 2.0%; expected relative uplift: 20% (i.e., 2.4% in treatment). Use two-sided alpha=0.05 and power=0.80. Provide required sample size (equal allocation) and discuss an option for unequal allocation.

Cadence, Frequency Capping, and MVT

Propose send cadence and frequency caps. Describe how to run multivariate subject-line/creative tests without contaminating the main treatment-control comparison.

Seasonality and Overlapping Campaigns

Describe controls for holidays/seasonality and how you will handle other overlapping member campaigns.

Compliance/Privacy

Address compliance constraints (opt-outs, HIPAA/PHI, content restrictions) and analytical implications.

Analysis Plan

Specify intention-to-treat (ITT) vs per-protocol, heterogeneity reads (pre-specified subgroups), multiple testing controls, and a backstop causal method if randomization is partially broken.

Design an email flu-shot experiment

Quick Overview

End-to-End Experiment Design: Email Campaign to Increase Verified Flu Vaccinations

Solution

Comments (0)

Design an email flu-shot experiment

Quick Overview

End-to-End Experiment Design: Email Campaign to Increase Verified Flu Vaccinations

Solution

Comments (0)