Design an email flu-shot experiment
Overview
This question evaluates end-to-end experiment design and causal inference skills for a Data Scientist, including A/B testing and randomization choices, interference control, metric definition and power/sample-size calculation, outcome sourcing from claims/EHR/pharmacy feeds, cadence/MVT design, seasonality controls, and compliance/privacy considerations. It is commonly asked in analytics and experimentation interviews because it tests both practical application and conceptual understanding of field experiment planning, real-world data challenges (missing outcomes, claim lag), and analysis decisions (ITT vs per-protocol, multiple testing) within the Analytics & Experimentation domain.
End-to-End Experiment Design: Email Campaign to Increase Verified Flu Vaccinations
Context: You are designing a 30-day email campaign to increase verified flu vaccinations among eligible health-plan/pharmacy members. Assume eligibility is: age ≥18, no verified flu shot this season, valid email and marketing opt-in, and reachable within the service area. The goal is to measure verified vaccinations (claims/EHR/pharmacy) within 30 days of first exposure.
Specify the following:
-
Targeting and Exclusions
- Define who is in-scope and who must be excluded (e.g., recent vaccination, bounced email, legal/marketing opt-outs, high-contact risk).
-
Randomization Unit and Interference
- Choose the randomization unit (individual vs household vs clinic). Explain why, and how you will limit spillover/interference across units.
-
Holdout Structure and Seeding
- Choose A/A, A/B, or multi-arm. Describe any pre-launch A/A checks, roll-out seeding/ramping, and how you will keep assignments stable.
-
Metrics and Outcome Sourcing
- Define the primary success metric and guardrail metrics. Explain how you will obtain ground truth (claims/EHR, pharmacy feeds) and handle missing outcomes for shots outside the network and claim lag.
-
Power and Sample Size
- Baseline 30-day vaccination rate: 2.0%; expected relative uplift: 20% (i.e., 2.4% in treatment). Use two-sided alpha=0.05 and power=0.80. Provide required sample size (equal allocation) and discuss an option for unequal allocation.
-
Cadence, Frequency Capping, and MVT
- Propose send cadence and frequency caps. Describe how to run multivariate subject-line/creative tests without contaminating the main treatment-control comparison.
-
Seasonality and Overlapping Campaigns
- Describe controls for holidays/seasonality and how you will handle other overlapping member campaigns.
-
Compliance/Privacy
- Address compliance constraints (opt-outs, HIPAA/PHI, content restrictions) and analytical implications.
-
Analysis Plan
- Specify intention-to-treat (ITT) vs per-protocol, heterogeneity reads (pre-specified subgroups), multiple testing controls, and a backstop causal method if randomization is partially broken.
Solution
Login required