Fall 2025 Flu Vaccination Uplift Experiment — Design and Evaluation

Context (assume a large US pharmacy with loyalty IDs)

• Audience: Adults with loyalty IDs in CA, NY, TX.
• Outcome window: Vaccinated in-store Sept–Nov 2025.
• Channels: SMS and Email. Per-send costs: SMS = $0.02, Email =$ 0.001.
• Deliverability: SMS 92%, Email 98%.

Tasks

1. Experimental design
   • Choose between A/B (one channel vs. control) or a 2×2 factorial (SMS on/off × Email on/off). Justify considering potential channel interaction, household interference, and send-cost constraints.
   • Define eligibility and exclusions (e.g., opt-outs, no valid contact, prior 2025 vaccination), the randomization unit (person vs. household), and stratification variables (e.g., age band, state, past pharmacy visits, contact availability).
   • Specify the primary metric (absolute lift in vaccination rate) and guardrail metrics (opt-outs/unsubscribes, spam complaints, no-show rate for appointments, store capacity).
2. Sample size and power (show formulas and numeric results)
   • Baseline vaccination rate p0 = 8.0%.
   • Minimum detectable effect (MDE) = 1.5 percentage points (pp).
   • Two-sided α = 0.05, power = 0.80.
   • Compute per-arm sample size ignoring clustering. Then discuss inflation for household clustering with ICC = 0.01 and average household size = 1.3.
3. Compliance and analysis
   • Only a fraction of treated customers are actually exposed (deliverability + opens). Define the Intention-To-Treat (ITT) estimator and compute the Treatment-On-The-Treated (TOT) using deliverability as an instrument. Show that TOT ≈ ITT / compliance and state required assumptions.
4. Attribution and measurement
   • Customers can receive both channels in a factorial design. Propose instrumentation (unique links/codes, timestamps) and an analysis plan to attribute incremental impact to each channel (e.g., factorial contrasts, regression with interactions, hierarchical models). Explain why self-reported "came because of SMS/Email" is biased and how, if at all, it should be used.
5. Edge case
   • In one market, 100 vaccinated customers from the SMS arm, but only 50 say they came because of a message. With control vaccination rate = 7.5% and SMS-arm rate = 9.0%, compute the ITT lift and discuss why self-reports do not change the causal estimate. If the lift is below the 1.5 pp target, what operational changes would you test next?
6. Reporting
   • Describe how you would monitor during rollout (sequential testing controls), finalize results (confidence intervals, CUPED if pre-period data exist), and recommend a scaled policy under a fixed budget.