Virtual Queue Pilot: Experiment Design and Analysis

Context

A theme park is piloting a virtual queue system designed to reduce average wait time by 20% while maintaining per-capita revenue within −2% of baseline. You are asked to design the experiment and analysis plan that is operationally feasible, statistically valid in the presence of network effects, and decision-oriented.

Tasks

1. Define metrics
   • Primary: average wait time reduction.
   • Guardrails: per-capita revenue, throughput, guest complaint rate; add any necessary operational guardrails (e.g., ride utilization, no-show rate).
2. Randomization unit and interference risks
   • Identify plausible units (guest/family, ride, time block, park-day).
   • Discuss interference/spillovers (families traveling together, cross-ride spillovers, capacity sharing) and implications for SUTVA.
3. Experiment design to mitigate network effects
   • Propose a cluster/switchback design (e.g., by ride-time blocks or park-level time blocks) with blocking/stratification for seasonality (weather, weekends) and arrival-rate variation by hour.
4. Power and duration
   • Estimate sample size and test duration with explicit assumptions (e.g., baseline means/SDs, ICC or cluster-period variance, adoption rate).
   • Show formulas and a worked numeric example.
5. In-flight monitoring and decision rules
   • Specify sequential testing or pre-registered peeks with alpha spending.
   • Define success/non-inferiority criteria and operational guardrails for stopping/continuation.
6. Analysis plan
   • Estimands (intent-to-treat vs. treatment-on-treated), model choices (difference-in-differences, CUPED), variance reduction, and handling heteroskedasticity/heavy tails.
7. Post-test segmentation and budget impact
   • Segment effects by ride popularity and guest demographics; describe method and multiple-testing controls.
   • Compute budget impact with formulas and a small numeric example.
8. Mixed results scenario
   • If waits drop 25% but per-capita revenue declines 3.5%, recommend a follow-up test (e.g., dynamic return windows, targeted upsell) and list the minimal data needed to inform rollout.