Compute cluster-aware significance and sequential corrections
Overview
This question evaluates competency in clustered randomized experiment analysis, including calculation of design effect and effective sample size, cluster-robust inference for differences in proportions, sequential alpha spending (O’Brien–Fleming-style) and comparisons with Bonferroni, Holm–Bonferroni adjustments for multiple guardrail metrics, and Bayesian ROPE interpretation. It is in the Statistics & Math domain and is commonly asked to probe how candidates handle intra-cluster correlation, control Type I error across interim looks and multiple metrics, and demonstrate both conceptual understanding and practical application of power, duration, and multiplicity trade-offs.
Cluster-Randomized Tipping UI Experiment: Power, Sequential Testing, and Multiplicity
Context: A creator-level (cluster) randomized experiment evaluates a tipping UI. Creators are clusters; viewers are units within clusters. The outcome is a binary purchase at the viewer-session level.
Given:
- Per arm: 10,000 creators (clusters)
- Average viewer sessions per creator: m = 100
- Viewer-level purchase rate: control p_c = 5.00%, treatment p_t = 5.20%
- Intra-cluster (creator) correlation of purchase: ρ = 0.02
Tasks:
- Compute the design effect DE = 1 + (m − 1)ρ and the effective viewer-level sample size per arm. Then compute the z-statistic and two-sided p-value for the difference in proportions using cluster-robust standard errors implied by DE.
- Suppose you plan 4 interim looks plus a final analysis (5 looks total). Provide an approximate O’Brien–Fleming-style spending schedule for overall α = 0.05 by giving conservative per-look two-sided α thresholds (assume equally spaced looks). Contrast this with a naive Bonferroni correction. Explain how these choices affect power and required duration.
- With four guardrail metrics, outline a Holm–Bonferroni adjustment procedure. Discuss when you might instead report Bayesian posterior intervals with a ROPE (Region of Practical Equivalence) to focus on practical significance.
Solution
Login required