Multiple Testing and Sequential Monitoring with 1 Primary and 12 Secondary Metrics
Context
You are monitoring an A/B experiment over 4 weekly looks. There is 1 primary metric and 12 secondary metrics, and many are correlated. You need strong familywise error rate (FWER) control for confirmatory claims and false discovery rate (FDR) control for exploratory findings.
Assume 5 metrics are pre-specified as confirmatory (including the primary), and the remaining 8 are exploratory.
Tasks
(a) Propose a hierarchy that preserves strong FWER control for confirmatory endpoints and FDR control for exploratory endpoints across 4 looks.
(b) If there are k = 5 confirmatory metrics at α = 0.05, compute the Holm–Bonferroni adjusted threshold for the smallest p-value.
(c) For the remaining k = 8 exploratory metrics, show how to apply the Benjamini–Hochberg (BH) procedure at q = 0.10, and discuss how to combine this with group-sequential alpha spending or always-valid p-values so repeated looks do not inflate error rates.