Plan and analyze an A/B test

A new dashboard change is expected to increase order conversion. Baseline daily unique-visitor conversion is 6.0%. You need 80% power at α=0.05 (two-sided) to detect a 5% relative lift (to 6.3%). 1) Compute the minimum per-variant sample size assuming independent Bernoulli trials; state the formula and plug in numbers. 2) Now assume users are clustered by traffic-source with an intra-class correlation of 0.02 and an average of 50 users per source; recompute the effective sample size using design effect. 3) You must monitor sequentially every day for 14 days; outline a valid sequential testing plan (e.g., alpha spending or group-sequential boundaries) and how confidence intervals are adjusted. 4) If you also track 8 secondary metrics, specify a multiple-comparisons control strategy and why it’s appropriate. 5) After launch you observe an overall +0.4 pp lift but a −0.6 pp lift for mobile Safari; explain three diagnostic checks to reconcile metric vs. anecdote discrepancies (e.g., bot traffic, implementation bugs, covariate imbalance) and how you would quantify each.