Experiment Analysis Plan: User-Level ITT with Robust Inference, Variance Reduction, Ratios, Skew, Non-Compliance, and Decision Framework

Context

You ran a randomized experiment with randomization at the user level. Post-period outcomes are recorded at both session and user levels. Some users never opened/engaged (non-compliance/partial exposure). There are ratio metrics (e.g., revenue per session, conversion rate) and some outcomes are skewed. You have reliable pre-period data for variance reduction.

Task

Show exactly how you will analyze the experiment:

1. Primary estimand and inference
   • Compute the intent-to-treat (ITT) lift at the user assignment level.
   • Use cluster-robust standard errors at the user level.
   • Use CUPED or pre-period covariates to reduce variance.
2. Ratio metrics and skewed outcomes
   • Correctly handle ratio metrics via the delta method or Fieller's theorem.
   • Address skewed outcomes with appropriate techniques.
3. Level-of-analysis rationale
   • Explain why session-level analysis is problematic here.
   • Provide fixes: aggregate to user, mixed models/GEE, cluster-robust SE.
4. Non-compliance / partial exposure
   • Handle users who never opened.
   • Estimate treatment-on-the-treated (TOT) via 2SLS using assignment as the instrument.
5. Decision framework
   • If the p-value is large, decide whether to fail to reject vs claim no effect.
   • Run a post-hoc power/MDE check.
   • Optionally run equivalence/non-inferiority tests (TOST) and/or compare to a Bayesian posterior with a ROPE.
6. Explore heterogeneity and control multiplicity
   • Outline heterogeneity analysis and multiple-testing control.