Analyze results and large p-values correctly
Overview
This question evaluates a candidate's mastery of experimental analysis and applied causal inference, touching on intent-to-treat estimation, cluster-robust inference, variance reduction, handling of ratio metrics and skewed outcomes, non-compliance and instrumental approaches, decision frameworks for large p-values, and heterogeneity with multiple-testing control. Commonly asked in the Statistics & Math domain to assess practical application with conceptual understanding, it measures the ability to reason about appropriate analysis level, validity of inference, and interpretation of ambiguous results rather than just computational skill.
Experiment Analysis Plan: User-Level ITT with Robust Inference, Variance Reduction, Ratios, Skew, Non-Compliance, and Decision Framework
Context
You ran a randomized experiment with randomization at the user level. Post-period outcomes are recorded at both session and user levels. Some users never opened/engaged (non-compliance/partial exposure). There are ratio metrics (e.g., revenue per session, conversion rate) and some outcomes are skewed. You have reliable pre-period data for variance reduction.
Task
Show exactly how you will analyze the experiment:
-
Primary estimand and inference
- Compute the intent-to-treat (ITT) lift at the user assignment level.
- Use cluster-robust standard errors at the user level.
- Use CUPED or pre-period covariates to reduce variance.
-
Ratio metrics and skewed outcomes
- Correctly handle ratio metrics via the delta method or Fieller's theorem.
- Address skewed outcomes with appropriate techniques.
-
Level-of-analysis rationale
- Explain why session-level analysis is problematic here.
- Provide fixes: aggregate to user, mixed models/GEE, cluster-robust SE.
-
Non-compliance / partial exposure
- Handle users who never opened.
- Estimate treatment-on-the-treated (TOT) via 2SLS using assignment as the instrument.
-
Decision framework
- If the p-value is large, decide whether to fail to reject vs claim no effect.
- Run a post-hoc power/MDE check.
- Optionally run equivalence/non-inferiority tests (TOST) and/or compare to a Bayesian posterior with a ROPE.
-
Explore heterogeneity and control multiplicity
- Outline heterogeneity analysis and multiple-testing control.
Solution
Login required