Stakeholder Influence And Analytical Integrity

What's being tested
Ability to translate stakeholder requests into rigorous, reproducible analyses while resisting pressure to compromise methods or metrics. Judges whether you can protect experiment validity, document trade-offs, and communicate uncertainty clearly.

Core knowledge

• Pre-analysis plan: lock hypotheses, metrics, segments, and analysis code before peeking at results.
• A/B testing assumptions: randomization, SUTVA, stable unit treatment value, and no interference.
• Multiple comparisons: use Bonferroni, Benjamini-Hochberg, or hierarchical testing to control false discovery.
• Causal tools: diff-in-diff, instrumental variables, regression with controls, propensity scores; use DAGs to identify confounders.
• Stopping rules: sequential testing (alpha spending) or pre-registered sample sizes to avoid peeking bias.
• Metrics design: pick primary metric (north star) plus guardrails and power calculations.
• Data integrity: lineage, event-schema, sampling biases, and reproducible notebooks or CI for analyses.

Worked example
Example prompt: "A PM asks you to exclude a segment to make experiment results significant." First, restate the business objective and why the PM believes exclusion helps. Check the pre-analysis plan: was the segment pre-specified? If not, explain the bias risk and propose alternatives: run the original analysis plus a prespecified sensitivity analysis that excludes the segment, report both with adjusted p-values and effect heterogeneity. Recommend a follow-up experiment or segmentation analysis if heterogeneity is real, and document decisions in writing.

A common pitfall
The tempting response is to rapidly comply: modify cohorts or metrics to produce a headline improvement and present the “clean” result. That creates p-hacking, undermines reproducibility, and hides bias. Worse, it leads to product decisions built on fragile effects; instead, insist on transparent re-analyses, write down changes, and quantify robustness (confidence intervals, pre/post comparisons).

Further reading

• Kohavi, Longbotham et al., "Trustworthy Online Controlled Experiments" (Microsoft/IEEE overview of best practices).
• Pearl & Mackenzie, "The Book of Why" — causal reasoning, DAGs, and confounding intuition.