Interview concept

Hypothesis Testing And Confidence Intervals

Asked of: Data Scientist

Last updated

Clean flowchart showing the hypothesis-testing workflow: define metric, set H0/HA, design checks, choose test, assumptions, compute p-value & 95% CI, interpret, report, and A/B example with pitfalls.

  1. What's being tested
    Your ability to translate a business question into formal hypotheses, pick the correct test and assumptions, compute/interpret p-values and confidence intervals, and judge practical versus statistical significance.

  2. Core knowledge

  • Null and alternative hypotheses: point vs composite, one-sided vs two-sided.
  • Type I/II errors; alpha (significance) and power (1−beta) tradeoffs.
  • t-test vs z-test vs two-proportion test vs chi-square; when CLT applies.
  • Paired versus independent designs and implications for variance estimation.
  • Confidence interval interpretation: estimator variability, not "probability true parameter".
  • Multiple comparisons corrections: Bonferroni, Holm, and Benjamini–Hochberg FDR.
  • Sample-size/power formulas: detectible effect, baseline variance/proportion, alpha, power.

  1. Worked example — "Comparing conversion rates between two groups (A/B test)"
    First define the metric (conversion probability) and estimand (p_A − p_B). State H0: p_A = p_B and HA appropriately (two-sided or directional). Check design: independent random assignment and sufficient counts for normal approximation (np, n(1−p) ≥ ~5). Choose two-proportion z-test (or exact test if small counts), decide pooled versus unpooled variance based on H0, and plan to report both p-value and a 95% CI for the difference. Finally, discuss practical significance (minimum detectable uplift) and any multiple-testing or sequential monitoring corrections.

  2. A common pitfall
    Interpreting a non-significant result as evidence of no effect—without considering power—is common. Similarly, treating p-values as the probability the null is true leads to incorrect conclusions. Another frequent error is ignoring dependence (e.g., repeated users) and using independent-sample formulas, which underestimates variance and inflates false positives.

  3. Further reading

  • Casella, G., & Berger, R. L., "Statistical Inference" — rigorous treatment of tests and intervals.
  • Efron, B., & Tibshirani, R., "An Introduction to the Bootstrap" — practical resampling for CIs and small-sample inference.

Related concepts