Hypothesis tests and p-values

Sooner or later in a data science screen you are handed a result and asked whether it is real or noise. A strong answer always has the same shape: state the null, measure how many standard errors the result sits from it, turn that distance into a p-value, then say what the p-value does and does not tell you. By the end of this lesson you will be able to run a z-test for a proportion and a paired or two-sample t-test by hand, choose between a one-sided and a two-sided test, explain type I and type II errors and power, and say why checking twenty metrics at once finds something by accident. Designing the experiment that produces the data (randomization, sample size, guardrails, when to stop) is covered in the A/B testing course. This lesson is the statistics underneath it.

Null, alternative and a test statistic

The null hypothesis, written H0, is the dull explanation, usually stated as an exact value: "the true rate is 80%", "the two engines have the same mean time". A one-sided null such as "the new engine is no faster" covers a range of values; it is tested at its boundary value, no difference, because a false alarm is most likely there. The alternative, H1, is what you will conclude if the data makes the null hard to believe. It can be two-sided ("the rate is not 80%") or one-sided ("the rate is below 80%").