Sampling, the CLT and confidence intervals

A data science interviewer hands you a number from a sample, say the average delivery time across 400 orders, and asks how far to trust it. The interviewer is checking whether you ask how the sample was drawn, and whether you can put an error bar on the number without overstating what it means. By the end of this lesson you will be able to spot a biased sample before doing any arithmetic, compute a standard error and a 95% confidence interval by hand, say when the central limit theorem will not rescue you, and bootstrap an interval in a few lines of numpy when the formula is hard to derive.

Population, sample and who got left out

The population is the full set you want to describe: every order last quarter, every user on the account list. A parameter is a number about the population, such as its mean μ. The sample is the part you actually observed, and a statistic is a number computed from it, such as the sample mean x̄. A parameter is fixed. A statistic changes from one sample to the next, and the rest of this lesson is about how much.

Every formula below assumes the sample was drawn at random from the population you care about. Sampling bias is any way the drawing process favors some units over others: