Z-tests versus t-tests: how each works and when to use it in experiments

Explain what a z-test and a t-test are, and how you decide which one to use. ```hint Compare what each test assumes Write the two test statistics side by side and ask which quantity each one treats as known. Then ask what changes as the sample gets larger. ``` ### Clarifying Questions - Is the metric a proportion, such as a conversion rate, or the mean of a continuous value, such as revenue per user? - Roughly how many observations does each group have: tens of thousands of users in an experiment, or a few dozen in a pilot? - Are you comparing one group with a fixed benchmark, two independent groups, or the same users before and after a change? ### What a Strong Answer Covers - The structure both tests share: an observed difference scaled by its standard error, compared with a reference distribution to get a p-value. - What separates the two tests, including how the spread of the data is handled and which reference distribution each one uses. - When each test fits: sample size, proportions versus means, and one-sample, two-sample and paired designs. - The assumptions behind each test and what to do when they break, such as skewed revenue or unequal variances. - A worked product-experiment example read correctly, without overstating what a p-value means. ### Follow-up Questions - With tens of thousands of users per arm, does the choice between a z-test and a t-test change the conclusion? Why or why not? - Revenue per user is heavily skewed by a few big spenders. Is a t-test still trustworthy, and what would you check or use instead? - You test ten variants against control and one comes out significant at the 5% level. How do you read that result?