Explain an A/B test regression table, p-value and R-squared to a non-technical partner

In a statistics role-play round, the interviewer describes a business use case, gives you time to review the data, and then plays a business partner with no statistical background while you walk through your findings. Expect to read a regression table and to explain p-values, statistical significance and $R^2$ in plain language. The report names this format and these topics but not the use case or the data, so this version supplies a representative A/B test and its regression output. **Use case.** A product team tested a redesigned home screen in its consumer app. Twenty thousand active users were randomly assigned, half to the current screen (control) and half to the new screen (treatment). The outcome is each user's number of app sessions in the 28 days after assignment. Control users averaged 7.9 sessions. The team wants to know whether to launch the new screen. **Regression output** (ordinary least squares; outcome = sessions in 28 days; n = 20,000): | Term | Coefficient | Std. error | t | p-value | | --- | --- | --- | --- | --- | | Intercept | 5.20 | 0.10 | 52.0 | < 0.001 | | `new_screen` | 0.18 | 0.083 | 2.17 | 0.030 | | `notifications_on` | 1.10 | 0.085 | 12.9 | < 0.001 | | `tenure_years` | 0.40 | 0.014 | 28.6 | < 0.001 | $R^2 = 0.05$ `new_screen` is 1 for treatment users and 0 for control users; `notifications_on` is 1 if the user had push notifications enabled; `tenure_years` is the age of the user's account in years. ### Clarifying Questions - Was sessions in 28 days chosen as the primary metric before the test started, and is more sessions actually what the new screen is meant to achieve? - Were `notifications_on` and `tenure_years` recorded before users saw either screen? - Did the split come out close to 10,000 users per group, and were all assigned users measured for the full 28 days? - Were other metrics or user segments examined as well? - How large an improvement would justify the cost of shipping and maintaining the new screen? ### Part 1 — Walk through the findings Explain to the business partner what the table says about the new screen, and whether you would recommend launching it. ```hint Which row answers the question Only one variable in the table was set by the team. Start from the question the team asked. ``` ```hint Size as well as certainty Put the treatment estimate next to the control average and next to its standard error before calling it a win. ``` #### What This Part Should Cover - Identifying the row that answers the team's question, and translating it into absolute and relative terms - The uncertainty around the estimate and what it implies for the decision - Which rows can be read as cause and effect, and why the others cannot - A recommendation with the checks that should come first ### Part 2 — Explain the p-value The business partner asks: "What does p = 0.03 mean? Is there a 97% chance the new screen works?" Answer in plain language. ```hint The world the test assumes Start from what the data would look like if the new screen changed nothing, then ask whether that is the same thing the partner is asking about. ``` #### What This Part Should Cover - A correct, jargon-free explanation of the p-value - Why the "97% chance" reading is wrong - What "significant at 5%" means as a decision rule - The difference between statistical and practical significance ### Part 3 — Explain $R^2$ The business partner says: "$R^2$ is only 0.05. Doesn't that mean the model is useless and we should ignore the result?" Answer in plain language. ```hint What R-squared measures Ask which question $R^2$ answers, and whether the launch decision depends on that question. ``` #### What This Part Should Cover - A plain-language meaning of $R^2$ - Why a low $R^2$ does or does not undermine the treatment estimate in a randomized test - When $R^2$ does matter, and what the other variables contribute ### What a Strong Answer Covers - A one-sentence bottom line first, then the supporting detail, in plain language throughout - Statistically correct interpretations with none of the common misreadings - Effect size, uncertainty and business relevance kept separate - Questions about metric choice and test validity before endorsing a launch - Explanations adapted to the listener, with checks that they were understood ### Follow-up Questions - The partner asks: "Users with notifications on have 1.1 more sessions. Should we turn notifications on for everyone?" How do you respond? - If the p-value for `new_screen` had been 0.08, what would you tell the team? - The analyst looked at ten different metrics, and this is the only one with p below 0.05. Does that change your recommendation? - Why include `notifications_on` and `tenure_years` in the regression at all, if randomization already balances the groups?