Linear Regression Feature Importance: Smallest P-Value or Largest Coefficient?

You fit an ordinary least squares regression of an outcome Y on 10 predictors and look at the statistical test results for each coefficient: the estimate, its standard error, the t-statistic and the p-value. You are then asked which of these two statements is correct, and why: - **A.** We should focus on the features with the smallest p-values; they are the most important. - **B.** We should focus on the features with the largest coefficient values. The report gives only the two statements. This version assumes the model includes an intercept and that each p-value comes from the usual two-sided t-test of the coefficient against zero. Be ready to say what "most important" should mean here and how you would rank the features. ```hint Change the units Suppose one predictor is re-recorded in different units, such as thousands instead of ones. Ask what happens to its coefficient and to its p-value. ``` ```hint Grow the sample Hold every true coefficient fixed and imagine collecting far more data. Ask what happens to the p-values. ``` ### Clarifying Questions - What does "most important" mean here: the largest change in Y for a realistic change in the feature, the most predictive power on new data, or a causal driver of Y? - Are the 10 predictors measured in the same units and on similar scales? - Are the predictors correlated with one another? - Roughly how many observations does the model use? ### What a Strong Answer Covers - What a coefficient's p-value measures, and how it depends on sample size, noise and the precision of the estimate - Why raw coefficients cannot be compared across predictors measured in different units, and how to put them on a common scale - A clear verdict on both statements, including the conditions under which each one misleads - How correlated predictors, omitted variables and testing ten coefficients at once affect either rule - An importance measure matched to the goal: interpretation, prediction or causal effect ### Follow-up Questions - Two of the predictors are highly correlated with each other. How does that affect each rule, and what would you do? - With millions of rows, every predictor has a p-value far below 0.001. How would you rank the features then? - If the model's purpose is prediction on new data, how would you measure feature importance instead? - If none of the 10 predictors truly affects Y, how likely is at least one p-value below 0.05, and how would you guard against that?