Ensemble Learning Trade-offs and Why Bagging Reduces Overfitting

Two multiple-choice questions in this assessment concerned ensemble methods: one on the advantages and disadvantages of ensemble learning (for example, interpretability and training cost), and one on why bagging helps prevent overfitting. Answer both as you would explain them in an interview, not only by naming the right option. ### Clarifying Questions - Which ensemble families are in scope: bagging and random forests, boosting, stacking, or all of them? - Should the bagging answer be qualitative, or include the variance argument behind it? ### Part 1 — Advantages and disadvantages of ensembles Explain what ensemble learning gains over a single model and what it costs, covering at least interpretability and training cost. ```hint Error has parts Think about which component of a model's error each kind of ensemble attacks, and what you pay at training time and at prediction time for having many models. ``` #### What This Part Should Cover - The accuracy and robustness benefits, tied to the kind of error each ensemble family reduces - The costs: interpretability, training and prediction compute, memory, latency and maintenance - When an ensemble does not help or is not worth its cost ### Part 2 — Why bagging reduces overfitting Explain why training many models on bootstrap samples and averaging their predictions reduces overfitting, and state what bagging does not fix. ```hint Averaging noisy estimates Consider the variance of an average of several noisy estimates, and what happens to it when the estimates are not independent. ``` #### What This Part Should Cover - The mechanism: bootstrap resampling plus averaging or voting, and its effect on variance - The role of correlation between the base models, and how random forests reduce it - Which base learners benefit, and why bagging does not reduce bias ### What a Strong Answer Covers - Correctly links bagging to variance reduction and boosting to bias reduction - A precise account of the interpretability and compute costs rather than a list of adjectives - The variance-of-an-average argument, including the correlation term - Practical judgement about when a single model is the better choice ### Follow-up Questions - Why does a random forest also sample a subset of features at each split, when bagging already resamples the rows? - Can boosting overfit, and how do you control it? - How would you explain one prediction of a 500-tree forest to a stakeholder? - What is the out-of-bag estimate, and why is it almost free with bagging?