Explain the bias-variance trade-off and use it to compare bagging with boosting. Describe what each ensemble does to its base learners, why its errors may improve, and when it can fail. ### Constraints & Assumptions - Discuss expected prediction error rather than calling a model simply “high bias” from one sample. - Include the role of correlation among base learners in bagging. - Distinguish gradient boosting's sequential residual fitting from averaging independent models. ### Clarifying Questions to Ask - Is the loss squared error, classification loss, or another objective? - Are the base learners deep unstable trees or shallow weak learners? - Is the dataset noisy, imbalanced, or small? - Are interpretability and training latency part of the trade-off? ```hint Start from the decomposition For squared error, separate irreducible noise, squared bias, and variance; then ask which term each ensemble most directly targets. ``` ### What a Strong Answer Covers - The squared-error decomposition and its assumptions. - Bootstrap sampling, feature randomness, averaging, and correlation in bagging. - Sequential loss reduction, shrinkage, and weak learners in boosting. - Overfitting, label noise, depth, learning rate, and early stopping. - Concrete guidance on when random forests or boosted trees are preferable. ### Follow-up Questions - Why does averaging highly correlated trees provide limited variance reduction? - Can boosting reduce variance as well as bias? - How do out-of-bag estimates work for bagged trees? - What happens when the base learner is already very stable?