# Explain Ensemble Learning and Its Main Families Explain why combining multiple models can improve predictions. Compare bagging, random forests, boosting, voting or averaging, and stacking. For each family, discuss how members are trained, how predictions are combined, what kind of error it tends to reduce, and the main leakage or overfitting risks. ### Constraints & Assumptions - Assume supervised learning with a fixed train, validation, and test split. - Base learners may be homogeneous or heterogeneous depending on the method. - The comparison should address both regression and classification where relevant. - Any meta-learner must be trained without seeing in-sample base predictions as if they were out of sample. ### Clarifying Questions to Ask - Is the dominant problem variance, bias, calibration, or robustness? - Can base learners train independently, or is sequential training acceptable? - Are inference latency and model size constrained? ```hint Look at correlated errors Many identical models making the same mistake do not gain much from averaging. ``` ```hint Protect the stack Generate meta-features from folds where each base prediction comes from a model that did not train on that row. ``` ### What a Strong Answer Covers - Diversity and error correlation as the reason aggregation can help. - Accurate distinctions among parallel bagging, randomized forests, sequential boosting, and stacking. - Out-of-fold training for a stacking meta-model. - Trade-offs in accuracy, interpretability, calibration, latency, and failure correlation. ### Follow-up Questions - Why can boosting overfit noisy labels even when training loss keeps improving? - How would you distill an expensive ensemble into one faster model?