Bagging vs Random Forest vs Boosting: How They Differ and Which Overfits More
Company: Quantbot Technologies
Role: Quantitative Researcher
Category: Machine Learning
Difficulty: easy
Interview Round: Technical Screen
Explain how bagging, random forests and boosting differ from one another. Which of them is more prone to overfitting, and why?
```hint Start from one tree
Ask what each method does to the bias and to the variance of a single decision tree.
```
```hint Keep adding trees
Think about what happens to each method's test error as you keep adding trees or rounds.
```
### Clarifying Questions
- Are the base learners decision trees in all three methods, and how deep are they?
- Should the comparison assume default settings, or each method tuned on validation data?
- What kind of data is in view, for example a large clean dataset or noisy targets with a weak signal?
### What a Strong Answer Covers
- How each method builds its ensemble: bootstrap samples, feature subsampling at each split, sequential fitting of errors
- The effect of each method on bias and variance, including why decorrelating the trees helps a random forest
- A clear, justified answer on which method overfits more, with the mechanism behind it
- The hyperparameters and validation practices that control overfitting in each
- Practical differences: parallel versus sequential training, sensitivity to noisy labels, tuning effort
### Follow-up Questions
- Why does adding trees to a random forest not cause overfitting, while adding boosting rounds can?
- What does the out-of-bag error estimate, and when is it unreliable?
- For very noisy targets such as next-day stock returns, which method and settings would you start with?
- How do the learning rate and the number of rounds interact in gradient boosting?
Overview: Compare bagging, random forests and boosting as tree ensemble methods and explain which of them is more prone to overfitting. Tests understanding of bias and variance, bootstrap sampling, feature subsampling, sequential fitting and the hyperparameters that control overfitting.