Bagging
Bagging, short for Bootstrap Aggregating, is an ensemble learning technique designed to improve the stability and accuracy of machine learning models. Instead of training a single model on the entire dataset, bagging trains multiple models independently on different bootstrap samples (random subsets with replacement). Their predictions are then combined, often through majority voting (for classification) or averaging (for regression).
Why it matters
Bagging reduces variance — a common problem in high-complexity models such as decision trees. By aggregating diverse learners, it creates a stronger predictor that generalizes better to unseen data.
Applications
- Random Forests: one of the most successful bagging methods, widely used in classification and regression.
- Finance: risk modeling, fraud detection.
- Healthcare: diagnostic support from patient data.
- General ML tasks: image classification, sentiment analysis, and more.
Challenges
- Increased computational cost due to training multiple models.
- May not help much with high-bias models, since bagging primarily addresses variance.
Bagging is particularly effective with unstable models—those that show high variance when trained on slightly different data samples. Decision trees are the classic example: a single tree can overfit noise, but an ensemble of trees trained on bootstrapped datasets tends to generalize much better. This explains why Random Forests, which combine bagging with feature randomness, became one of the most widely used algorithms in machine learning.
Another important feature of bagging is its ability to reduce overfitting without requiring heavy regularization. By averaging across models, the ensemble smooths out idiosyncrasies of individual learners, leading to more reliable predictions. However, bagging does not inherently address bias; if the base model is too simplistic, the ensemble may still perform poorly.
In modern practice, bagging remains a cornerstone of ensemble learning, often compared with boosting. While boosting focuses on sequentially correcting errors, bagging relies on parallel diversity. Both approaches can be complementary, depending on the problem domain.
📚 Further Reading
- Breiman, L. (1996). Bagging Predictors. Machine Learning.
- Géron, A. (2019). Hands-On Machine Learning with Scikit-Learn, Keras & TensorFlow.
