Introduction
Ensemble models are machine learning models that combine predictions from simpler, less powerful models to achieve better results. This approach works because combining multiple models helps reduce both bias and variance, leading to better generalization performance.
Types of Ensemble Models
Ensemble models come in several types:
Bagging (bootstrap aggregating)
Creates random subsets of the dataset, processes them through an algorithm (typically a Decision Tree), and then combines the results.
Boosting
Trains models sequentially, with each new model correcting errors from the previous one.
Stacking
Combines outputs from different models into a “meta-classifier” model, such as logistic regression.
Voting
Uses predictions from multiple models, either by selecting the most-voted class (hard voting) or averaging probability scores (soft voting).
Ensemble models work best with noisy or complex datasets where overfitting and bias are likely, and when robust predictions are essential.
However, these models require significant computational resources and can be difficult to interpret.
Bagging in Ensemble Models
Bagging consists of three main steps:
- Bootstrapping: Creating random subsets of the original dataset, each approximately the same size as the original through data duplication and removal.
- Model Training: Training a separate model (typically a Decision Tree) on each subset.
- Result Aggregation: Combining the predictions from all models—using majority voting for classification tasks or averaging the results for regression tasks.
Bagging effectively reduces variance and prevents overfitting, though it has minimal impact on bias.
Random Forest is the most widely recognized implementation of bagging.
Boosting in Ensemble Models
Unlike bagging, boosting applies models in sequence. Each subsequent model works to correct errors made by previous models, with the final prediction being an average across all models.
Boosting models offer a key advantage: they can reduce variance and bias.
However, they require more complex implementation than bagging models.
The most prominent boosting algorithms include AdaBoost, Gradient Boosting, XGBoost, LightGBM, and CatBoost.
| BAGGING | BOOSTING | |
|---|---|---|
| Training | Models trained independently | Models trained sequentially |
| Error focus | No special attention to errors | Focuses on previous errors |
| Complexity | Less complex | More complex |
| Overfitting risk | Low | Moderate/high |
Using Bagging and Boosting in Python
In this example, we’ll create a synthetic dataset containing two classes with non-linear distributions. We’ll implement bagging with a Random Forest model and boosting with an AdaBoost model, then visualize the decision boundaries for both approaches.
import numpy as np
import matplotlib.pyplot as plt
from sklearn.datasets import make_moons
from sklearn.model_selection import train_test_split
from sklearn.ensemble import RandomForestClassifier, AdaBoostClassifier
from sklearn.tree import DecisionTreeClassifier
from mlxtend.plotting import plot_decision_regions
# Generate synthetic dataset
X, y = make_moons(n_samples=400, noise=0.25, random_state=42)
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.3, random_state=42)
# Bagging model (Random Forest)
bagging_model = RandomForestClassifier(n_estimators=50, random_state=42)
bagging_model.fit(X_train, y_train)
# Boosting model (AdaBoost)
boosting_model = AdaBoostClassifier(estimator=DecisionTreeClassifier(max_depth=1),
n_estimators=50, random_state=42)
boosting_model.fit(X_train, y_train)
# Function to plot decision boundaries side by side
def plot_side_by_side_decision_boundaries(model1, model2, title1, title2):
fig, axes = plt.subplots(1, 2, figsize=(16, 6)) # Create a figure with 2 subplots side by side
# Plot first model (Bagging)
plot_decision_regions(X, y, clf=model1, ax=axes[0], legend=2)
axes[0].set_title(title1)
axes[0].set_xlabel("Feature 1")
axes[0].set_ylabel("Feature 2")
# Plot second model (Boosting)
plot_decision_regions(X, y, clf=model2, ax=axes[1], legend=2)
axes[1].set_title(title2)
axes[1].set_xlabel("Feature 1")
axes[1].set_ylabel("Feature 2")
plt.tight_layout() # Optimize space between subplots
plt.show()
# Plot decision boundaries side by side
plot_side_by_side_decision_boundaries(
bagging_model, boosting_model,
"Decision Boundary - Bagging (Random Forest)",
"Decision Boundary - Boosting (AdaBoost)"
)
The decision boundary of the bagging model appears smoother and less detailed than the boosting model’s boundary. While bagging may miss some complex patterns, boosting runs the risk of overfitting to noise in the data.
Conclusions
Ensemble models combine multiple “weak” machine learning algorithms to create stronger predictions. Among the various combination techniques, bagging and boosting are the most widely used. Choosing between bagging and boosting depends on your problem’s specific goals and dataset characteristics. Choose bagging when robustness and generalization are priorities. Opt for boosting when you need high performance on complex data—just be mindful of potential overfitting.