Apprendre AdaBoost: Theory and Implementation

AdaBoost, or Adaptive Boosting, is an ensemble learning technique that sequentially builds a strong classifier by combining multiple weak learners. After each weak learner is trained, AdaBoost adapts by increasing the weights of misclassified samples and decreasing the weights of correctly classified ones. This adaptive weighting ensures that each new learner focuses more on the difficult cases that previous learners struggled with, enabling the ensemble to improve its overall accuracy by correcting mistakes over successive iterations.

Key AdaBoost Formulas: Weighted Error and Learner Weight

AdaBoost relies on two core calculations for each weak learner:

Weighted error ( $\varepsilon_t$ ): measures how much the current learner misclassifies the weighted training data;
Learner weight ( $\alpha_t$ ): determines how much influence the current learner has in the final prediction.

The formulas are:

Weighted error: $\varepsilon_t = \frac{\sum_{i=1}^{n} w_i \cdot [y_i \ne h_t(x_i)]}{\sum_{i=1}^{n} w_i}$
- $w_i$ : weight of sample $i$ ;
- $y_i$ : true label of sample $i$ ;
- $h_t(x_i)$ : prediction of the $t$ -th weak learner for sample $i$ ;
- $[y_i \ne h_t(x_i)]$ : 1 if the prediction is incorrect, 0 otherwise.

Learner weight:

\alpha_t = \frac{1}{2} \ln\left(\frac{1-\varepsilon_t}{\varepsilon_t}\right)

$\alpha_t$ : amount of say the $t$ -th learner gets in the final prediction;
Learners with lower error ( $\varepsilon_t$ ) receive higher $\alpha_t$ , so their votes count more.

These calculations ensure that each new learner focuses on the most challenging samples and that more accurate learners have a greater impact on the ensemble's output.


              1234567891011121314151617181920212223242526272829303132333435363738
            
from sklearn.ensemble import AdaBoostClassifier
from sklearn.tree import DecisionTreeClassifier
from sklearn.datasets import make_classification
from sklearn.model_selection import train_test_split
from sklearn.metrics import accuracy_score
import matplotlib.pyplot as plt

# Generate a synthetic binary classification dataset
X, y = make_classification(n_samples=500, n_features=10, n_informative=7, n_classes=2, random_state=42)

# Split into training and test sets
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.3, random_state=42)

# Create a weak learner: a decision tree with max_depth=1 (stump)
base_estimator = DecisionTreeClassifier(max_depth=1, random_state=42)

# Fit AdaBoost with 50 weak learners and a learning rate of 1.0
ada = AdaBoostClassifier(
    estimator=base_estimator,
    n_estimators=50,
    learning_rate=1.0,
    random_state=42
)
ada.fit(X_train, y_train)

# Predict and evaluate
y_pred = ada.predict(X_test)

# Plot staged learning curve (test accuracy at each boosting iteration)
test_accuracies = [accuracy_score(y_test, y_pred_stage) for y_pred_stage in ada.staged_predict(X_test)]
plt.figure(figsize=(8, 4))
plt.plot(range(1, len(test_accuracies) + 1), test_accuracies, marker='o')
plt.title(f"AdaBoost Test Accuracy over Iterations (Average accuracy: {accuracy_score(y_test, y_pred):.2f})")
plt.xlabel("Number of Weak Learners")
plt.ylabel("Test Accuracy")
plt.grid(True)
plt.tight_layout()
plt.show()

Tout était clair ?

Merci pour vos commentaires !

Section 3. Chapitre 2

Demandez à l'IA

Posez n'importe quelle question ou essayez l'une des questions suggérées pour commencer notre discussion

Suggested prompts:

Can you explain how the weights are updated after each iteration in AdaBoost?

What does the learning rate parameter do in AdaBoost?

How does the choice of weak learner affect AdaBoost's performance?

Glissez pour afficher le menu