Building Steps of Random Forests

Random Forests construct an ensemble of decision trees through a series of structured steps that introduce randomness and diversity. The main building steps are:

1. Bootstrap Sampling:
   • Draw a random sample from the original dataset with replacement to create a bootstrap sample for each tree;
   • Each tree receives a different bootstrap sample, so some data points may appear multiple times while others may be omitted in a given tree's training set.
2. Feature Subset Selection at Each Split:
   • When growing each tree, at every split, select a random subset of features instead of considering all features;
   • The best split is chosen only from this random subset, forcing each tree to consider different features and split points.
3. Tree Training:
   • Each tree is trained independently on its bootstrap sample, using only the selected features at each split;
   • Trees are grown to their specified depth or until other stopping criteria are met.
4. Aggregation of Predictions:
   • For classification tasks, collect the predicted class from each tree and use majority vote to determine the final class;
   • For regression tasks, average the predictions from all trees to produce the final output.

This process ensures that each tree in the forest is unique, both in the data it sees and the features it uses, which leads to a more robust and accurate ensemble model.

Aggregation Formula for Random Forest Regression

In Random Forest regression, the final prediction for a data point is the average of the predictions made by all individual trees. If you have $n$ trees and each tree predicts a value $\hat{y}_i$ for an input $x$, the aggregated prediction $\hat{y}$ is:

This averaging helps reduce the impact of errors from individual trees, leading to more stable and accurate predictions.