Random Forests and Bagging: Where the Variance Goes
An in-depth look at the mathematical mechanics behind Random Forests, explaining how averaging deep decision trees systematically eliminates variance.
Stock photo for illustration only, not from the actual event
- Random Forests average deep decision trees to eliminate variance rather than bias.
- Bagging and Random Feature Selection are what make decision trees diverse.
- Variance formulas prove why adding the thousandth tree yields diminishing returns.
- Out-of-bag estimation leverages roughly 37 percent of unused bootstrap data.
A random forest averages many deep trees, each fitted to a different random view of the data. This averaging process cannot make individual trees better on their own and it cannot remove bias. What it successfully removes is variance, and the exact amount can be derived through simple arithmetic that also explains why adding the thousandth tree does nothing meaningful.
Bagging is the first mechanism, where each tree is fitted on a bootstrap sample drawn with replacement, ensuring every tree observes a distinct dataset and makes unique mistakes. Random feature selection is the second component, allowing the tree to consider only a random subset of columns at every split, preventing a single dominant feature from dictating the root of every tree.
Stock photo for illustration only, not from the actual event
Individual trees are grown deep and left unpruned, creating low-bias, high-variance estimators that adapt tightly to training data. This volatile behavior serves as the ideal ingredient for averaging, contrasting sharply with the shallow stumps utilized by boosting algorithms.
Taking a held-out row with a true value of 41 predicted by five trees yielding values of 30, 45, 38, 52, and 35, the mean individual squared error calculates to 60.6. Meanwhile, the ensemble prediction averages out to 40, resulting in an ensemble squared error of just 1.0, demonstrating that member disagreement functions as the core mechanism rather than a defect.
From a statistical standpoint, model disagreement is not a flaw to minimize but the primary engine of ensemble learning. If an ensemble consists of identical trees, the spread term drops to zero and ensemble error equals individual error. Every design choice in a random forest intentionally maximizes this member disagreement.
Stock photo for illustration only, not from the actual event
For validation purposes, setting oob_score=True in scikit-learn leverages the approximately 37 percent of rows left out during bootstrap sampling to provide an honest held-out estimate without wasting precious training data. However, if correlated rows such as repeat customer entries span across multiple rows, out-of-bag validation can leak data unless grouped splits are utilized.
Source: Dev.to
Found something wrong in this article? Report an issue with this article
Comments
Leave a Comment