Random Forests & Ensembles
A single decision tree is unstable: change the data slightly and the tree can be rebuilt into something quite different. The fix is simple. Grow a few hundred trees, each on a slightly different sample of the data, and let them vote. The individual trees' errors largely cancel, and the vote is far more stable than any one tree. That is a random forest.
Why averaging many trees reduces error
A single deep tree is a high-variance learner: it fits its training sample almost perfectly, including the noise, so a different sample would give a visibly different tree. Averaging reduces that variance. If you have many models whose errors are partly independent, the error of their average is far smaller than a typical single model's, because the overshoots and undershoots cancel. But this needs independence: trees trained on the same data all make the same mistakes, so averaging identical trees gains nothing. A random forest is designed to make its trees differ, so that their errors are less correlated.
Two sources of randomness
- Bagging (bootstrap aggregating). Each tree is grown on a bootstrap sample: draw n rows with replacement from the training set, so each tree is trained on a slightly different dataset (some rows twice, others not at all). That alone decorrelates the trees a little.
- Feature randomness. At every split, a tree is only allowed to consider a random subset of the predictors, not all of them. This step separates a random forest from plain bagging. It stops one dominant variable from being the top split in every tree, so different trees are built on different predictors and the correlation between trees drops further. Lower correlation means the average cancels more noise.
Prediction is then a plain vote: to classify a new case, run it down all the trees and take the majority (for regression, average the numbers). Individual trees draw jagged, contradictory boundaries, but the vote proportion varies smoothly, and a smooth boundary is a sign of low variance.
🌲 Grow a forest, one tree at a time
Two classes (indigo and orange) with a curved true boundary plus label noise. Each tree is deep and jagged on its own. The shading is the ensemble's vote proportion, so it becomes smoother as trees are added. Below: the forest's test accuracy against the number of trees. It climbs and then plateaus without overfitting. The dashed line is the average single tree's accuracy, and the forest soon rises above it.
The forest's accuracy is above the average single tree's from very early on. That gap comes entirely from variance reduction: the vote cancels errors that each tree makes on its own. Adding more trees also does not make the forest worse: the curve rises and flattens. The number of trees is a matter of budget (more trees are steadier but slower), not of overfitting. The settings that can cause overfitting are inside each tree: its depth and minimum leaf size.
Out-of-bag error: validation for free
Bagging has a useful side effect. Each bootstrap sample leaves out about 37% of the rows by chance, so every row is "out-of-bag" for roughly a third of the trees, the ones that never saw it. Predict each row using only those trees, compare with the truth, and you have a held-out error estimate without a separate validation set. Out-of-bag (OOB) error is a convenient default and usually tracks cross-validated error closely, but for a final reported number a separate held-out test set is still the standard.
Variable importance — and its asterisks
A forest can also rank the predictors, either by adding up how much each one reduced impurity across all splits or, better, by permutation importance: shuffle one column and see how much accuracy falls. Read these rankings with care. Impurity-based importance is biased toward continuous and many-valued predictors. And when two predictors are correlated, the forest splits the credit between them, so an important variable can look modest because a correlated predictor took some of its share. Importance tells you what helped the model predict, not what causes the outcome. For causal questions you need a study design that can answer them.
Importance also tells you nothing about the shape of a relationship, only its size. For shape, a partial-dependence plot moves one predictor across its range and, at each value, averages the forest's prediction over the other predictors as they occur in the data. The resulting curve shows the direction and the bends the forest learned. The same caution applies: the curve describes the model, not the world.
Boosting: trees grown in sequence
A forest grows its trees independently of one another and averages away their variance. Boosting grows them in sequence: fit a small tree, look at what it got wrong, fit the next tree to those residuals, and add it in at a fraction of its full size (the learning rate). Every tree is deliberately weak, and a few hundred weak trees, each correcting the ones before it, add up to a strong model. On tabular data, forests are usually compared with gradient boosting and the libraries that implement it (XGBoost, LightGBM).
The main difference is what the number of trees does in each. Grow both on the same 200 rows and follow the test error. The bagged forest's error falls from .32 with a single tree to .19 by about fifteen trees, and stays there all the way to 300. The boosted model's error reaches its minimum of .18 after five trees and then climbs, reaching .24 by 500, a third worse than its own best. It reached 100% training accuracy at tree 43. So in a forest more trees only cost computing time, but in boosting more trees lead to overfitting.
A smaller learning rate delays the overfitting but does not prevent it. On the same data the test error reaches its minimum at five trees with a rate of 0.3, at sixteen with 0.1 and at fifty-nine with 0.03, and in all three it rises afterward. So boosting software includes early stopping and a validation curve, and a forest is still the better first model when you want a strong result without tuning anything.
Why it matters: a random forest is often the strongest model you can fit with almost no tuning. Bag many deep trees, add feature randomness so they differ, average the votes, and get the OOB error at no extra cost. You lose interpretability: the single tree's readable flowchart becomes a black box that you can examine only through importance scores and partial-dependence plots. When accuracy and stability matter more than a printable set of rules, use the forest.
Common questions
Why doesn't a random forest overfit as you add more trees?
Because adding trees reduces variance without adding the kind of flexibility that fits noise. Each tree is a full, high-variance model that overfits its own bootstrap sample in its own way. Averaging their votes cancels those individual errors, so the ensemble's prediction becomes stable. More trees only make that average more stable. They do not let the forest draw ever-finer boundaries around single points, so test accuracy rises and then levels off rather than peaking and falling. The number of trees affects computing time (more trees are steadier but slower), not overfitting. Overfitting is controlled by each tree's depth or minimum leaf size.
Do I need to tune a random forest, or are the defaults good enough?
The defaults are good, which is a large part of why the model is so popular, but check one of them. The important setting is the number of predictors each split may consider, called mtry in R and max_features in Python, because it controls how different the trees are from one another. The usual default is the square root of the number of predictors for classification and a third of them for regression. Python has a trap here. RandomForestRegressor ships with max_features=1.0, meaning every predictor at every split. Its own documentation notes that this is equivalent to bagged trees rather than a random forest, so set it to "sqrt" or something like 0.3 if you want the feature randomness this lesson describes. Beyond that, grow enough trees that the out-of-bag error has flattened, leave tree depth alone unless the data is small and noisy, and spend the effort you saved on choosing good predictors.
Can I trust a random forest's variable importance?
Treat it as a useful hint, not as proof of causation or an exact ranking. Standard impurity-based (Gini) importance is biased toward high-cardinality and continuous predictors. And when two predictors are correlated, the forest splits its credit between them, so a variable that matters can look weak when a correlated predictor has absorbed part of its importance. Permutation importance (shuffle one column and see how much accuracy drops) is generally more reliable, but it too can mislead when predictors are strongly correlated. Importance tells you what helped the model predict, which is not the same as what causes the outcome. A causal claim has to come from how the study was designed, not from the model.