Random Forests & Ensembles
One decision tree is clever but jumpy — nudge the data and it rebuilds itself into something new. The fix is almost comically simple: don't grow one tree, grow a few hundred, each on a slightly different slice of the data, and let them vote. The individual jitters cancel, and out of a crowd of unreliable trees comes a strikingly stable predictor. That's a random forest, and it's the wisdom of crowds applied to models.
Why a crowd beats an expert
A single deep tree is a high-variance learner: it fits its training sample almost perfectly, including the noise, so a different sample would give a visibly different tree. Averaging is the antidote. If you have many models whose errors are somewhat independent, their average error is far smaller than any one model's, because the over-shoots and under-shoots cancel. The catch is that word "independent": trees trained on the same data all make the same mistakes, so averaging clones buys you nothing. A random forest's whole design is a recipe for making its trees disagree in useful, uncorrelated ways.
Two dashes of randomness
- Bagging (bootstrap aggregating). Each tree is grown on a bootstrap sample: draw n rows with replacement from the training set, so each tree sees a slightly different dataset (some rows twice, others not at all). That alone decorrelates the trees a little.
- Feature randomness. At every split, a tree is only allowed to consider a random subset of the predictors, not all of them. This is the secret ingredient: it stops one dominant variable from being the top split in every single tree, forcing the forest to explore, and driving the trees' correlation down further. Lower correlation means the average cancels more noise.
Then prediction is a plain vote: to classify a new case, run it down all the trees and take the majority (for regression, average the numbers). Where individual trees draw jagged, contradictory boundaries, the vote proportion varies smoothly, and a smooth boundary is exactly what a low-variance model looks like.
🌲 Grow a forest, one tree at a time
Two classes (indigo and orange) with a curved true boundary plus label noise. Each tree is deep and jagged on its own; the shading is the ensemble's vote proportion, so it smooths as trees accumulate. Below: ensemble test accuracy vs number of trees — it climbs, then plateaus (not overfits). The dashed line is the average single tree's accuracy; watch the crowd overtake the individual.
Two things stand out. First, the ensemble's accuracy sits above the average single tree from very early on — that gap is pure variance reduction, error the crowd canceled that no individual tree could. Second, adding trees never makes it worse: the curve rises and flattens. The number of trees is a budget (more is steadier but slower), not an overfitting dial. The dials that can overfit live inside each tree: its depth and minimum leaf size.
Out-of-bag error: validation for free
Bagging hands you a bonus. Because each bootstrap sample leaves out about 37% of the rows by chance, every row is "out-of-bag" for roughly a third of the trees, the ones that never saw it. Predict each row using only those trees, compare to the truth, and you have an honest, held-out-style error estimate without ever carving off a separate validation set. Out-of-bag (OOB) error is a handy default that usually tracks cross-validated error closely — though for a final reported number, a clean held-out test set is still the standard.
Variable importance — and its asterisks
A forest can also rank which predictors mattered, by tallying how much each one reduced impurity across all splits, or, better, by permutation importance: shuffle one column, and see how much accuracy falls. Useful, but read it with care. Impurity-based importance is biased toward continuous and many-valued predictors, and when two predictors are correlated the forest splits the credit between them, so a genuinely important variable can look modest because a collaborator stole some of its thunder. Importance tells you what helped the model predict — not what causes the outcome. For causation you still need design, not a forest.
Importance also carries no information about the shape of a relationship, only its weight. For shape, a partial-dependence plot sweeps one predictor across its range and, at each value, averages the forest's prediction over the other predictors as they actually occur in the data. The curve that falls out shows the direction and the bends the forest learned, with the same caveat attached: it describes the model, not the world.
The other way to build a crowd: boosting
A forest grows its trees in parallel, each one blind to the others, and averages away their variance. Boosting grows them in sequence instead: fit a small tree, look at what it got wrong, fit the next tree to those residuals, and add it in at a fraction of its full size (the learning rate). Every tree is deliberately weak, and a few hundred weak trees correcting one another add up to a strong model. Gradient boosting, and the libraries that implement it (XGBoost, LightGBM), is what a forest usually gets benchmarked against on tabular data.
The difference worth carrying away is what the tree count does in each. Grow both on the same 200 rows and follow the test error. The bagged forest falls from .32 with a single tree to .19 by about fifteen trees, and sits there all the way to 300. The boosted model bottoms out at .18 after five trees and then climbs, reaching .24 by 500, a third worse than its own best, having hit 100% training accuracy at tree 43. More trees is a budget in a forest and an overfitting dial in boosting.
Shrinking the learning rate buys time rather than immunity. On that same data the test error bottoms out at five trees with a rate of 0.3, at sixteen with 0.1, at fifty-nine with 0.03, and in all three it turns upward afterward. That is why boosting ships with early stopping and a validation curve, and why a forest is still the better first model when you want a strong answer without tuning anything.
Why it matters: a random forest is often the strongest model you can fit with almost no tuning — bag many deep trees, add feature randomness so they disagree, average the votes, and read OOB error for free. The price is interpretability: you trade the single tree's readable flowchart for a black box you probe with importance scores and partial-dependence plots. When accuracy and robustness matter more than a printable rulebook, the crowd wins.
Common questions
Why doesn't a random forest overfit as you add more trees?
Because adding trees reduces variance without adding the kind of flexibility that fits noise. Each tree is a full, high-variance model that overfits its own bootstrap sample in its own way; averaging their votes cancels those individual quirks, so the ensemble's prediction settles down. Crucially, more trees only make that average more stable — they do not let the forest carve ever-finer boundaries around single points, so test accuracy rises and then plateaus rather than peaking and falling. The number of trees is a budget knob (more is steadier but slower), not an overfitting dial. The real overfitting controls are each tree's depth or minimum leaf size.
Do I need to tune a random forest, or are the defaults good enough?
The defaults are genuinely good, which is a large part of why the model is so popular, though one of them is worth a look. The setting that matters is how many predictors each split may consider, called mtry in R and max_features in Python, because that is the knob doing the decorrelating; the usual default is the square root of the number of predictors for classification and a third of them for regression. Python holds a trap here. RandomForestRegressor ships with max_features=1.0, meaning every predictor at every split, which its own documentation notes is equivalent to bagged trees rather than a random forest, so set it to "sqrt" or something like 0.3 if you want the feature randomness this lesson describes. Otherwise: grow enough trees that the out-of-bag error has flattened, leave tree depth alone unless the data is small and noisy, and spend the effort you saved on your predictors instead.
Can I trust a random forest's variable importance?
Treat it as a useful hint, not proof of causation or a clean ranking. Standard impurity-based (Gini) importance is biased toward high-cardinality and continuous predictors, and when two predictors are correlated the forest splits its credit between them, so an important variable can look weak simply because a collaborator absorbed some of its share. Permutation importance — shuffle one column and see how much accuracy drops — is generally more trustworthy, but it too can mislead under strong correlation. Importance tells you what helped the model predict, which is not the same as what causes the outcome; for causal questions you still need design, not a forest.