Boosted Trees › Visualizing Boosted Trees

Michael BrenndoerferJuly 3, 20254 min read
Machine Learning from ScratchAll chapters
Part 5 of 9Tree-Based Models
Chapter 43Boosted Trees › Visualizing Boosted Trees
Reading section 3 of 8 · Visualizing Boosted Trees · 4 minSection 3 of 8 · 4 min

Book contents

Machine Learning from Scratch9 parts · 66 chaptersView the whole book →
Active chapter 43 of 66

Section 3 of 8 · 4 min read

Boosted Trees › Visualizing Boosted TreesOne section · Sections in this chapter3 / 8 ↓
Sections in this chapter8
Reading tools Definitions & inline links

Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.

Article links

Make inline references clickable

Let's create visualizations that demonstrate how boosted trees work and how they differ from other ensemble methods.

Out[5]:
Visualization
Scatter plot of one-dimensional regression training data with four differently styled step functions labeled after one, two, three, and four boosted trees.
Sequential learning in a boosted-tree regressor. The four prediction functions show the ensemble after each of its first four shallow trees. Each new stage changes the current function to correct residual errors, while the complete training dataset remains visible for context.
Scatter plot of the same training data with a dashed single-tree prediction and a solid boosted-ensemble prediction using one hundred trees.
A single depth-three decision tree compared with a 100-tree boosted ensemble on the same training data. The single tree is visibly stepwise, whereas repeated residual correction produces a more detailed ensemble prediction.

Now let's visualize how the number of trees and learning rate affect the boosting performance:

Out[6]:
Visualization
Training and test mean squared error curves from 10 to 200 boosted trees, with a star and dashed vertical reference marking the tree count with minimum test error.
Training and test error for gradient-boosted regression at learning rate 0.1 as the ensemble grows from 10 to 200 trees. Training error decreases steadily, while the marked minimum of the test curve identifies the best generalization performance in this range. The widening gap after that point signals increasing overfit.