XGBoost › Example

Michael BrenndoerferJuly 18, 20257 min read
Machine Learning from ScratchAll chapters
Part 8 of 15Tree-Based Models
Chapter 62XGBoost › Example
Reading section 5 of 10 · Example · 7 minSection 5 of 10 · 7 min

Section 5 of 10 · 7 min read

XGBoost › ExampleSection view · Sections in this chapter5 / 10 ↓
Reading tools Definitions, links & notes

Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.

Article links

Make inline references clickable

Notes & comments

Let's work through a concrete example to demonstrate how XGBoost builds trees and makes predictions. We'll use a simple regression problem with a small dataset to make the calculations manageable.

Consider a dataset with 6 samples for predicting house prices based on two features: square footage (sqft) and number of bedrooms (bedrooms):

SamplesqftbedroomsPrice (y)
110002200000
212003250000
38001150000
415003300000
511002220000
613004280000

We'll use mean squared error as our loss function: l(y,y^)=12(y−y^)2l(y, \hat{y}) = \frac{1}{2}(y - \hat{y})^2. The first and second derivatives are:

  • gi=∂l∂y^=−(yi−y^i)g_i = \frac{\partial l}{\partial \hat{y}} = -(y_i - \hat{y}_i)
  • hi=∂2l∂y^2=1h_i = \frac{\partial^2 l}{\partial \hat{y}^2} = 1

Step 1: Initialize with the mean We start by predicting the mean of all target values:

y^(0)=200000+250000+150000+300000+220000+2800006=233,333\hat{y}^{(0)} = \frac{200000 + 250000 + 150000 + 300000 + 220000 + 280000}{6} = 233,333

Step 2: Calculate gradients for the first tree For each sample, we calculate the gradient:

  • Sample 1: g1=−(200000−233333)=33,333g_1 = -(200000 - 233333) = 33,333, h1=1h_1 = 1
  • Sample 2: g2=−(250000−233333)=−16,667g_2 = -(250000 - 233333) = -16,667, h2=1h_2 = 1
  • Sample 3: g3=−(150000−233333)=83,333g_3 = -(150000 - 233333) = 83,333, h3=1h_3 = 1
  • Sample 4: g4=−(300000−233333)=−66,667g_4 = -(300000 - 233333) = -66,667, h4=1h_4 = 1
  • Sample 5: g5=−(220000−233333)=13,333g_5 = -(220000 - 233333) = 13,333, h5=1h_5 = 1
  • Sample 6: g6=−(280000−233333)=−46,667g_6 = -(280000 - 233333) = -46,667, h6=1h_6 = 1

Step 3: Build the first tree Let's say we create a simple tree that splits on sqft < 1100:

  • Left leaf (sqft < 1100): samples 1, 3, 5
  • Right leaf (sqft ≥ 1100): samples 2, 4, 6

For the left leaf:

w1=−33333+83333+133331+1+1+λ=−1300003+λw_1 = -\frac{33333 + 83333 + 13333}{1 + 1 + 1 + \lambda} = -\frac{130000}{3 + \lambda}

For the right leaf:

w2=−−16667−66667−466671+1+1+λ=−−1300003+λ=1300003+λw_2 = -\frac{-16667 - 66667 - 46667}{1 + 1 + 1 + \lambda} = -\frac{-130000}{3 + \lambda} = \frac{130000}{3 + \lambda}

Note: The negative sign in the right leaf calculation comes from the fact that we're taking the negative of a negative sum, which results in a positive value. This reflects that samples in the right leaf (larger houses) need positive corrections to their predictions.

If we set λ=1\lambda = 1 (L2 regularization):

  • w1=−1300004=−32,500w_1 = -\frac{130000}{4} = -32,500
  • w2=1300004=32,500w_2 = \frac{130000}{4} = 32,500

Step 4: Update predictions After the first tree:

  • Sample 1: y^(1)=233333+(−32500)=200,833\hat{y}^{(1)} = 233333 + (-32500) = 200,833
  • Sample 2: y^(1)=233333+32500=265,833\hat{y}^{(1)} = 233333 + 32500 = 265,833
  • Sample 3: y^(1)=233333+(−32500)=200,833\hat{y}^{(1)} = 233333 + (-32500) = 200,833
  • Sample 4: y^(1)=233333+32500=265,833\hat{y}^{(1)} = 233333 + 32500 = 265,833
  • Sample 5: y^(1)=233333+(−32500)=200,833\hat{y}^{(1)} = 233333 + (-32500) = 200,833
  • Sample 6: y^(1)=233333+32500=265,833\hat{y}^{(1)} = 233333 + 32500 = 265,833

We can see that the first tree has started to correct the initial predictions, moving them closer to the actual values. The process continues iteratively, with each subsequent tree focusing on the remaining prediction errors.

VisualizationOut [11]
CollapseExpand
Bar chart comparing actual house prices with XGBoost predictions after first tree.
House price prediction comparison showing actual prices, initial mean prediction, and predictions after the first XGBoost tree. The plot demonstrates how XGBoost corrects the initial mean prediction by learning patterns in the data, with samples split based on square footage threshold.
VisualizationOut [12]
CollapseExpand
Diagram showing decision tree structure with split conditions and leaf weights.
Visual representation of the first decision tree structure with split condition, leaf assignments, and calculated leaf weights. The tree splits on square footage < 1100, assigning samples to left and right leaves with weights calculated using XGBoost's second-order optimization formula.

Comments

Add to the conversation

A question, a correction, or an example from your own work.

1 comment

  1. Sharath MohanMember

    Thank you so much for breaking down the math behind XGB. The visualization after the math, helped me to actually understand how the algorithm works.

Newsletter

Stay up to date

Get new articles, book updates, and notes on AI and technology.

Unsubscribe at any time. Privacy policy