LightGBM › Example

Let's work through a concrete mathematical example to demonstrate how LightGBM builds trees and makes predictions. We'll use a simple regression problem with a small dataset to make the calculations transparent.

Note: This is a simplified pedagogical example designed to illustrate the core concepts. Real-world applications typically involve much larger datasets and more complex feature interactions.

Suppose we have the following training data for predicting house prices based on size and age:

The initial prediction for all samples is the mean of the target variable:

$$\hat{y}^{(0)} = \frac{1}{n}\sum_{i=1}^{n} y_i = \frac{200 + 280 + 350 + 220 + 300 + 450 + 260 + 290}{8} = 280$$

For regression with squared error loss, the gradient is the negative residual:

$$g_i = -\frac{\partial l(y_i, \hat{y}_i^{(0)})}{\partial \hat{y}_i^{(0)}} = -(y_i - \hat{y}_i^{(0)}) = \hat{y}_i^{(0)} - y_i$$

Example calculation for House 1:

• $y_1 = 200$ (actual price)
• $\hat{y}_1^{(0)} = 280$ (initial prediction)
• $g_1 = \hat{y}_1^{(0)} - y_1 = 280 - 200 = 80$

Calculating gradients for each sample:

LightGBM evaluates potential splits and calculates the gain for each. Let's consider splitting on the 'size' feature. We sort the data by size and consider splits at midpoints between consecutive values:

Potential split points: 110, 130, 145, 155, 170, 190, 225

Let's calculate the gain for splitting at size ≤ 155:

Left child (size ≤ 155): Houses 1, 4, 7, 2, 8

• $\sum_{i \in I_L} g_i = 80 + 60 + 20 + 0 + (-10) = 150$
• $|I_L| = 5$

Right child (size > 155): Houses 5, 3, 6

• $\sum_{i \in I_R} g_i = (-20) + (-70) + (-170) = -260$
• $|I_R| = 3$

Original node:

• $\sum_{i \in I} g_i = 80 + 0 + (-70) + 60 + (-20) + (-170) + 20 + (-10) = -110$
• $|I| = 8$

Let's do the calculations step by step:

• Left child gradient sum: $80 + 60 + 20 + 0 + (-10) = 150$
• Right child gradient sum: $(-20) + (-70) + (-170) = -260$
• Original node gradient sum: $80 + 0 + (-70) + 60 + (-20) + (-170) + 20 + (-10) = -110$

Note: For this simplified example, we assume the Hessian (second-order gradient) equals 1 for all samples, which reduces the gain formula to a simpler form. In practice, LightGBM uses the full second-order approximation.

Using the simplified gain formula (assuming hessian = 1 for all samples):

$$\text{Gain} = \frac{(\sum_{i \in I_L} g_i)^2}{|I_L|} + \frac{(\sum_{i \in I_R} g_i)^2}{|I_R|} - \frac{(\sum_{i \in I} g_i)^2}{|I|}$$ $$\text{Gain} = \frac{150^2}{5} + \frac{(-260)^2}{3} - \frac{(-110)^2}{8}$$ $$\text{Gain} = \frac{22500}{5} + \frac{67600}{3} - \frac{12100}{8}$$ $$\text{Gain} = 4500 + 22533.33 - 1512.5 = 25520.83$$

The gain here quantifies how much the split at size ≤ 155 improves the model's objective function (i.e., reduces the loss) compared to not splitting. In gradient boosting, each split is chosen to maximize this gain, which represents the improvement in the model's fit to the data.

• A higher gain means the split creates child nodes that are more "pure" (i.e., the gradients within each child are more similar), leading to better predictions.
• The gain calculation uses the sum of gradients in each child and the parent.
• In this example, a gain of 25,520.83 indicates a substantial reduction in the loss function, making this split highly desirable.

In summary, the gain measures the effectiveness of a split: the larger the gain, the more the split helps the model learn from the data.

LightGBM would evaluate all possible splits and select the one with the highest gain. In this example, the split at size ≤ 155 shows a substantial gain of 25,520.83, indicating that this split significantly reduces the loss function.

The algorithm would then:

1. Perform this split, creating two child nodes
2. Continue the process by finding the next best split among all current leaves
3. Focus on the leaf that would provide the largest gain reduction

This demonstrates how LightGBM's leaf-wise growth strategy prioritizes the most informative splits, building trees that focus computational effort where it can achieve the greatest loss reduction.