LightGBM › Visualizing LightGBM

Michael BrenndoerferJuly 12, 20257 min read
Machine Learning from ScratchAll chapters
Part 5 of 9Tree-Based Models
Chapter 45LightGBM › Visualizing LightGBM
Reading section 4 of 9 · Visualizing LightGBM · 7 minSection 4 of 9 · 7 min

Book contents

Machine Learning from Scratch9 parts · 66 chaptersView the whole book →
Active chapter 45 of 66

Section 4 of 9 · 7 min read

LightGBM › Visualizing LightGBMOne section · Sections in this chapter4 / 9 ↓
Sections in this chapter9
Reading tools Definitions & inline links

Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.

Article links

Make inline references clickable

Let's create visualizations that demonstrate LightGBM's key characteristics and performance advantages.

Note: The performance comparison below uses relatively small synthetic datasets (10,000 samples, 20 features) for demonstration purposes. LightGBM's true advantages become more apparent with larger, real-world datasets containing millions of samples and hundreds or thousands of features.

Out[7]:
Visualization
Bar chart comparing training times for LightGBM and XGBoost on classification tasks.
Training time comparison for classification tasks showing LightGBM and XGBoost performance. Training time is measured in seconds, with lower values indicating faster training. This comparison demonstrates the computational efficiency of both algorithms on classification problems.
Bar chart comparing training times for LightGBM and XGBoost on regression tasks.
Training time comparison for regression tasks showing LightGBM and XGBoost performance. Training time is measured in seconds.
Out[8]:
Visualization
Bar chart comparing classification accuracy for LightGBM and XGBoost.
Accuracy comparison for classification tasks showing the predictive performance of LightGBM and XGBoost. Accuracy is measured as the proportion of correctly classified instances, with higher values indicating better performance. This comparison demonstrates the classification accuracy of both algorithms.
Bar chart comparing Mean Squared Error for LightGBM and XGBoost on regression tasks.
Mean Squared Error (MSE) comparison for regression tasks showing the prediction error of LightGBM and XGBoost. Lower MSE values indicate better predictive performance.

Note: The results presented above may seem unexpected, especially considering LightGBM's strong reputation for speed and efficiency. However, several key factors help explain why XGBoost outperformed LightGBM in three out of four metrics in this particular comparison.

The results presented above may seem unexpected, especially considering LightGBM's strong reputation for speed and efficiency. However, several key factors help explain why XGBoost outperformed LightGBM in three out of four metrics in this particular comparison.

First, the dataset characteristics matter. In this case, the synthetic datasets generated with make_classification and make_regression are relatively small, each containing 10,000 samples and 20 features. LightGBM's strengths are most pronounced with much larger datasets, especially those with millions of samples, and with higher-dimensional data. On smaller datasets, the overhead introduced by LightGBM's histogram-based binning and its leaf-wise tree growth strategy can sometimes outweigh the performance benefits these features provide.

Second, differences in default parameters between the two models can influence results. Both LightGBM and XGBoost were used with their default settings, but these defaults are optimized for different scenarios. XGBoost's default parameters may be better suited to the size and complexity of the datasets used here, while LightGBM's defaults.such as num_leaves=31.are designed for larger datasets where its efficiency advantages are more likely to be realized.

In summary, LightGBM's true advantages become most apparent in situations involving large datasets (over 100,000 samples), high-dimensional data (over 1,000 features), or sparse datasets, where its histogram-based approach and Exclusive Feature Bundling (EFB) offer significant benefits. Additionally, LightGBM excels with categorical features due to its built-in handling (eliminating the need for one-hot encoding) and is particularly well-suited for memory-constrained environments thanks to its lower memory footprint.

Out[9]:
Visualization
Tree diagram showing level-wise growth with balanced structure where all nodes at each level are expanded before moving to the next level.
Level-wise tree growth (traditional approach) where all nodes at the same depth are expanded before moving to the next level. This method creates balanced trees but may be less efficient as it expands nodes that don't contribute significantly to loss reduction. The visualization shows a complete binary tree structure with nodes at each level expanded uniformly.
Tree diagram showing leaf-wise growth with asymmetric structure where leaves are expanded based on loss reduction rather than depth.
Leaf-wise tree growth (LightGBM approach) where the leaf with the largest loss reduction is expanded next, regardless of its depth. This method creates more asymmetric trees but is more efficient as it focuses computational resources on splits that provide the most benefit. The visualization shows an asymmetric tree structure with nodes expanded based on loss reduction rather than depth.