LightGBM › Implementation in LightGBM

Michael BrenndoerferJuly 12, 20256 min read
Machine Learning from ScratchAll chapters
Part 5 of 9Tree-Based Models
Chapter 45LightGBM › Implementation in LightGBM
Reading section 6 of 9 · Implementation in LightGBM · 6 minSection 6 of 9 · 6 min

Book contents

Machine Learning from Scratch9 parts · 66 chaptersView the whole book →
Active chapter 45 of 66

Section 6 of 9 · 6 min read

LightGBM › Implementation in LightGBMOne section · Sections in this chapter6 / 9 ↓
Sections in this chapter9
Reading tools Definitions & inline links

Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.

Article links

Make inline references clickable

LightGBM provides both a scikit-learn compatible interface and its native API. Let's demonstrate both approaches:

In[12]:
Code
import numpy as np
import pandas as pd
from sklearn.datasets import make_classification
from sklearn.model_selection import train_test_split

# Note: Using a small dataset for demonstration
# In practice, LightGBM excels with larger datasets
np.random.seed(42)
X, y = make_classification(
    n_samples=1000,
    n_features=10,
    n_informative=8,
    n_redundant=2,
    random_state=42,
)

# Convert to DataFrame with feature names to avoid warnings
feature_names = [f"feature_{i}" for i in range(10)]
X_df = pd.DataFrame(X, columns=feature_names)
X_train, X_test, y_train, y_test = train_test_split(
    X_df, y, test_size=0.2, random_state=42
)

Scikit-learn Interface

In[13]:
Code
# Using LightGBM with scikit-learn interface
lgb_classifier = lgb.LGBMClassifier(
    n_estimators=10,  # Number of boosting rounds
    learning_rate=0.1,  # Step size shrinkage
    max_depth=6,  # Maximum tree depth
    num_leaves=31,  # Maximum number of leaves
    min_child_samples=20,  # Minimum samples in a leaf
    subsample=0.8,  # Row sampling ratio
    colsample_bytree=0.8,  # Column sampling ratio
    reg_alpha=0.1,  # L1 regularization
    reg_lambda=0.1,  # L2 regularization
    random_state=42,
    verbose=-1,  # Suppress output
)

lgb_classifier.fit(X_train, y_train)

y_pred = lgb_classifier.predict(X_test)
y_pred_proba = lgb_classifier.predict_proba(X_test)
Out[14]:
Console
LightGBM Classification Results:
Accuracy: 0.7900

Classification Report:
              precision    recall  f1-score   support

           0       0.83      0.76      0.79       106
           1       0.75      0.82      0.79        94

    accuracy                           0.79       200
   macro avg       0.79      0.79      0.79       200
weighted avg       0.79      0.79      0.79       200

The scikit-learn interface gives a familiar API for users already comfortable with scikit-learn. The accuracy score indicates strong predictive performance on this classification task. The classification report shows precision, recall, and F1-scores for each class. High precision indicates that when the model predicts a class, it's usually correct, while high recall means the model successfully identifies most instances of each class.

Native LightGBM API

In[15]:
Code
# Using LightGBM native API for more control
train_data = lgb.Dataset(X_train, label=y_train)
test_data = lgb.Dataset(X_test, label=y_test, reference=train_data)

# Define parameters
params = {
    "objective": "binary",
    "metric": "binary_logloss",
    "boosting_type": "gbdt",
    "num_leaves": 31,
    "learning_rate": 0.1,
    "feature_fraction": 0.8,
    "bagging_fraction": 0.8,
    "bagging_freq": 5,
    "verbose": -1,
}

model = lgb.train(
    params,
    train_data,
    valid_sets=[test_data],
    num_boost_round=100,
    callbacks=[lgb.early_stopping(stopping_rounds=10)],
)

y_pred_native = model.predict(X_test, num_iteration=model.best_iteration)
y_pred_binary = (y_pred_native > 0.5).astype(int)
Out[16]:
Console
Native LightGBM API Results:
Accuracy: 0.8650
Best iteration: 75

The native API offers more fine-grained control over the training process and is useful for advanced users who need to customize training behavior. The model achieved comparable accuracy to the scikit-learn interface. The best iteration value shows where early stopping determined the optimal number of boosting rounds, helping prevent overfitting by stopping training when validation performance plateaus.

Feature Importance

In[17]:
Code
feature_importance = lgb_classifier.feature_importances_

# Create importance DataFrame using the actual feature names
importance_df = pd.DataFrame(
    {"feature": X_train.columns, "importance": feature_importance}
).sort_values("importance", ascending=False)
Out[18]:
Console
Feature Importance:
     feature  importance
0  feature_0          32
7  feature_7          27
2  feature_2          26
8  feature_8          20
3  feature_3          18
6  feature_6          15
1  feature_1          14
5  feature_5          13
9  feature_9           9
4  feature_4           8

Feature importance scores help identify which features contribute most to the model's predictions. Features with higher importance values have a stronger influence on the model's decision-making process. This information is valuable for feature selection, model interpretation, and understanding which aspects of your data drive predictions. In practice, you might consider removing features with very low importance to simplify the model without sacrificing performance.

Cross-Validation

In[19]:
Code
cv_scores = cross_val_score(
    lgb_classifier, X_train, y_train, cv=5, scoring="accuracy"
)
Out[20]:
Console
Cross-validation scores: [0.85625 0.8     0.8375  0.85625 0.80625]
Mean CV accuracy: 0.8313 (+/- 0.0481)

Cross-validation provides a more robust estimate of model performance by evaluating the model on multiple train-test splits. The mean CV accuracy represents the average performance across all folds, while the standard deviation (shown as +/-) indicates the variability in performance. Low variability suggests the model performs consistently across different data subsets, which is a good indicator of generalization capability. If the CV scores vary significantly, it might indicate that the model is sensitive to the specific training data or that hyperparameter tuning is needed.

Key Parameters

Below are the main parameters that affect how LightGBM works and performs.

  • n_estimators: Number of boosting rounds or trees to build (default: 100). More trees generally improve performance but increase training time and memory usage. Start with 100 and increase if validation performance continues to improve.
  • learning_rate: Step size shrinkage used to prevent overfitting (default: 0.1). Smaller values require more trees but often lead to better performance. Typical values range from 0.01 to 0.3.
  • max_depth: Maximum depth of each tree (default: -1, meaning no limit). Limiting depth helps prevent overfitting. Values between 3 and 10 work well for most datasets.
  • num_leaves: Maximum number of leaves in each tree (default: 31). This is the main parameter controlling tree complexity in LightGBM's leaf-wise growth. Should be less than 2max_depth2^{\text{max\_depth}} to prevent overfitting.
  • min_child_samples: Minimum number of samples required in a leaf node (default: 20). Higher values prevent overfitting by requiring more data support for splits. Increase for noisy data or small datasets.
  • subsample (or bagging_fraction): Fraction of data to use for each tree (default: 1.0). Values less than 1.0 enable bagging, which can improve generalization. Typical values are 0.7 to 0.9.
  • colsample_bytree (or feature_fraction): Fraction of features to use for each tree (default: 1.0). Reduces correlation between trees and speeds up training. Values between 0.6 and 1.0 are common.
  • reg_alpha: L1 regularization term (default: 0.0). Encourages sparsity in leaf weights. Useful for feature selection and preventing overfitting.
  • reg_lambda: L2 regularization term (default: 0.0). Smooths leaf weights to prevent overfitting. More commonly used than L1 regularization.
  • random_state: Seed for reproducibility (default: None). Set to an integer to ensure consistent results across runs.
  • verbose: Controls the verbosity of training output (default: 1). Set to -1 to suppress all output, or higher values for more detailed logging.

Key Methods

The following are the most commonly used methods for interacting with LightGBM models.

  • fit(X, y): Trains the LightGBM model on training data X and target values y. Supports optional parameters like eval_set for validation data and callbacks for early stopping.
  • predict(X): Returns predicted class labels (classification) or values (regression) for input data X. For classification, returns the class with the highest probability.
  • predict_proba(X): Returns probability estimates for each class (classification only). Useful for setting custom decision thresholds or understanding prediction confidence.
  • score(X, y): Returns the mean accuracy (classification) or R² score (regression) on the given test data. Convenient for quick model evaluation.
  • feature_importances_: Property that returns the importance of each feature based on how often they are used for splitting. Higher values indicate more important features.