XGBoost › Implementation using XGBoost by DMLC

Michael BrenndoerferJuly 18, 202515 min read
Machine Learning from ScratchAll chapters
Part 5 of 9Tree-Based Models
Chapter 44XGBoost › Implementation using XGBoost by DMLC
Reading section 7 of 10 · Implementation using XGBoost by DMLC · 15 minSection 7 of 10 · 15 min

Book contents

Machine Learning from Scratch9 parts · 66 chaptersView the whole book →
Active chapter 44 of 66

Section 7 of 10 · 15 min read

XGBoost › Implementation using XGBoost by DMLCOne section · Sections in this chapter7 / 10 ↓
Sections in this chapter10
Reading tools Definitions & inline links

Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.

Article links

Make inline references clickable

XGBoost can be implemented using the scikit-learn compatible XGBoost library, which provides a familiar interface while using XGBoost's advanced optimization features. Let's demonstrate this with both classification and regression examples.

Classification Example

In[14]:
Code
import numpy as np

np.random.seed(42)
In[15]:
Code
from sklearn.model_selection import cross_val_score

X_class, y_class = make_classification(
    n_samples=1000,
    n_features=20,
    n_informative=15,
    n_redundant=5,
    n_classes=3,
    random_state=42,
)

X_train_class, X_test_class, y_train_class, y_test_class = train_test_split(
    X_class, y_class, test_size=0.3, random_state=42
)

xgb_classifier = xgb.XGBClassifier(
    n_estimators=100,
    max_depth=6,
    learning_rate=0.1,
    subsample=0.8,
    colsample_bytree=0.8,
    random_state=42,
    verbosity=0,  # Suppress output
)

xgb_classifier.fit(X_train_class, y_train_class)

y_pred_class = xgb_classifier.predict(X_test_class)
y_pred_proba = xgb_classifier.predict_proba(X_test_class)

accuracy = accuracy_score(y_test_class, y_pred_class)

# Cross-validation score
cv_scores = cross_val_score(xgb_classifier, X_class, y_class, cv=5)
Out[16]:
Console
Classification Report:
Accuracy: 0.8267
Cross-validation accuracy: 0.8100 (+/- 0.0610)
              precision    recall  f1-score   support

           0       0.80      0.82      0.81        99
           1       0.86      0.83      0.85       114
           2       0.81      0.83      0.82        87

    accuracy                           0.83       300
   macro avg       0.82      0.83      0.83       300
weighted avg       0.83      0.83      0.83       300
Out[17]:
Visualization
Heatmap displaying confusion matrix with correct predictions on diagonal and misclassifications in off-diagonal cells for XGBoost classifier.
Confusion matrix showing the classification performance of XGBoost on the test set. The diagonal elements represent correct predictions, while off-diagonal elements show misclassifications between different classes.
Box plot with individual points showing distribution of XGBoost accuracy scores across 5-fold cross-validation, indicating model stability.
Cross-validation scores across 5 folds demonstrating the stability and consistency of XGBoost's performance. The box plot shows the distribution of accuracy scores, while individual points represent each fold's performance.

Regression Example

In[18]:
Code
from sklearn.datasets import make_regression
from sklearn.model_selection import cross_val_score

X_reg, y_reg = make_regression(
    n_samples=1000, n_features=20, n_informative=15, noise=0.1, random_state=42
)

X_train_reg, X_test_reg, y_train_reg, y_test_reg = train_test_split(
    X_reg, y_reg, test_size=0.3, random_state=42
)

xgb_regressor = xgb.XGBRegressor(
    n_estimators=100,
    max_depth=6,
    learning_rate=0.1,
    subsample=0.8,
    colsample_bytree=0.8,
    random_state=42,
    verbosity=0,
)

xgb_regressor.fit(X_train_reg, y_train_reg)

y_pred_reg = xgb_regressor.predict(X_test_reg)

from sklearn.metrics import mean_squared_error, r2_score

mse = mean_squared_error(y_test_reg, y_pred_reg)
r2 = r2_score(y_test_reg, y_pred_reg)

cv_scores_reg = cross_val_score(xgb_regressor, X_reg, y_reg, cv=5, scoring="r2")
Out[19]:
Console
Regression Report:
Mean Squared Error: 8607.3039
R² Score: 0.8293
Cross-validation R²: 0.8127 (+/- 0.0374)
Out[20]:
Visualization
Scatter plot of actual vs predicted values with diagonal reference line, showing how closely XGBoost regression predictions match true values.
Actual vs Predicted values scatter plot showing the regression performance of XGBoost. Points close to the diagonal line indicate accurate predictions, while deviations show prediction errors.
Residuals scatter plot against predicted values, showing error distribution around zero with no systematic patterns indicating good model fit.
Residuals plot displaying the distribution of prediction errors. A random scatter around zero indicates good model performance, while patterns suggest systematic bias or non-linear relationships not captured by the model.

Hyperparameter Tuning Example

In[21]:
Code
from sklearn.model_selection import GridSearchCV

param_grid = {  # Define parameter grid to test
    "n_estimators": [50, 100, 200],
    "max_depth": [3, 6, 9],
    "learning_rate": [0.01, 0.1, 0.2],
    "subsample": [0.8, 0.9, 1.0],
}

grid_search = GridSearchCV(
    xgb.XGBClassifier(random_state=42, verbosity=0),
    param_grid,
    cv=3,
    scoring="accuracy",
    n_jobs=-1,
)

grid_search.fit(X_train_class, y_train_class)

best_model = grid_search.best_estimator_  # Evaluate on test set
test_accuracy = best_model.score(X_test_class, y_test_class)
Out[22]:
Console
Best parameters:
{'learning_rate': 0.2, 'max_depth': 6, 'n_estimators': 200, 'subsample': 0.8}
Best cross-validation score: 0.7843
Test set accuracy with best parameters: 0.8233
Out[23]:
Console
Top 10 Hyperparameter Combinations:
================================================================================
Rank n_est  max_d  lr     sub    Mean CV    Std CV    
--------------------------------------------------------------------------------
1    200.0  6.0    0.2    0.8    0.7843     0.0169    
2    100.0  6.0    0.2    0.8    0.7814     0.0164    
3    200.0  3.0    0.2    0.9    0.7800     0.0126    
4    100.0  9.0    0.2    0.8    0.7800     0.0196    
5    50.0   6.0    0.2    0.8    0.7771     0.0129    
6    200.0  9.0    0.2    0.8    0.7771     0.0164    
7    200.0  3.0    0.1    0.8    0.7743     0.0184    
8    200.0  6.0    0.1    0.9    0.7743     0.0184    
9    200.0  3.0    0.2    1.0    0.7728     0.0124    
10   200.0  3.0    0.2    0.8    0.7714     0.0105    

Best Parameters Found:
========================================
learning_rate: 0.2
max_depth: 6
n_estimators: 200
subsample: 0.8
Best CV Score: 0.7843
Out[24]:
Visualization
Line plot showing training and validation loss curves over boosting iterations.
Training progress visualization showing how XGBoost's performance improves over iterations. The plot demonstrates the learning curves for both training and validation sets, illustrating the algorithm's ability to reduce error while maintaining good generalization performance.
Out[25]:
Visualization
Horizontal bar chart showing top 10 feature importances in XGBoost classification model.
Feature importance analysis showing the top 10 most influential features in the XGBoost classification model. The horizontal bar chart displays the relative importance of each feature, helping identify which variables contribute most to the model's predictions and providing insights for feature selection and model interpretation.

The implementation demonstrates several key aspects of using XGBoost in practice. The scikit-learn compatible interface makes it easy to integrate XGBoost into existing machine learning workflows, while the extensive hyperparameter options allow for fine-tuning performance. The feature importance analysis shows how XGBoost can help identify the most relevant features in the dataset, which is useful for both model interpretation and feature selection.

Key takeaways from this implementation:

  • XGBoost provides excellent out-of-the-box performance with minimal tuning
  • The algorithm handles both classification and regression tasks effectively
  • Hyperparameter tuning can significantly improve performance
  • Feature importance analysis shows which features influence the model's decisions

Ranking Implementation Example

XGBoost also supports learning-to-rank tasks used in applications like search engines, recommendation systems, and information retrieval. Let's demonstrate how to implement ranking with XGBoost.

In[26]:
Code
import numpy as np

np.random.seed(42)
In[27]:
Code
def generate_ranking_data(n_queries=100, n_docs_per_query=10, n_features=5):
    """Generate synthetic ranking data with queries, documents, and relevance scores."""
    documents = []
    relevance_scores = []
    query_groups = []

    for query_id in range(n_queries):
        # Generate documents for this query
        n_docs = np.random.randint(5, n_docs_per_query + 1)

        # Generate features for each document
        doc_features = np.random.randn(n_docs, n_features)

        # Generate relevance scores (0-4 scale)
        relevance = np.random.choice(
            [0, 1, 2, 3, 4], size=n_docs, p=[0.1, 0.2, 0.3, 0.3, 0.1]
        )

        # Add query-specific bias to make some queries easier to rank
        query_bias = np.random.randn(n_features) * 0.5
        doc_features += query_bias

        documents.extend(doc_features)
        relevance_scores.extend(relevance)
        query_groups.extend([n_docs])  # Group size for each query

    return (
        np.array(documents),
        np.array(relevance_scores),
        np.array(query_groups),
    )


# Generate training and test data
train_docs, train_relevance, train_groups = generate_ranking_data(80, 8, 5)
test_docs, test_relevance, test_groups = generate_ranking_data(20, 8, 5)
In[28]:
Code
print(
    f"Training data: {len(train_docs)} documents across {len(train_groups)} queries"
)
print(
    f"Test data: {len(test_docs)} documents across {len(test_groups)} queries"
)
print(f"Training group sizes: {train_groups}")
print(f"Test group sizes: {test_groups}")
In[29]:
Code
xgb_ranker = xgb.XGBRanker(
    n_estimators=100,
    max_depth=6,
    learning_rate=0.1,
    subsample=0.8,
    colsample_bytree=0.8,
    random_state=42,
    verbosity=0,
    objective="rank:pairwise",  # Use pairwise ranking objective
)

xgb_ranker.fit(
    train_docs,
    train_relevance,
    group=train_groups,  # Group by query
    eval_set=[(test_docs, test_relevance)],
    eval_group=[test_groups],
    verbose=False,
)

train_predictions = xgb_ranker.predict(train_docs)
test_predictions = xgb_ranker.predict(test_docs)
Out[31]:
Visualization
Bar chart comparing NDCG scores for training and test sets at different cutoff values.
Ranking performance visualization showing NDCG scores at different cutoff values (k=5 and k=10) for both training and test sets. The plot demonstrates XGBoost's effectiveness in learning-to-rank tasks, with consistent performance across different evaluation metrics.
Out[32]:
Visualization
Horizontal bar chart showing top 10 feature importances in XGBoost ranking model.
Feature importance analysis for the XGBoost ranking model showing the top 10 most influential features. The horizontal bar chart displays the relative importance of each feature in determining document rankings.
Out[33]:
Visualization
Vertical bar chart showing documents ranked by their actual relevance scores from highest to lowest.
Actual ranking by relevance showing the ground truth ordering of documents based on their relevance scores. Documents are sorted by their true relevance values, with the most relevant documents appearing first.
Vertical bar chart showing documents ranked by XGBoost's predicted relevance scores.
Predicted ranking by XGBoost showing the model's learned ordering of documents. The model's ranking is based on predicted scores.

Ranking Comparison for First Query Group (size: 8):
Actual ranking (by relevance): ['Doc 6', 'Doc 4', 'Doc 7', 'Doc 5', 'Doc 3', 'Doc 1', 'Doc 8', 'Doc 2']
Predicted ranking (by XGBoost): ['Doc 4', 'Doc 2', 'Doc 3', 'Doc 6', 'Doc 7', 'Doc 1', 'Doc 8', 'Doc 5']

The ranking implementation demonstrates XGBoost's capability to handle learning-to-rank problems effectively. Key aspects of this implementation include:

  • Group-based training: XGBoost uses query groups to understand which documents belong to the same query
  • Pairwise ranking objective: The model optimizes for correct pairwise document ordering
  • NDCG evaluation: Standard ranking metrics are used to assess performance
  • Feature importance: Understanding which features contribute most to ranking decisions

This makes XGBoost particularly valuable for applications like search engines, recommendation systems, and any scenario where relative ordering of items is more important than absolute scores.