CatBoost › Implementation in CatBoost
Machine Learning from ScratchAll chapters CatBoost › Implementation in CatBoostOne section · Sections in this chapter5 / 8 ↓
Reading tools Definitions & inline links
Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.
Article links
Make inline references clickable
CatBoost provides a clean, scikit-learn compatible interface that makes it easy to implement in machine learning pipelines. The framework handles categorical features automatically and provides excellent default parameters that work well across many datasets.
Let's walk through a complete example demonstrating CatBoost's key features, including automatic categorical feature handling, model training, and evaluation.
import numpy as np
import pandas as pd
from catboost import CatBoostClassifier, Pool
from sklearn.datasets import make_classification
from sklearn.model_selection import cross_val_score, train_test_split
# Create a dataset with mixed data types
np.random.seed(42)
X, y = make_classification(
n_samples=1000,
n_features=8,
n_informative=6,
n_redundant=2,
n_classes=2,
random_state=42,
)
# Convert some features to categorical using quantile-based binning
# This ensures consistent categories across train/test splits
X_cat = X.copy()
X_cat[:, 0] = pd.qcut(
X_cat[:, 0], q=4, labels=False, duplicates="drop"
) # 4 categories
X_cat[:, 1] = pd.qcut(
X_cat[:, 1], q=3, labels=False, duplicates="drop"
) # 3 categories
# Create DataFrame with proper data types
df = pd.DataFrame(X_cat, columns=[f"feature_{i}" for i in range(8)])
# Convert to string type for CatBoost compatibility
df["feature_0"] = df["feature_0"].astype(str)
df["feature_1"] = df["feature_1"].astype(str)
df["target"] = y
# Split the data
X_train, X_test, y_train, y_test = train_test_split(
df.drop("target", axis=1), df["target"], test_size=0.3, random_state=42
)
# Identify categorical features
categorical_features = ["feature_0", "feature_1"]
categorical_indices = [df.columns.get_loc(col) for col in categorical_features]
# Create CatBoost Pool for efficient data handling
train_pool = Pool(X_train, y_train, cat_features=categorical_indices)
test_pool = Pool(X_test, y_test, cat_features=categorical_indices)
# Initialize CatBoost with optimized parameters
catboost_model = CatBoostClassifier(
iterations=100, # Number of boosting iterations
learning_rate=0.1, # Learning rate
depth=6, # Tree depth
l2_leaf_reg=3, # L2 regularization
random_seed=42, # For reproducibility
verbose=False, # Suppress training output
)
# Train the model
catboost_model.fit(train_pool)
# Make predictions
y_pred = catboost_model.predict(test_pool)
y_pred_proba = catboost_model.predict_proba(test_pool)[:, 1]
cv_scores = cross_val_score(
catboost_model, X_train, y_train, cv=5, scoring="roc_auc"
)
feature_importance = catboost_model.get_feature_importance()
feature_names = [f"feature_{i}" for i in range(8)]CatBoost Classification Report:
precision recall f1-score support
0 0.91 0.87 0.89 147
1 0.88 0.92 0.90 153
accuracy 0.90 300
macro avg 0.90 0.90 0.90 300
weighted avg 0.90 0.90 0.90 300
Cross-validation ROC-AUC scores: [0.95448051 0.9667279 0.96387755 0.98244898 0.96714286]
Mean CV ROC-AUC: 0.9669 (+/- 0.0180)The classification report shows strong performance across both classes, with precision, recall, and F1-scores all above 0.85. The cross-validation ROC-AUC scores demonstrate model stability across different data splits, with a mean score around 0.93 and low standard deviation. This indicates that CatBoost has successfully learned meaningful patterns from the mixed categorical and numerical features without requiring extensive preprocessing.
Key Implementation Notes:
-
Categorical Feature Handling: CatBoost automatically processes categorical features without requiring preprocessing. We simply specify which features are categorical using the
cat_featuresparameter. -
Pool Data Structure: CatBoost's
Poolclass provides efficient data handling and preprocessing, particularly beneficial for large datasets with mixed data types. -
Default Parameters: CatBoost's default parameters work well across many datasets, reducing the need for extensive hyperparameter tuning. The framework includes intelligent defaults for regularization and learning rate.
-
Feature Importance: CatBoost provides multiple types of feature importance, including prediction value change, loss function change, and feature interaction strength, helping with model interpretation.
Key Parameters
Below are some of the main parameters that affect how CatBoost works and performs.
-
iterations: Number of boosting iterations (trees to build). Default is 1000. More iterations generally improve performance but increase training time. Start with 100-500 for initial experiments, then increase if validation scores continue to improve. -
learning_rate: Step size for updating the model at each iteration. Default is 0.03. Lower values (0.01-0.1) require more iterations but often lead to better generalization. Higher values speed up training but may cause overfitting. -
depth: Maximum depth of each tree. Default is 6. Deeper trees (8-10) can capture more complex patterns but risk overfitting. Shallower trees (4-6) are more conservative and generalize better on smaller datasets. -
l2_leaf_reg: L2 regularization coefficient for leaf values. Default is 3. Higher values (5-10) provide stronger regularization and reduce overfitting. Lower values (1-3) allow the model to fit training data more closely. -
cat_features: List or array of categorical feature indices or names. CatBoost will automatically apply target encoding to these features. Can be specified as column indices (integers) or column names (strings). -
random_seed: Seed for reproducibility. Default is None. Set to an integer to ensure consistent results across runs, which is important for debugging and comparing models. -
verbose: Controls training output verbosity. Set toFalseto suppress output, or an integer to print progress every N iterations. Useful for monitoring training progress on large datasets. -
loss_function: Loss function to optimize. Default is 'Logloss' for binary classification, 'MultiClass' for multi-class, and 'RMSE' for regression. Other options include 'CrossEntropy', 'MAE', and custom loss functions. -
eval_metric: Metric used for overfitting detection and best model selection. Common choices include 'AUC', 'Accuracy', 'F1', 'Precision', 'Recall' for classification, and 'RMSE', 'MAE', 'R2' for regression. -
early_stopping_rounds: Number of iterations without improvement before stopping training. Default is None (no early stopping). Setting to 20-50 helps prevent overfitting and saves training time.
Key Methods
The following are the most commonly used methods for interacting with CatBoost models.
-
fit(X, y, cat_features=None, eval_set=None): Trains the CatBoost model on training data X and target values y. Thecat_featuresparameter specifies categorical features, andeval_setprovides validation data for monitoring. -
predict(X): Returns predicted class labels (classification) or values (regression) for input data X. For classification, returns the class with highest probability. -
predict_proba(X): Returns probability estimates for each class (classification only). Useful for setting custom decision thresholds or when you need confidence scores rather than hard predictions. -
get_feature_importance(): Returns feature importance scores showing the relative contribution of each feature to the model's predictions. Higher values indicate more influential features. -
score(X, y): Returns the mean accuracy (classification) or R² score (regression) on the given test data. Convenient for quick model evaluation.