CART (Classification and Regression Trees) › Implementation in Scikit-learn
Machine Learning from ScratchAll chapters CART (Classification and Regression Trees) › Implementation in Scikit-learnOne section · Sections in this chapter6 / 9 ↓
Reading tools Definitions & inline links
Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.
Article links
Make inline references clickable
Scikit-learn implements CART through the DecisionTreeClassifier and DecisionTreeRegressor classes. Let's walk through training and evaluating decision trees for both classification and regression tasks.
Data Preparation
We'll start by importing the necessary libraries and creating synthetic datasets for both classification and regression examples.
from sklearn.datasets import make_classification, make_regression
from sklearn.model_selection import train_test_split
# Create classification dataset with 1000 samples and 4 features
X_class, y_class = make_classification(
n_samples=1000,
n_features=4,
n_redundant=0,
n_informative=4,
n_clusters_per_class=1,
random_state=42,
)
# Create regression dataset with 1000 samples and 4 features
X_reg, y_reg = make_regression(
n_samples=1000, n_features=4, noise=10, random_state=42
)
# Split both datasets into training and testing sets
X_class_train, X_class_test, y_class_train, y_class_test = train_test_split(
X_class, y_class, test_size=0.2, random_state=42
)
X_reg_train, X_reg_test, y_reg_train, y_reg_test = train_test_split(
X_reg, y_reg, test_size=0.2, random_state=42
)Classification Implementation
Now let's build a decision tree classifier. We'll configure several key parameters to control the tree's complexity and prevent overfitting.
from sklearn.metrics import accuracy_score
from sklearn.model_selection import cross_val_score
clf = DecisionTreeClassifier(
max_depth=5,
min_samples_split=20,
min_samples_leaf=10,
max_features="sqrt",
random_state=42,
)
# Train the classifier on the training data
clf.fit(X_class_train, y_class_train)
# Make predictions on the test set
y_class_pred = clf.predict(X_class_test)
y_class_pred_proba = clf.predict_proba(X_class_test)
# Calculate accuracy on the test set
accuracy = accuracy_score(y_class_test, y_class_pred)
# Perform 5-fold cross-validation on the training set
cv_scores = cross_val_score(clf, X_class_train, y_class_train, cv=5)Let's examine the classification performance:
Test Accuracy: 0.9250 Cross-Validation Scores: [0.88125 0.90625 0.9 0.93125 0.89375] Mean CV Score: 0.9025 (+/- 0.0332)
The test accuracy shows how well the model performs on unseen data. The cross-validation scores provide a more robust estimate of model performance by averaging results across multiple train-test splits. The standard deviation indicates the stability of the model's performance.lower values suggest more consistent predictions across different data subsets.
Regression Implementation
For regression tasks, we use DecisionTreeRegressor with similar parameters but adjusted for the continuous target variable.
from sklearn.metrics import mean_squared_error
reg = DecisionTreeRegressor(
max_depth=6,
min_samples_split=15,
min_samples_leaf=8,
max_features="sqrt",
random_state=42,
)
# Train the regressor on the training data
reg.fit(X_reg_train, y_reg_train)
# Make predictions on the test set
y_reg_pred = reg.predict(X_reg_test)
# Calculate root mean squared error
mse = mean_squared_error(y_reg_test, y_reg_pred)
rmse = np.sqrt(mse)
# Perform 5-fold cross-validation
cv_scores_reg = cross_val_score(
reg, X_reg_train, y_reg_train, cv=5, scoring="neg_mean_squared_error"
)
cv_rmse = np.sqrt(-cv_scores_reg)Let's examine the regression performance:
Test RMSE: 54.3534 Cross-Validation RMSE: [64.15932352 51.64740602 52.72624029 47.96812047 52.09732228] Mean CV RMSE: 53.7197 (+/- 10.9537)
The RMSE (Root Mean Squared Error) measures the average prediction error in the same units as the target variable. Lower RMSE values indicate better predictive accuracy. The cross-validation RMSE provides a more reliable estimate of how the model will perform on new data, with the standard deviation showing the consistency of predictions across different data splits.
Feature Importance Analysis
Decision trees automatically calculate feature importance based on how much each feature reduces impurity across all splits.
feature_importance = clf.feature_importances_Feature Importance Scores: Feature 0: 0.7161 Feature 1: 0.0766 Feature 2: 0.0419 Feature 3: 0.1653
These importance scores indicate which features contribute most to the model's predictions. Features with higher scores have a greater influence on the decision-making process. In this example, the scores show how the tree prioritizes different features when making splits.
Tree Visualization
Visualizing the tree structure helps understand the decision-making process:
use_mls_style()
fig, ax = plt.subplots(figsize=(14, 8))
detailed_tree_artists = plot_tree(
clf,
feature_names=[f"Feature_{i}" for i in range(X_class.shape[1])],
class_names=["Class_0", "Class_1"],
filled=True,
rounded=True,
fontsize=8,
ax=ax,
)
style_tree_artists(detailed_tree_artists)
ax.set_title("Decision Tree Structure")
fig.subplots_adjust(left=0.01, right=0.99, bottom=0.02, top=0.93)
plt.show()
Note: This visualization is intended for illustrative purposes. It's normal if not all feature details are visible in the tree plot.
The tree visualization shows the complete decision-making hierarchy. Each node displays the splitting criterion (feature and threshold), the Gini impurity, the number of samples, and the class distribution. The color intensity indicates node purity.darker colors represent more homogeneous nodes. This visual representation makes it easy to trace any prediction back through the specific sequence of decisions that led to it.
Key Parameters
Below are the main parameters that control how CART decision trees work and perform.
-
max_depth: Maximum depth of the tree (default: None). Limiting depth prevents overfitting by restricting tree complexity. Start with values like 5-10 and adjust based on cross-validation performance. -
min_samples_split: Minimum samples required to split an internal node (default: 2). Higher values prevent the tree from creating splits on small subsets, reducing overfitting. Values of 10-20 work well for most datasets. -
min_samples_leaf: Minimum samples required in a leaf node (default: 1). This ensures leaves have sufficient data for reliable predictions. Values of 5-10 help prevent overfitting while maintaining predictive power. -
max_features: Number of features to consider for the best split (default: None for all features). Options include'sqrt'(square root of total features),'log2'(log base 2 of total features), or an integer. Using fewer features can reduce overfitting and speed up training. -
criterion: Function to measure split quality. For classification:'gini'(default) or'entropy'. For regression:'squared_error'(default) or'absolute_error'. Gini and squared_error are computationally faster and work well in most cases. -
random_state: Seed for reproducibility (default: None). Set to an integer to ensure consistent results across runs, especially important when usingmax_featureswith randomization. -
min_impurity_decrease: Minimum impurity decrease required for a split (default: 0.0). A node will split only if the decrease in impurity is greater than or equal to this value. Useful for pruning weak splits. -
ccp_alpha: Complexity parameter for minimal cost-complexity pruning (default: 0.0). Higher values result in more aggressive pruning, creating simpler trees. Use cross-validation to find the optimal value.
Key Methods
The following are the most commonly used methods for interacting with CART decision tree models.
-
fit(X, y): Trains the decision tree on the training data X and target values y. The tree is built by recursively finding the best splits according to the specified criterion. -
predict(X): Returns predicted class labels (classification) or values (regression) for input data X. For classification, returns the majority class in the leaf node. For regression, returns the mean value of samples in the leaf node. -
predict_proba(X): Returns probability estimates for each class (classification only). Probabilities are calculated as the fraction of training samples of each class in the leaf node. Useful for setting custom decision thresholds or understanding prediction confidence. -
score(X, y): Returns the mean accuracy (classification) or R² score (regression) on the given test data. Provides a quick way to evaluate model performance. -
get_depth(): Returns the depth of the decision tree. Useful for understanding tree complexity and diagnosing potential overfitting. -
get_n_leaves(): Returns the number of leaves in the decision tree. More leaves indicate a more complex model that may be overfitting. -
apply(X): Returns the index of the leaf that each sample ends up in. Useful for understanding which samples follow similar decision paths or for creating custom features based on tree structure.