Random Forest › Implementation in Scikit-learn
Michael BrenndoerferJune 30, 20256 min read
Machine Learning from ScratchAll chapters Part 5 of 9Tree-Based Models
Random Forest › Implementation in Scikit-learnOne section · Sections in this chapter6 / 9 ↓
Sections in this chapter9
Reading tools Definitions & inline links
Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.
Article links
Make inline references clickable
Random Forest is implemented in scikit-learn through the RandomForestClassifier and RandomForestRegressor classes. Let's implement a complete example with proper preprocessing and evaluation:
In[8]:
Codeimport numpy as np
from sklearn.ensemble import RandomForestClassifier
from sklearn.model_selection import cross_val_score, train_test_split
np.random.seed(42)
n_samples = 1000
age = np.random.normal(40, 15, n_samples) # Generate features
age = np.clip(age, 18, 80) # Clip to realistic range
income = np.random.normal(60, 20, n_samples)
income = np.clip(income, 20, 150) # Clip to realistic range
experience = np.random.normal(10, 8, n_samples)
experience = np.clip(experience, 0, 30) # Clip to realistic range
# Create target based on realistic relationship
# Higher age, income, and experience increase probability of purchase
log_odds = -5 + 0.02 * age + 0.01 * income + 0.05 * experience
prob_purchase = 1 / (1 + np.exp(-log_odds))
y_class = np.random.binomial(1, prob_purchase, n_samples)
# Combine features
X_class = np.column_stack([age, income, experience])
X_train_class, X_test_class, y_train_class, y_test_class = train_test_split(
X_class, y_class, test_size=0.2, random_state=42, stratify=y_class
)
rf_classifier = RandomForestClassifier(
n_estimators=100,
max_depth=10,
min_samples_split=5,
min_samples_leaf=2,
random_state=42,
n_jobs=-1, # Use all available cores
)
rf_classifier.fit(X_train_class, y_train_class)
y_pred_class = rf_classifier.predict(X_test_class)
y_pred_proba_class = rf_classifier.predict_proba(X_test_class)[:, 1]
cv_scores = cross_val_score(
rf_classifier, X_train_class, y_train_class, cv=5, scoring="roc_auc"
)Out[9]:
ConsoleClassification Performance:
Accuracy: 0.950
ROC AUC: 0.665
Feature Importance:
Age: 0.342
Income: 0.363
Experience: 0.295
Sample Predictions:
Age: 21.8, Income: 57.0, Exp: 2.7 -> Prob: 0.002, Actual: 0
Age: 39.8, Income: 90.6, Exp: 7.6 -> Prob: 0.030, Actual: 0
Age: 39.6, Income: 64.3, Exp: 8.7 -> Prob: 0.089, Actual: 1
Age: 35.4, Income: 28.3, Exp: 0.0 -> Prob: 0.000, Actual: 0
Age: 52.4, Income: 87.8, Exp: 11.8 -> Prob: 0.111, Actual: 0
Classification Report:
precision recall f1-score support
0 0.95 1.00 0.97 190
1 0.00 0.00 0.00 10
accuracy 0.95 200
macro avg 0.47 0.50 0.49 200
weighted avg 0.90 0.95 0.93 200
Confusion Matrix:
[[190 0]
[ 10 0]]
Cross-validation ROC AUC scores:
[0.60855263 0.51809211 0.74177632 0.46875 0.67763158]
Mean CV ROC AUC: 0.603 (+/- 0.200)Now let's implement a regression example:
In[10]:
Codenp.random.seed(42)
house_size = np.random.normal(2000, 500, n_samples) # Generate features
house_size = np.clip(house_size, 800, 4000) # Clip to realistic range
bedrooms = np.random.poisson(3, n_samples)
bedrooms = np.clip(bedrooms, 1, 6) # Clip to realistic range
bathrooms = np.random.poisson(2, n_samples)
bathrooms = np.clip(bathrooms, 1, 4) # Clip to realistic range
# Create target based on realistic relationship
# House price depends on size, bedrooms, and bathrooms
house_price = (
50000
+ 100 * house_size
+ 20000 * bedrooms
+ 15000 * bathrooms
+ np.random.normal(0, 10000, n_samples)
)
house_price = np.clip(house_price, 100000, 800000) # Clip to realistic range
X_reg = np.column_stack([house_size, bedrooms, bathrooms]) # Combine features
X_train_reg, X_test_reg, y_train_reg, y_test_reg = train_test_split(
X_reg, house_price, test_size=0.2, random_state=42
)
rf_regressor = RandomForestRegressor(
n_estimators=100,
max_depth=10,
min_samples_split=5,
min_samples_leaf=2,
random_state=42,
n_jobs=-1, # Use all available cores
)
rf_regressor.fit(X_train_reg, y_train_reg)
y_pred_reg = rf_regressor.predict(X_test_reg)Out[11]:
ConsoleRegression Performance: R² Score: 0.949 RMSE: 13565.25 Feature Importance: House Size: 0.713 Bedrooms: 0.232 Bathrooms: 0.055 Sample Predictions: Size: 2272, Beds: 3, Baths: 1 -> Pred: $357774, Actual: $339322 Size: 2491, Beds: 3, Baths: 3 -> Pred: $409143, Actual: $393120 Size: 1080, Beds: 4, Baths: 2 -> Pred: $277398, Actual: $260181 Size: 1713, Beds: 1, Baths: 4 -> Pred: $292142, Actual: $314224 Size: 1438, Beds: 2, Baths: 1 -> Pred: $253533, Actual: $247561
Hyperparameter Tuning
Random Forest has several important hyperparameters that can significantly affect performance:
In[12]:
Codeparam_grid = {
"n_estimators": [50, 100, 200],
"max_depth": [5, 10, 15, None],
"min_samples_split": [2, 5, 10],
"min_samples_leaf": [1, 2, 4],
}
# Use a smaller subset for faster tuning
X_train_small = X_train_class[:500]
y_train_small = y_train_class[:500]
grid_search = GridSearchCV(
RandomForestClassifier(random_state=42, n_jobs=-1),
param_grid,
cv=3, # 3-fold CV for faster tuning
scoring="roc_auc",
n_jobs=-1,
)
grid_search.fit(X_train_small, y_train_small)Out[13]:
ConsoleBest parameters:
{'max_depth': 5, 'min_samples_leaf': 4, 'min_samples_split': 10, 'n_estimators': 50}
Best cross-validation score: 0.691
Test set ROC AUC: 0.604Key Parameters
Below are some of the main parameters that affect how Random Forest works and performs.
n_estimators: Number of trees in the forest (default: 100). More trees generally improve performance but increase computation time. Start with 100 and increase if cross-validation shows improvement. Values between 100-500 work well for most datasets.max_depth: Maximum depth of each tree (default: None, meaning nodes expand until all leaves are pure or contain fewer thanmin_samples_splitsamples). Limiting depth helps prevent overfitting. Try values like 10, 15, or 20 for complex datasets.min_samples_split: Minimum number of samples required to split an internal node (default: 2). Higher values prevent overfitting by requiring more data for splits. Values of 5-10 work well for most datasets.min_samples_leaf: Minimum number of samples required in a leaf node (default: 1). Higher values smooth the model and prevent overfitting. Values of 2-4 are commonly used.max_features: Number of features to consider when looking for the best split (default: 'sqrt' for classification, 1.0 for regression). Controls the diversity of trees in the forest. Use 'sqrt' for classification and 'log2' or p/3 for regression.bootstrap: Whether to use bootstrap samples when building trees (default: True). Setting to False uses the entire dataset for each tree, which reduces diversity but may be useful in some cases.oob_score: Whether to use out-of-bag samples to estimate generalization performance (default: False). Set to True to get an unbiased estimate without needing a separate validation set.random_state: Seed for reproducibility (default: None). Set to an integer to ensure consistent results across runs.n_jobs: Number of CPU cores to use for parallel training (default: None, meaning 1 core). Set to -1 to use all available cores for faster training.
Key Methods
The following are the most commonly used methods for interacting with Random Forest models.
fit(X, y): Trains the Random Forest on the training data X and target values y. Each tree is trained on a bootstrap sample of the data.predict(X): Returns predicted class labels (classification) or values (regression) for input data X by aggregating predictions from all trees.predict_proba(X): Returns probability estimates for each class (classification only). Useful for setting custom decision thresholds or understanding prediction confidence.score(X, y): Returns the mean accuracy (classification) or R² score (regression) on the given test data and labels.feature_importances_: Attribute (not a method) that provides the importance of each feature, calculated as the mean decrease in impurity across all trees.