Random Forest › Example - Classification

Let's work through a detailed step-by-step example of Random Forest using a simple dataset. Suppose we want to predict whether a customer will purchase a product using their age and income together with their years of experience.

Given Data:

• Age: [25, 30, 35, 40, 45, 50, 55, 60]
• Income (thousands): [40, 50, 60, 70, 80, 90, 100, 110]
• Years of Experience: [2, 5, 8, 12, 15, 18, 20, 22]
• Purchase (1) or No Purchase (0): [0, 0, 1, 1, 1, 1, 1, 1]

Step 1: Create Bootstrap Samples

Random Forest creates multiple bootstrap samples from the original data. Let's create 3 bootstrap samples for illustration:

Bootstrap Sample 1: [1, 2, 3, 4, 5, 6, 7, 8] → [1, 1, 2, 3, 4, 5, 6, 7]

Bootstrap Sample 2: [1, 2, 3, 4, 5, 6, 7, 8] → [2, 3, 4, 4, 5, 6, 7, 8]

Bootstrap Sample 3: [1, 2, 3, 4, 5, 6, 7, 8] → [1, 2, 3, 5, 6, 7, 8, 8]

Note: In practice, bootstrap samples are created by sampling with replacement, so some observations may appear multiple times while others may not appear at all.

Step 2: Train Individual Trees

Each tree is trained on its bootstrap sample using random feature selection. At each split, only a random subset of features is considered.

Tree 1 (trained on Bootstrap Sample 1):

• Root split: Age ≤ 35 (using Age and Income features)
• Left child: Age ≤ 35 → No Purchase (0)
• Right child: Age > 35 → Purchase (1)

Tree 2 (trained on Bootstrap Sample 2):

• Root split: Income ≤ 75 (using Income and Experience features)
• Left child: Income ≤ 75 → No Purchase (0)
• Right child: Income > 75 → Purchase (1)

Tree 3 (trained on Bootstrap Sample 3):

• Root split: Experience ≤ 10 (using Age and Experience features)
• Left child: Experience ≤ 10 → No Purchase (0)
• Right child: Experience > 10 → Purchase (1)

Step 3: Make Predictions

For a new customer with Age=42, Income=85, Experience=14:

Tree 1 Prediction: Age=42 > 35 → Right child → Purchase (1)

Tree 2 Prediction: Income=85 > 75 → Right child → Purchase (1)

Tree 3 Prediction: Experience=14 > 10 → Right child → Purchase (1)

Step 4: Combine Predictions

Random Forest combines the predictions from all trees:

• Tree 1: Purchase (1)
• Tree 2: Purchase (1)
• Tree 3: Purchase (1)

Final Prediction: Majority vote = Purchase (1)

Step 5: Calculate Probabilities

The probability of purchase is the proportion of trees that predicted purchase:

$$P(\text{Purchase}) = \frac{3}{3} = 1.0$$

Step 6: Feature Importance

Random Forest calculates feature importance based on how much each feature contributes to reducing impurity across all trees:

• Age: Used in Tree 1 and Tree 3, contributes to 2/3 of the splits
• Income: Used in Tree 2, contributes to 1/3 of the splits
• Experience: Used in Tree 3, contributes to 1/3 of the splits

Normalized Feature Importance:

• Age: 2/3 ≈ 0.67
• Income: 1/3 ≈ 0.33
• Experience: 1/3 ≈ 0.33

This example demonstrates how Random Forest combines multiple decision trees to make more reliable predictions. Each tree learns different patterns from the data, and the ensemble approach reduces the risk of overfitting while maintaining good predictive performance.