Random Forest › Example - Regression

Now let's work through a regression example where we want to predict house prices based on house size, number of bedrooms, and number of bathrooms.

Given Data:

• House Size (sq ft): [1200, 1500, 1800, 2000, 2200, 2500, 2800, 3000]
• Bedrooms: [2, 3, 3, 4, 4, 4, 5, 5]
• Bathrooms: [1, 2, 2, 2, 3, 3, 3, 4]
• House Price (thousands): [180, 220, 260, 300, 340, 380, 420, 460]

Step 1: Create Bootstrap Samples

For regression, we create bootstrap samples just like in classification:

Bootstrap Sample 1: [1, 2, 3, 4, 5, 6, 7, 8] → [1, 1, 2, 3, 4, 5, 6, 7]

Bootstrap Sample 2: [1, 2, 3, 4, 5, 6, 7, 8] → [2, 3, 4, 4, 5, 6, 7, 8]

Bootstrap Sample 3: [1, 2, 3, 4, 5, 6, 7, 8] → [1, 2, 3, 5, 6, 7, 8, 8]

Step 2: Train Individual Trees

Each tree is trained on its bootstrap sample using random feature selection and regression splitting criteria (typically mean squared error).

Tree 1 (trained on Bootstrap Sample 1):

• Root split: House Size ≤ 2000 (using House Size and Bedrooms features)
• Left child: House Size ≤ 2000 → Average Price = 240
• Right child: House Size > 2000 → Average Price = 400

Tree 2 (trained on Bootstrap Sample 2):

• Root split: Bedrooms ≤ 3 (using Bedrooms and Bathrooms features)
• Left child: Bedrooms ≤ 3 → Average Price = 220
• Right child: Bedrooms > 3 → Average Price = 380

Tree 3 (trained on Bootstrap Sample 3):

• Root split: Bathrooms ≤ 2 (using House Size and Bathrooms features)
• Left child: Bathrooms ≤ 2 → Average Price = 250
• Right child: Bathrooms > 2 → Average Price = 380

Step 3: Make Predictions

For a new house with Size=2100, Bedrooms=4, Bathrooms=3:

Tree 1 Prediction: Size=2100 > 2000 → Right child → Price = 400

Tree 2 Prediction: Bedrooms=4 > 3 → Right child → Price = 380

Tree 3 Prediction: Bathrooms=3 > 2 → Right child → Price = 380

Step 4: Combine Predictions

Random Forest combines the predictions from all trees by averaging:

• Tree 1: 400
• Tree 2: 380
• Tree 3: 380

Final Prediction: Average = (400 + 380 + 380) / 3 = 386.67

Step 5: Calculate Prediction Variance

The variance of predictions across trees provides an estimate of prediction uncertainty:

$$\text{Variance} = \frac{1}{B-1} \sum_{b=1}^{B} (\hat{y}_b - \bar{\hat{y}})^2$$

where:

• $\text{Variance}$: Sample variance of predictions across all trees
• $B$: Total number of trees in the Random Forest (3 in this example)
• $\hat{y}_b$: Prediction from the $b$-th tree
• $\bar{\hat{y}}$: Mean prediction across all trees, calculated as $\bar{\hat{y}} = \frac{1}{B} \sum_{b=1}^{B} \hat{y}_b$

For our example:

$$\text{Variance} = \frac{1}{2} [(400-386.67)^2 + (380-386.67)^2 + (380-386.67)^2]$$ $$= \frac{1}{2} [177.78 + 44.44 + 44.44] = 133.33$$

Step 6: Feature Importance

Random Forest calculates feature importance based on how much each feature contributes to reducing mean squared error across all trees:

• House Size: Used in Tree 1 and Tree 3, contributes to 2/3 of the splits
• Bedrooms: Used in Tree 2, contributes to 1/3 of the splits
• Bathrooms: Used in Tree 2 and Tree 3, contributes to 2/3 of the splits

Normalized Feature Importance:

• House Size: 2/3 ≈ 0.67
• Bedrooms: 1/3 ≈ 0.33
• Bathrooms: 2/3 ≈ 0.67

This regression example demonstrates how Random Forest provides both point predictions and uncertainty estimates through the variance of individual tree predictions. The ensemble approach helps create more stable and reliable predictions compared to individual decision trees.