CatBoost › Example

Michael BrenndoerferJuly 9, 20253 min read
Machine Learning from ScratchAll chapters
Part 5 of 9Tree-Based Models
Chapter 46CatBoost › Example
Reading section 4 of 8 · Example · 3 minSection 4 of 8 · 3 min

Book contents

Machine Learning from Scratch9 parts · 66 chaptersView the whole book →
Active chapter 46 of 66

Section 4 of 8 · 3 min read

CatBoost › ExampleOne section · Sections in this chapter4 / 8 ↓
Sections in this chapter8
Reading tools Definitions & inline links

Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.

Article links

Make inline references clickable

Let's work through a concrete example to understand how CatBoost processes categorical features and builds symmetric trees. We'll use a simplified dataset to make the calculations clear and traceable.

Dataset Setup: Consider a small dataset with 6 observations, 2 numerical features, and 1 categorical feature:

IDAgeIncomeCityTarget
12550000NYC1
23060000SF1
33570000NYC0
42855000LA1
54080000SF0
63265000LA1

Step 1: Categorical Feature Processing

CatBoost processes the categorical feature "City" by computing target statistics. For each category, we calculate:

x^City,i=∑j=1n[Cityj=Cityi]⋅Targetj+α⋅p∑j=1n[Cityj=Cityi]+α\hat{x}_{City,i} = \frac{\sum_{j=1}^{n} [City_j = City_i] \cdot Target_j + \alpha \cdot p}{\sum_{j=1}^{n} [City_j = City_i] + \alpha}

With α=1\alpha = 1 and p=46=0.667p = \frac{4}{6} = 0.667 (prior probability):

  • NYC: x^NYC=1⋅1+1⋅0+1⋅0.6672+1=1.6673=0.556\hat{x}_{NYC} = \frac{1 \cdot 1 + 1 \cdot 0 + 1 \cdot 0.667}{2 + 1} = \frac{1.667}{3} = 0.556
  • SF: x^SF=1⋅1+1⋅0+1⋅0.6672+1=1.6673=0.556\hat{x}_{SF} = \frac{1 \cdot 1 + 1 \cdot 0 + 1 \cdot 0.667}{2 + 1} = \frac{1.667}{3} = 0.556
  • LA: x^LA=1⋅1+1⋅1+1⋅0.6672+1=2.6673=0.889\hat{x}_{LA} = \frac{1 \cdot 1 + 1 \cdot 1 + 1 \cdot 0.667}{2 + 1} = \frac{2.667}{3} = 0.889

Step 2: Building a Symmetric Tree

CatBoost builds symmetric trees where all nodes at the same level use the same feature. For a tree of depth 2, we need to select 2 features for the 2 levels.

Let's say we select "Age" for level 1 and "City_encoded" for level 2. The tree structure would be:

Level 1: Age < 32.5? ├── Yes (Age < 32.5) │ └── Level 2: City_encoded < 0.722? │ ├── Yes → Leaf 1 (NYC: 0.556) │ └── No → Leaf 2 (LA: 0.889) └── No (Age ≥ 32.5) └── Level 2: City_encoded < 0.722? ├── Yes → Leaf 3 (NYC: 0.556) └── No → Leaf 4 (SF: 0.556)

Step 3: Leaf Value Calculation

For each leaf, we calculate the optimal leaf value using the gradients. The leaf values are computed to minimize the loss function, typically using the average of the gradients in each leaf.

This symmetric structure ensures that the tree is balanced and uses the same features at each level, which helps prevent overfitting and creates more interpretable models.