Naive Bayes › Example: A Two-Feature Gaussian Calculation

The formulas are compact, but compact formulas hide their arithmetic. Before writing any code, we work one small Gaussian naive Bayes problem by hand, in full, so that every symbol in the score has a concrete number attached to it. For the Gaussian parameter estimates in this worked example, subscripts give the class first and the feature second; the general Gaussian formulas above use feature first and class second. The training data have two features and two classes, with five observations per class:

The first thing to check is whether anything needs smoothing. Each class has five observations, no feature is constant within a class, and no variance is zero, so the unsmoothed maximum likelihood estimates are safe here and we can use them directly.

The priors are equal because each class has five observations: $\pi_0 = \pi_1 = 0.5$. We want to classify the new point $\mathbf{x} = (4.5, 4.0)$, which sits in the gap between the two clusters but is closer to class 1.

We begin with the sufficient statistics. For class 0, the feature means are

$$\hat{\mu}_{0,1} = \frac{1+2+3+1+2}{5} = 1.8, \qquad \hat{\mu}_{0,2} = \frac{2+3+2+1+2}{5} = 2.0.$$

The variances use division by $n_k=5$:

$$\hat{\sigma}_{0,1}^2 = \frac{(1-1.8)^2 + (2-1.8)^2 + (3-1.8)^2 + (1-1.8)^2 + (2-1.8)^2}{5} = \frac{2.8}{5} = 0.56,$$ $$\hat{\sigma}_{0,2}^2 = \frac{(2-2)^2 + (3-2)^2 + (2-2)^2 + (1-2)^2 + (2-2)^2}{5} = \frac{2.0}{5} = 0.40.$$

For class 1, the means are

$$\begin{aligned} \hat{\mu}_{1,1} &= \frac{6+7+8+7+8}{5} = \frac{36}{5} = 7.2, \\ \hat{\mu}_{1,2} &= \frac{5+6+7+6+5}{5} = \frac{29}{5} = 5.8. \end{aligned}$$

The variances use division by $n_k=5$ in the same way:

$$\begin{aligned} \hat{\sigma}_{1,1}^2 &= \frac{(6-7.2)^2 + (7-7.2)^2 + (8-7.2)^2 + (7-7.2)^2 + (8-7.2)^2}{5} \\ &= \frac{(-1.2)^2 + (-0.2)^2 + (0.8)^2 + (-0.2)^2 + (0.8)^2}{5} \\ &= \frac{1.44 + 0.04 + 0.64 + 0.04 + 0.64}{5} \\ &= \frac{2.8}{5} = 0.56, \end{aligned}$$ $$\begin{aligned} \hat{\sigma}_{1,2}^2 &= \frac{(5-5.8)^2 + (6-5.8)^2 + (7-5.8)^2 + (6-5.8)^2 + (5-5.8)^2}{5} \\ &= \frac{(-0.8)^2 + (0.2)^2 + (1.2)^2 + (0.2)^2 + (-0.8)^2}{5} \\ &= \frac{0.64 + 0.04 + 1.44 + 0.04 + 0.64}{5} \\ &= \frac{2.8}{5} = 0.56. \end{aligned}$$

where:

• $6, 7, 8, 7, 8$: the class 1 values of feature 1 used in the mean
• $5, 6, 7, 6, 5$: the class 1 values of feature 2 used in the mean
• $36$: the sum of the class 1 values of feature 1
• $29$: the sum of the class 1 values of feature 2
• $7.2$: the class 1 mean for feature 1, $\hat{\mu}_{1,1}$
• $5.8$: the class 1 mean for feature 2, $\hat{\mu}_{1,2}$
• $2.8$: the sum of squared deviations for feature 1 and, separately, for feature 2
• $5$: the number of class 1 observations, $n_k$

With the statistics in hand, we evaluate one log density at a time. Each density has the same two-part structure: a normalizing constant that depends only on the variance, and a negative penalty that grows with the squared distance from the mean. For class 0 and feature 1, with $x_1=4.5$:

$$\log p(4.5 \mid y=0) = -\frac{1}{2}\log(2\pi \cdot 0.56) - \frac{(4.5-1.8)^2}{2 \cdot 0.56}.$$

where:

• $4.5$: the value of feature 1 for the new point $\mathbf{x}=(4.5, 4.0)$
• $1.8$: the class 0 mean for feature 1, $\hat{\mu}_{0,1}$
• $0.56$: the class 0 variance for feature 1, $\hat{\sigma}_{0,1}^2$
• $-\log(2\pi \cdot 0.56)/2$: the log-density normalizing contribution from the variance
• $(4.5-1.8)^2/(2 \cdot 0.56)$: the squared distance from the mean measured in units of the variance

The first term is $-\frac{1}{2}\log(2\pi \cdot 0.56) \approx -0.6290$. The squared distance is $(2.7)^2 = 7.29$, and the denominator is $1.12$, so the second term is $-7.29/1.12 \approx -6.5089$. The squared-distance term dominates the normalizing constant here. The new point is far from the class 0 mean, and in this calculation the distance penalty ($6.5089$) is much larger than the variance normalizing constant ($0.6290$). Thus

$$\log p(4.5 \mid y=0) \approx -0.6290 - 6.5089 = -7.1379.$$

For class 0 and feature 2, with $x_2=4.0$:

$$\log p(4.0 \mid y=0) = -\frac{1}{2}\log(2\pi \cdot 0.40) - \frac{(4.0-2.0)^2}{2 \cdot 0.40}.$$

where:

• $4.0$: the value of feature 2 for the new point
• $2.0$: the class 0 mean for feature 2, $\hat{\mu}_{0,2}$
• $0.40$: the class 0 variance for feature 2, $\hat{\sigma}_{0,2}^2$

The first term is $-\frac{1}{2}\log(2\pi \cdot 0.40) \approx -0.4608$. The squared distance is $(4.0-2.0)^2 = 4.0$, and the denominator is $2 \cdot 0.40 = 0.8$, so the second term is $-4.0/0.8 = -5.0$. Thus

$$\log p(4.0 \mid y=0) \approx -0.4608 - 5.0 = -5.4608.$$

The total log score for class 0 is the log prior plus the two log densities, evaluated before rounding:

$$s_0(\mathbf{x}) = \log(0.5) + \log p(4.5 \mid y=0) + \log p(4.0 \mid y=0) \approx -13.2919.$$

Now we repeat the calculation for class 1. For feature 1 the arithmetic is symmetric, because the distance from $4.5$ to $7.2$ is also $2.7$ and the variance is the same:

$$\log p(4.5 \mid y=1) \approx -7.1379.$$

For class 1 and feature 2, with $x_2=4.0$ and mean $5.8$:

$$\log p(4.0 \mid y=1) = -\frac{1}{2}\log(2\pi \cdot 0.56) - \frac{(4.0-5.8)^2}{2 \cdot 0.56}.$$

where:

• $5.8$: the class 1 mean for feature 2, $\hat{\mu}_{1,2}$
• $0.56$: the class 1 variance for feature 2, $\hat{\sigma}_{1,2}^2$

The squared distance is $(-1.8)^2 = 3.24$, so the second term is $-3.24/1.12 \approx -2.8929$. The first term is $-\frac{1}{2}\log(2\pi \cdot 0.56) \approx -0.6290$. Thus

$$\log p(4.0 \mid y=1) \approx -0.6290 - 2.8929 = -3.5219.$$

This is the decisive term. Feature 1 contributes identically to both classes, so the entire difference between the two scores comes from feature 2, where the new point is much closer to the class 1 mean. The total log score for class 1, again evaluated before rounding, is

$$s_1(\mathbf{x}) = \log(0.5) + \log p(4.5 \mid y=1) + \log p(4.0 \mid y=1) \approx -11.3530.$$

Class 1 has the larger score by $s_1 - s_0 = -11.3530 - (-13.2919) = 1.9389$. To convert the scores into a posterior probability, we use the logistic form for two classes:

$$p(y=1 \mid \mathbf{x}) = \frac{\exp(s_1)}{\exp(s_0)+\exp(s_1)} = \frac{1}{1+\exp(s_0-s_1)}.$$

where:

• $s_0$: the unnormalized log score for class 0, computed above as $s_0(\mathbf{x}) \approx -13.2919$
• $s_1$: the unnormalized log score for class 1, computed above as $s_1(\mathbf{x}) \approx -11.3530$
• $s_0-s_1$: the score difference, which is the only quantity that matters after normalization

Substituting $s_0-s_1 = -13.2919 - (-11.3530) = -1.9389$ gives

$$p(y=1 \mid \mathbf{x}) = \frac{1}{1+\exp(-1.9389)} \approx 0.874.$$

So we classify the point as class 1 with a predicted probability of about $87.4\%$. Notice how the log-sum-exp normalization works here in a two-class setting: the difference of scores is the only quantity that matters, and the sigmoid turns that difference into a probability without ever computing the raw likelihoods.

Within class 0, the sample correlation between the two features is about $0.42$. Within class 1, it is about $0.43$. These sample correlations are a diagnostic to investigate; five observations per class cannot establish population conditional dependence. The fitted model nevertheless multiplies the two marginal likelihoods. It favors class 1, while the calibration of its probability remains a separate question.

We can verify the arithmetic with a short computation:

The printed means, variances, and scores match the hand calculation at the displayed precision. The posterior from the printed scores comes out at approximately $0.874$, matching the earlier sigmoid calculation. The arithmetic agrees with the formulas. The sample correlations motivate checking the class-specific dependence and calibration on more data; they do not establish population dependence from five observations per class.

The printed scores come from a product of one-dimensional densities. The next figure shows those densities directly for feature 2, with the new observation marked.