Naive Bayes

Two measurements can carry nearly the same information. A classifier that multiplies their class-conditional likelihoods can then count that information twice: the predicted class may stay useful while its reported probability becomes too extreme. Naive Bayes makes this tradeoff explicit. It models each class with simple feature distributions and combines them using a conditional-independence assumption. For text counts, the corresponding multinomial model instead assumes independent token draws given the class and document length.

We will fit both Gaussian measurements and multinomial word counts, compare the primitive calculations with scikit-learn, and inspect a seeded example with nearly duplicated measurements. Training uses class counts and feature summaries; prediction adds one contribution per feature or observed token count. These simple operations make it possible to trace a prediction back to its inputs.

The model is generative rather than discriminative. Logistic regression, which we studied in Logistic Regression, models $p(y \mid \mathbf{x})$ directly. Naive Bayes models $p(\mathbf{x} \mid y)$ and $p(y)$, then combines them with Bayes' theorem. This difference matters: a generative model can marginalize unobserved features under an appropriate observation model, but neither class implemented here silently accepts missing entries. Its probability estimates still depend on its distributional assumptions.

This chapter derives the Bayes decision rule, explains the conditional independence assumption, develops the Gaussian and multinomial event models, shows how smoothing and log-sum-exp keep computations stable, and examines what happens when independence fails. We will implement Gaussian and multinomial naive Bayes from NumPy primitives, compare them with scikit-learn, and study calibration and class imbalance.