Naive Bayes › Summary

Naive Bayes is a generative classifier that applies Bayes' theorem. The Gaussian model factors the feature densities conditional on the class. The multinomial model instead treats token draws as independent given the class and document length; its count components share a fixed total and are generally dependent. The decision rule compares class priors times the corresponding event-model likelihoods. Taking logs turns the product into a sum, and log-sum-exp normalization converts the scores into stable posterior probabilities.

The two event models developed here are Gaussian and multinomial. Gaussian naive Bayes estimates a mean and variance for each feature within each class, making it suitable for continuous measurements. Multinomial naive Bayes estimates class-specific word probabilities from counts, making it suitable for text and other count data. Smoothing with pseudo-counts prevents zero probabilities, and positive effective variance smoothing can prevent division by zero; globally constant training features still require explicit handling.

The independence assumption is the model's main limitation. When features are correlated, naive Bayes can double-count evidence and produce overconfident probabilities. It can still classify well, but good classification does not imply good calibration. Class imbalance can also cause the prior to dominate, so we should evaluate with class-specific metrics and consider prior adjustment or threshold tuning.

In practice, naive Bayes is a fast, interpretable baseline for high-dimensional problems, especially text classification. It is not a universal replacement for models that represent feature interactions or full covariance structure. We should use it when its assumptions and computational profile fit the problem, and we should validate its probability estimates before relying on them for decisions.