Statistical Modelling › Model Fit: Quantifying Prediction Quality

When we build a statistical model, our primary goal is to create a function that accurately predicts outcomes for new observations. Model fit refers to how well our model's predictions align with actual observed values. Quantifying this fit allows us to compare different models, diagnose problems, and communicate model quality to stakeholders.

For regression tasks where we predict continuous outcomes, R-squared (the coefficient of determination) provides an intuitive measure of model fit. R-squared quantifies the proportion of variance in the dependent variable that our model successfully explains. The formula captures this relationship through the ratio of explained variance to total variance:

$$R^2 = 1 - \frac{SS_{\text{res}}}{SS_{\text{tot}}} = 1 - \frac{\sum_{i=1}^n (y_i - \hat{y}_i)^2}{\sum_{i=1}^n (y_i - \bar{y})^2}$$

Where:

• $y_i$: Actual observed value for observation $i$
• $\hat{y}_i$: Model's predicted value for observation $i$
• $\bar{y}$: Mean of all actual values
• $n$: Number of observations

R-squared typically ranges from 0 to 1 for models that include an intercept, with values closer to 1 indicating better fit. Values can be negative when a model performs worse than simply predicting the mean. An R-squared of 0.85, for instance, tells us that our model explains 85% of the variance in the outcome variable, leaving only 15% unexplained. However, R-squared has important limitations. It increases when we add more predictors, even if those predictors add no real explanatory power. Additionally, a high R-squared doesn't guarantee that our model assumptions are met or that we've captured causal relationships rather than spurious correlations.

While R-squared provides a proportional measure of fit, Root Mean Squared Error (RMSE) offers an absolute measure in the same units as our target variable. RMSE calculates the standard deviation of prediction errors, giving us a concrete sense of how far off our predictions typically are:

$$\text{RMSE} = \sqrt{\frac{1}{n} \sum_{i=1}^n (y_i - \hat{y}_i)^2}$$

The appeal of RMSE lies in its interpretability. If we're predicting house prices in dollars and obtain an RMSE of $25,000, we know our typical prediction error is about$25,000. RMSE penalizes large errors more heavily than small ones due to the squaring operation, making it particularly sensitive to outliers. This sensitivity can be advantageous when large errors are especially costly, but it also means that a few extreme mispredictions can dominate the metric.

When comparing models, lower RMSE values indicate better fit. Unlike R-squared, RMSE doesn't have an upper bound and depends on the scale of the target variable, so it's most useful for comparing models predicting the same outcome rather than for assessing model quality in isolation.

The following visualization demonstrates how different levels of model fit manifest in both R-squared and RMSE metrics. We'll compare scenarios ranging from a close fit to a poor one.

One of the most fundamental challenges in statistical modeling is finding the right level of model complexity. Too simple a model fails to capture important patterns in the data, while too complex a model learns noise and spurious relationships that don't generalize. This tension between simplicity and complexity manifests as the bias-variance tradeoff, and understanding it is important for building models that perform well on new data.

Underfitting occurs when our model is too simple to capture the underlying structure in the data. An underfit model has high bias, meaning it makes strong assumptions that don't align with reality. For example, trying to fit a straight line to data that follows a curved pattern results in underfitting. The model consistently misses the mark because it lacks the flexibility to represent the true relationship.

The symptoms of underfitting are straightforward to diagnose. The model performs poorly on both training data and test data, suggesting it hasn't learned the fundamental patterns. Predictions systematically deviate from actual values in ways that reveal the model's inadequacy. In practice, underfitting often stems from using overly simple model architectures, insufficient features, or excessive regularization that constrains the model too severely.

Overfitting represents the opposite problem. An overfit model is so complex that it learns the underlying patterns and the random noise present in the training data. This model has high variance, meaning small changes in the training data lead to large changes in the fitted model. While an overfit model achieves excellent performance on training data, it fails when confronted with new observations because the patterns it learned were specific to the training set rather than general truths.

Detecting overfitting requires comparing training and test performance. The hallmark sign is a large gap between the two: near-perfect training accuracy but poor test accuracy. The model has essentially memorized the training examples rather than learning transferable patterns. Common causes of overfitting include using models with too many parameters relative to the amount of training data, training for too many iterations, or failing to apply appropriate regularization.

The goal of model building is to find the sweet spot between underfitting and overfitting where the model captures real patterns without learning noise. This optimal complexity level minimizes total error, balancing the bias from oversimplification against the variance from excessive complexity. Achieving this balance requires careful attention to model selection, regularization strategies, and validation procedures.

Practical strategies for managing this tradeoff include starting with simple models and gradually increasing complexity while monitoring validation performance. Regularization techniques like L1 and L2 penalties add constraints that prevent overfitting without requiring us to manually limit model complexity. Feature selection helps by removing irrelevant predictors that add noise. Most importantly, proper validation using held-out test data ensures we detect overfitting before deploying models to production.

This visualization demonstrates how model complexity affects fit quality. We'll fit polynomial models of increasing degree to the same data, showing how insufficient complexity leads to underfitting while excessive complexity produces overfitting.