GPTQ › Perplexity Comparison

The ultimate test of quantization quality is model performance on downstream tasks. For language models, perplexity on held-out text gives a principled proxy: it measures how well the model predicts unseen text, capturing both fluency and factual accuracy in a single number. Lower perplexity means better predictions.

Perplexity is defined as the exponentiated average negative log-likelihood per token:

$$\text{PPL} = \exp\left(-\frac{1}{N} \sum_{i=1}^{N} \log P(x_i \mid x_{<i})\right)$$

where:

• $\text{PPL}$: perplexity, where lower values indicate better language modeling
• $N$: the total number of tokens in the evaluation set
• $x_i$: the $i$-th token in the sequence
• $P(x_i \mid x_{<i})$: the model's predicted probability for token $x_i$ given all preceding tokens
• $\exp(\cdot)$: the exponential function that converts average log-probability into a perplexity score

A perplexity of 5 means the model is, on average, as uncertain as if it had to choose uniformly among 5 equally likely options for each token. A perplexity of 4.5 instead of 5.0 is a real improvement in language understanding. The following table shows typical GPTQ performance for LLaMA models evaluated on WikiText-2:

The perplexity degradation shrinks with model size, and this is not a coincidence. Larger models have more parameters, which means more redundancy in the weight space. More redundancy means that the compensation mechanism has more "slack" to work with: when one weight is rounded away from its optimal value, there are more other weights available to absorb the error. At 65B parameters, the difference between FP16 and GPTQ 4-bit is barely measurable on most benchmarks. The effective information density of a quantized 65B model exceeds that of a full-precision 7B model, making GPTQ quantization a worthwhile trade at every scale.

The relationship between model size and quantization robustness has an important practical implication: if you must choose between a smaller full-precision model and a larger quantized model of similar memory size, the larger quantized model usually wins on accuracy. A GPTQ 4-bit LLaMA-13B fits in roughly the same GPU memory as an FP16 LLaMA-3B, but performs substantially better on almost every benchmark.