GGUF Format › Quantization Types

GGUF supports an extensive array of quantization schemes. Understanding the naming conventions and trade-offs helps you choose the right model variant for your hardware constraints. GGUF quantization has evolved considerably since the format's introduction, with newer methods achieving better quality at the same bit widths through more sophisticated compression strategies. Tracing these advances in lossy weight compression reveals what makes quantization work well.

Think of the quantization types as a dial ranging from maximum compression to maximum quality. At one extreme you have IQ1 methods that store roughly one bit per weight, achieving extraordinary file sizes but at considerable quality cost. At the other extreme you have FP16 or Q8_0, which preserve nearly all of the original model's capabilities but require substantially more memory. Most users choose an operating point in the middle based on their hardware and quality requirements.

Quantization types in GGUF follow a naming pattern: Q{bits}_{variant}. The bits indicate the average bits per weight, and the variant specifies the quantization method. This naming convention makes it straightforward to compare options and understand roughly what to expect from each format in terms of file size and quality.

The basic quantization types represent the first generation of GGUF quantization. They use a simple scheme where a block of 32 weights shares a single scale factor (and optionally a minimum value), and each weight is mapped to a small integer relative to that scale:

• Q4_0: 4-bit quantization with a single scale factor per block of 32 weights. The simplest and most portable format, supported by essentially all GGUF implementations.
• Q4_1: 4-bit with both scale and minimum values per block, allowing better representation of asymmetric distributions.
• Q5_0: 5-bit quantization, storing 4 bits directly plus 1 extra bit per weight in a separate array, giving finer granularity than Q4 without jumping all the way to Q8.
• Q5_1: 5-bit with scale and minimum, the asymmetric variant of Q5_0.
• Q8_0: 8-bit quantization. This provides near-lossless quality at the cost of roughly twice the file size of Q4. Often used as an intermediate format during model conversion.

K-quant methods, introduced in mid-2023, brought significant quality improvements over the basic types. They use different block sizes and more sophisticated quantization schemes, and the "K" designation indicates that these methods employ a key innovation: they use super-blocks that group multiple basic blocks together with shared higher-level statistics. This hierarchical structure allows the quantizer to make better decisions about how to allocate precision across a wider range of weights:

• Q2_K: Aggressive 2-bit quantization, smallest files but noticeable quality loss. Each super-block of 256 weights uses 4-bit scales and 6-bit super-scales, averaging about 2.6 bits per weight including overhead.
• Q3_K_S: 3-bit "small" variant targeting smaller files.
• Q3_K_M: 3-bit "medium", balanced size and quality.
• Q3_K_L: 3-bit "large", prioritizing quality over size within the 3-bit tier.
• Q4_K_S: 4-bit small, a good balance of size and quality for memory-constrained setups.
• Q4_K_M: 4-bit medium, the most popular choice for most users and the typical recommendation for a first try.
• Q5_K_S: 5-bit small, excellent quality at moderate size.
• Q5_K_M: 5-bit medium, very high quality that is often indistinguishable from FP16 on many tasks.
• Q6_K: 6-bit, near-original quality with only about 40% size reduction versus FP16.

The K-quant methods improve on the basic Q4/Q5 formats by using non-uniform quantization. Rather than dividing the weight range evenly into $2^k$ levels, they place more quantization levels where weights are densest. This approach recognizes that weight distributions in neural networks are not uniform: they tend to cluster around zero with long tails, a shape that benefits greatly from non-linear spacing of quantization levels. K-quant methods also use the super-block structure to pool statistics across 256-weight regions, which lets the quantizer make better decisions about scale factors and reduces the per-weight overhead from storing those scale factors.

More recently, "importance matrix" (I-quant) methods have emerged as the frontier of GGUF quantization. These methods incorporate information about which weights matter most for the model's outputs, concentrating the quantization budget where it does the most good:

• IQ1_S, IQ1_M: Extreme 1-bit quantization using importance weighting, producing the smallest possible GGUF files at the cost of substantial quality loss.
• IQ2_XXS, IQ2_XS, IQ2_S, IQ2_M: Various 2-bit importance-weighted variants covering the spectrum from ultra-small to moderate quality.
• IQ3_XXS, IQ3_XS, IQ3_S, IQ3_M: 3-bit variants offering better quality with moderate size.
• IQ4_NL, IQ4_XS: 4-bit with non-linear quantization, often matching or exceeding Q4_K_M quality at similar or smaller sizes.

I-quant methods use calibration data to measure which weights have the most impact on model outputs, then allocate more precision to those weights. This connects to the ideas behind AWQ that we covered in the previous chapter, though implemented at the GGUF level rather than as a training-time procedure. The fundamental insight is the same: not all weights matter equally for model quality, so a smart quantization scheme should spend its limited precision budget where it counts most.

The importance matrix is typically computed by passing representative text through the model and measuring how much each weight's quantization error affects the output activations. Weights that, when perturbed, cause large changes in downstream activations receive more quantization levels. This calibration step takes extra time compared to generating a K-quant model, but the quality improvement at the same bit width is often significant, particularly at the 2-3 bit range where simple quantization struggles most.

The practical trade-offs between quantization types can be substantial. The table below provides estimates for a 7 billion parameter model, showing how different quantization choices affect both storage requirements and expected quality.

The "bits per weight" column includes overhead from scale factors and other metadata, so it is always slightly higher than the base quantization level. This overhead varies by method because different quantization schemes store different amounts of auxiliary information alongside the quantized weights. For most use cases, Q4_K_M offers an excellent balance: models are about 4 times smaller than FP16 with quality losses that are often imperceptible in practice. Empirical benchmarks on standard tasks like MMLU and HellaSwag typically show less than 1% accuracy drop from FP16 to Q4_K_M for well-trained 7B models, which is well within the noise of benchmark variance. This sweet spot makes Q4_K_M the default recommendation for users who want to run large models on consumer hardware while maintaining reasonable quality.

The quality degradation at lower bit widths is not uniform across tasks. Creative writing and casual conversation degrade gradually, while tasks requiring precise factual recall (like answering specific questions about historical dates or scientific constants) degrade faster. This asymmetry is worth keeping in mind when choosing a quantization level: if your application is a coding assistant or a creative writing helper, you can tolerate more aggressive quantization than if you need reliable factual responses.