AWQ › Key Parameters

Understanding AWQ's quantization parameters allows you to tune the algorithm for your specific model and deployment target. The most important parameters are:

• w_bit: The bit width for weight quantization. The standard choice is 4 bits (INT4), which gives a 4x compression ratio relative to FP16. Some deployments use 8 bits for higher accuracy with a 2x compression ratio, while aggressive compression targets use 3 bits at the cost of more accuracy degradation.

• q_group_size: The number of consecutive weights that share a single quantization scale and zero point. Smaller group sizes (e.g., 32 or 64) give finer-grained quantization with better accuracy but require more metadata storage. Larger group sizes (e.g., 256 or 512) reduce metadata overhead but may allow more quantization error. The value 128 is a commonly effective default that balances these concerns.

• zero_point: Whether to use asymmetric quantization (True) or symmetric (False). Asymmetric quantization with a zero point allows the integer grid to stand for values that are not centered at zero, which better handles weight distributions that are skewed. Most modern language model weights benefit from asymmetric quantization. Symmetric quantization is simpler and may be preferred on hardware that does not support efficient asymmetric dequantization.

• version: The kernel version for inference. "GEMM" kernels are optimized for general matrix multiplication and work well across a wide range of batch sizes and sequence lengths. "GEMV" kernels are specialized for single-vector multiplication (batch size 1) and give better throughput for interactive, low-latency use cases.