INT4 Quantization › Practical Implementation with bitsandbytes
Language AI HandbookAll chapters INT4 Quantization › Practical Implementation with bitsandbytesOne section · Sections in this chapter8 / 11 ↓
Reading tools Definitions & inline links
Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.
Article links
Make inline references clickable
The bitsandbytes library provides efficient 4-bit quantization for PyTorch models. It implements NF4 quantization with group-wise scaling and double quantization. This makes it easy to load large models in 4-bit precision. The library handles format conversion, scale management, and dequantization during inference. This provides a simple API that abstracts away the complexity we have been building up in this chapter.
The bitsandbytes approach is called "load-time quantization": rather than requiring a separate offline quantization step, you quantize the model as you load it, with no need for calibration data. This makes it extremely convenient: you can take any model from Hugging Face and run it in 4 bits with just a few configuration lines. The trade-off is that load-time quantization uses simple per-group symmetric quantization without the benefit of calibration data to identify which parameters are most sensitive. More sophisticated approaches like GPTQ (covered in the next chapter) use calibration data to achieve better accuracy at the same bit width.
# Note: This code demonstrates the API but is not executed
# as bitsandbytes requires GPU and specific CUDA setup
# from transformers import BitsAndBytesConfig
#
# # Configure 4-bit quantization
# quantization_config = BitsAndBytesConfig(
# load_in_4bit=True,
# bnb_4bit_quant_type="nf4", # Use NF4 format
# bnb_4bit_compute_dtype=torch.float16, # Compute in FP16
# bnb_4bit_use_double_quant=True, # Enable double quantization
# )
print("BitsAndBytesConfig parameters for INT4 quantization:")
print(" load_in_4bit=True")
print(" bnb_4bit_quant_type='nf4'")
print(" bnb_4bit_compute_dtype=torch.float16")
print(" bnb_4bit_use_double_quant=True")
# Load model with 4-bit quantization
# model = AutoModelForCausalLM.from_pretrained(
# "meta-llama/Llama-2-7b-hf",
# quantization_config=quantization_config,
# device_map="auto"
# )
# tokenizer = AutoTokenizer.from_pretrained("meta-llama/Llama-2-7b-hf")The bnb_4bit_compute_dtype=torch.float16 setting is important and often misunderstood. The weights are stored in INT4, but bitsandbytes dequantizes them to FP16 before performing the actual matrix multiplication. This means the arithmetic operations happen in FP16, not INT4. The reason is practical: current GPU hardware does not have efficient native INT4 matrix multiply units (unlike INT8 and FP16, which have dedicated tensor cores). By dequantizing on the fly, bitsandbytes gets the memory savings of INT4 storage while using the hardware-accelerated FP16 compute. The dequantization step is fast enough that it adds only modest latency overhead.
Let's examine the memory savings:
# Calculate memory savings for a 7B parameter model
n_params = 7_000_000_000
# FP16 (2 bytes per param)
fp16_bytes = n_params * 2
# INT4 + Double Quantization
# 4 bits per weight + quantization overhead (approx 0.125 bits + double quant)
# Roughly 4.2 bits per parameter total
int4_bits_per_param = 4.2
int4_bytes = n_params * (int4_bits_per_param / 8)Model size: 7B parameters ------------------------------ FP16 Memory: 14.00 GB INT4 Memory: 3.67 GB (approximate) Compression: 3.8x
This dramatic reduction allows a 7B model to fit comfortably within the 6 GB or 8 GB VRAM limits of many consumer GPUs, making powerful LLMs accessible on commodity hardware. What was previously possible only on expensive data center GPUs becomes achievable on hardware that many of you already own. For a 70B model, the numbers are even more striking: from 140 GB (requiring multiple A100s) down to roughly 37 GB (fitting on a single A100-80G or two 24 GB consumer GPUs). This is the concrete impact of the techniques covered in this chapter.