Weight Quantization Basics › Weight Quantization Basics: Scale, Zero-Point & Calibration

Large language models are memory hogs. A 7 billion parameter model stored in 16-bit floating point requires 14 GB just for the weights, and larger models scale proportionally: 70 billion parameters means roughly 140 GB of weight storage before you have loaded a single input token. The practical implication is stark. Even if you have a GPU powerful enough to run these models in principle, you may simply not have enough VRAM to hold the weights. The model never gets a chance to run. Memory capacity is the hard constraint that determines whether a model is deployable at all.

But even when you have enough memory, a second constraint governs inference speed. Modern GPUs are extraordinarily fast at computation, but moving data between GPU memory and the compute units costs time. Every forward pass must load billions of parameters across this memory bus, and this data movement dominates inference latency when the batch size is small. Most production LLM serving operates with small or unit batch sizes, because users arrive one at a time, and each request must be answered quickly. Under these conditions, the GPU spends most of its time waiting for data rather than computing. You have a sports car stuck in traffic.

Quantization provides a direct solution to both problems: represent model weights using fewer bits. Instead of 16 or 32 bits per parameter, we can use 8, 4, or even fewer bits. This shrinks memory footprint, reduces bandwidth requirements, and often enables hardware-accelerated integer arithmetic that runs faster than floating-point operations. A 4-bit quantized 70B model fits in roughly 35 GB instead of 140 GB, which makes it deployable on hardware that could never touch the original. The bandwidth reduction is proportional: 4-bit weights move four times less data across the memory bus compared to 16-bit weights, directly accelerating generation speed.

But quantization is not free. Cramming continuous values into discrete bins inevitably loses information. Floating-point numbers can represent a large range of values with fine granularity: a 32-bit float can distinguish values that differ by one part in ten million within a given range. When we compress that into 8 integers, we can only distinguish 256 different values across the same range. Some information is irretrievably lost. The goal of quantization is minimizing this information loss while maximizing compression, and it turns out that neural networks are remarkably tolerant of carefully applied quantization. The challenge lies in doing it carefully.

This chapter covers the basic concepts that every quantization technique builds upon. We start with the arithmetic of how floating-point values map to integers: what scale and zero-point mean, how to derive them, and what the resulting error looks like. We then examine the two main quantization schemes, symmetric and asymmetric, which trade computational simplicity against representation efficiency. We cover quantization granularity, from a single scale for an entire tensor to separate scales for each group of values, and explain why finer granularity dramatically improves accuracy at modest storage cost. Finally, we examine calibration: the process of determining optimal quantization parameters from data. Every advanced quantization method you will encounter in later chapters builds on these foundations. Understanding them deeply is the prerequisite for understanding everything that follows.