Product Quantization: Vector Compression for ANN Search › PQ for Scale

Michael BrenndoerferJanuary 28, 20264 min read
Language AI HandbookAll chapters
Part 44 of 67Retrieval-Augmented Generation
Chapter 341Product Quantization: Vector Compression for ANN Search › PQ for Scale
Reading section 9 of 12 · PQ for Scale · 4 minSection 9 of 12 · 4 min

Book contents

Language AI Handbook67 parts · 513 chaptersView the whole book →
Active chapter 341 of 513

Section 9 of 12 · 4 min read

Product Quantization: Vector Compression for ANN Search › PQ for ScaleOne section · Sections in this chapter9 / 12 ↓
Sections in this chapter12
Reading tools Definitions & inline links

Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.

Article links

Make inline references clickable

The real power of PQ shows when you scale to billions of vectors. Let us put the numbers in perspective.

Most discussions of PQ focus on the compression ratio, but the more important question is what compression enables that would otherwise be impossible. At a billion vectors, even the question of which machine architecture to use changes depending on whether you can fit the index in RAM. If you cannot, you are forced into a hybrid architecture where part of the index lives on disk or on remote machines, and you pay latency costs for every remote access. If you can, you get sub-millisecond in-process lookups with no network round-trips. PQ is often the technology that decides which side of that divide you fall on.

For a billion-vector index with 768-dimensional float32 embeddings:

  • Flat index: 109×768×4=3.0710^9 \times 768 \times 4 = 3.07 TB. This requires a distributed fleet of machines and terabytes of RAM.
  • PQ with M=96, nbits=8: 109×96×1=9610^9 \times 96 \times 1 = 96 GB (96 bytes/vector, since each of the 96 subspaces stores one byte). This fits on a single server with 128 GB RAM.
  • PQ with M=48, nbits=8: 109×48×1=4810^9 \times 48 \times 1 = 48 GB (48 bytes/vector). This fits on a workstation.

The difference is not incremental: it is the difference between a system that requires purpose-built infrastructure and one that runs on commodity hardware. A researcher studying billion-scale retrieval can now run experiments on a single rented cloud instance instead of managing a cluster. An engineering team can ship a production retrieval system without a dedicated infrastructure team. PQ makes these deployments feasible.

Out[9]:
Visualization
Log-scale bar chart of memory footprint in GB for five index configurations at 1 billion vectors, comparing flat float32 storage exceeding 3 TB against PQ variants with M=96 to M=12, with reference lines at 48 GB, 128 GB, and 1 TB.
Memory footprint in GB (log-scale y-axis) for a one-billion-vector index of 768-dimensional float32 embeddings across five storage configurations: full-precision flat storage at over 3 TB, and PQ with M=96, 48, 24, and 12 subspaces (nbits=8). Bar labels show exact memory in GB or TB. PQ reduces memory by one to two orders of magnitude relative to the flat index. Horizontal reference lines mark 48 GB (workstation), 128 GB (server), and 1 TB (cluster node), illustrating that PQ with M=48 or below fits on a single commodity server while flat storage requires a distributed cluster.

When combined with IVFPQ, you get the full stack: IVF reduces the number of vectors you compare (by a factor of nlist/nprobe\text{nlist} / \text{nprobe}), and PQ reduces the cost of each comparison. Production systems such as Pinecone and Weaviate use variants of IVFPQ as compressed index formats, as does Milvus. The upcoming chapters on hybrid search and reranking will show how these approximate indexes fit into a larger retrieval pipeline, where you use PQ for first-pass retrieval and then apply a full-precision reranker to the top candidates.