GGUF Format › From GGML to GGUF

Understanding where GGUF came from helps you appreciate the design choices it embodies. The evolution from GGML to GGUF was driven by practical pain points that accumulated as the llama.cpp ecosystem scaled from a single model to dozens of architectures.

The story of GGUF begins with GGML (Georgi Gerganov Machine Learning), a tensor library written in C that Georgi Gerganov created in 2022. Unlike frameworks such as PyTorch or TensorFlow that were designed for training on GPUs, GGML focused exclusively on inference and was optimized primarily for CPUs. This seemingly counterintuitive choice proved wise: while GPUs excel at parallel matrix operations, CPUs are available everywhere and can efficiently run quantized models that fit in system memory. A CPU with 16 GB of RAM can comfortably handle a 4-bit quantized 7B model, which requires only about 4 GB for the weights plus some overhead for the KV cache and activations.

The original GGML library used a simple file format with a binary header followed by tensor data. Each model architecture had its own loading code, and metadata about the model (vocabulary, hyperparameters, tokenizer configuration) was stored in separate files or hardcoded into the loading routines. This worked, but it was fragile and did not scale well as the community wanted to support more architectures. Adding support for a new model family like Falcon or MPT required writing bespoke parsing code, carefully tracking which version of the format a given file used, and hoping that the separate tokenizer files were present and compatible. Version drift was a constant problem: a GGML file created with one version of llama.cpp might not load correctly with a newer version that had changed the header layout or added new fields.

In August 2023, the community introduced GGUF as a successor format. The key improvements are worth understanding in depth, because each one addresses a real problem that GGML users had encountered:

• Self-contained metadata: All model information lives in a single file, including the full tokenizer vocabulary and special token definitions. You no longer need a separate tokenizer.model or config.json alongside the weights.
• Architecture-agnostic design: The same format works for any supported architecture. A single parser can read any GGUF file; architecture-specific logic only kicks in when the inference engine decides how to construct the compute graph.
• Extensibility: New metadata fields can be added without breaking backward compatibility. A GGUF reader that does not understand a particular key can simply skip it, allowing the format to evolve gracefully.
• Memory mapping: Tensor data is aligned such that it can be accessed directly from disk without loading the entire file into RAM first. This enables models larger than available RAM to be used via virtual memory, with the OS paging in tensor data on demand.
• Alignment guarantees: Data is aligned to specific boundaries for efficient memory access. Modern CPUs benefit significantly from aligned memory access, and SIMD instructions often require or strongly prefer it.

The transition was not instantaneous. For several months you would see both .ggml and .gguf files floating around the community. Today, GGUF has essentially replaced GGML entirely, and the llama.cpp project only supports the newer format. The ecosystem learned from the GGML experience, and GGUF's design reflects those lessons clearly.