GGUF Format › GGUF File Structure

A GGUF file consists of three main sections: a header, metadata key-value pairs, and tensor data. Understanding this structure helps when you need to inspect models, debug loading issues, or build your own tooling. The design philosophy behind this organization prioritizes both human readability during debugging and machine efficiency during loading. By separating concerns into distinct sections, parsers can quickly validate a file, extract configuration without touching the weights, or memory-map only the tensor data for inference.

Think of the file structure as a well-organized book. The header is the title page and copyright information, giving you the basics at a glance. The metadata section is the table of contents and appendices. This provides all the reference material you need to understand what comes next. The tensor data section is the main body of the book, the actual content that carries the knowledge. A reader who needs only the table of contents does not need to page through the entire book, and a GGUF parser that needs only the model's architecture configuration does not need to touch the weight data.

The file begins with a fixed header that identifies it as GGUF and provides the counts needed to parse the rest. This header is both a validation mechanism and a roadmap for the parser: it tells parsers exactly how much data to expect in subsequent sections.

Each field in the header plays a specific role:

• Offset: the byte position from the start of the file where each field begins, measured in bytes from position 0.
• Size: the number of bytes allocated for each field.
• Magic number (0x46554747): a specific 4-byte sequence that spells "GGUF" in little-endian byte order, used to quickly verify file type validity. When a parser reads these first four bytes and finds this exact sequence, it can identify the file as GGUF rather than another format or corrupted data. This is a standard technique in binary file formats: PNG files begin with \x89PNG, ZIP files with PK, and GGUF files with GGUF.
• Version (currently 3): the format version number, allowing parsers to handle different format iterations while maintaining backward compatibility. Versions 1 and 2 had different alignment and encoding rules; version 3 is the current standard.
• Number of tensors: the total count of weight tensors stored in this file, allowing parsers to allocate appropriate data structures before reading tensor metadata.
• Number of metadata key-value pairs: the count of metadata entries following the header, enabling efficient parsing by eliminating the need to scan for end markers.

The magic number lets tools quickly verify they are looking at a GGUF file without parsing any further. This matters in practice: when a user accidentally tries to load a SafeTensors file or a corrupt download, a tool that checks the magic number immediately reports "not a GGUF file" rather than producing confusing errors deep in the parsing process. The versioning strategy means that as the community discovers better ways to organize data or needs to support new quantization methods, the format can adapt without breaking existing files.

Following the header, metadata is stored as a sequence of key-value pairs. This section contains all the information an inference engine needs to understand and execute the model, from basic architecture parameters to complete tokenizer vocabularies. Each pair consists of three components: a string key (UTF-8, length-prefixed), a type code indicating the value type, and the value itself.

The main point about GGUF's metadata design is that it is self-describing at every level. The type codes tell the parser how many bytes to read for each value, so parsing requires no external schema. A parser can read a GGUF file it has never seen before and correctly interpret every metadata field, even if it does not know what some fields mean. This provides forward compatibility: when new metadata keys are introduced, old parsers can skip them without corrupting the parse state.

The format supports these value types:

• Scalars: uint8, int8, uint16, int16, uint32, int32, uint64, int64, float32, float64, bool
• String: UTF-8 encoded, length-prefixed with a 64-bit length
• Array: a type code, followed by a count, followed by that many values of the specified type

GGUF defines a standard set of metadata keys with specific meanings. The most important ones for configuring the inference engine include:

• general.architecture: the model family (e.g., "llama", "falcon", "mpt")
• general.name: human-readable model name
• general.quantization_version: version of the quantization process used
• {arch}.context_length: maximum sequence length the model supports
• {arch}.embedding_length: hidden dimension size
• {arch}.block_count: number of transformer layers
• {arch}.attention.head_count: number of attention heads
• {arch}.attention.head_count_kv: number of key-value heads (relevant for grouped-query attention)
• tokenizer.ggml.model: tokenizer type ("llama", "gpt2", etc.)
• tokenizer.ggml.tokens: array of vocabulary tokens
• tokenizer.ggml.scores: token scores for SentencePiece models
• tokenizer.ggml.token_type: type of each token (normal, special, byte fallback, etc.)

The {arch} placeholder substitutes the architecture name from general.architecture. So for a LLaMA model, the context length key is llama.context_length. This namespacing prevents conflicts between architectures that might use the same parameter names with different semantics.

The self-describing nature of GGUF means that a single inference engine can load any supported model architecture without prior knowledge of its specific configuration. Before GGUF, adding support for a new model family required modifying the inference engine's source code to add new parsing rules. With GGUF, the engine reads general.architecture, looks up the corresponding layer structure, and uses the metadata keys to parameterize that structure. New architectures can be added without changing the file format at all.

After the metadata comes information about each tensor. This section acts as an index or table of contents for the actual weight data. This provides everything needed to locate and interpret each tensor without having to parse the tensor data itself. For each tensor, the file stores:

1. Name (UTF-8 string, identifying which layer and weight type this tensor represents)
2. Number of dimensions (a small integer, typically 1 or 2 for weight matrices)
3. Dimensions array (the shape of the tensor)
4. Data type (which quantization format is used for this tensor)
5. Offset into the data section (byte position where this tensor's data begins)

This section contains only tensor metadata, not the weights themselves. The tensor data comes later, after alignment padding. This separation enables efficient loading. By placing tensor metadata before the data, parsers can build a complete map of all tensors, allocate memory appropriately, and then load only the tensors needed for inference. In a speculative decoding setup, for example, a tool might want to inspect tensor shapes and sizes before deciding how to distribute layers across multiple devices.

The data type field is particularly important because it determines how the inference engine interprets the bytes in the tensor data section. A float32 tensor and a Q4_K_M tensor containing the same layer weights will have completely different byte layouts and sizes. The type field ensures the engine uses the correct decoding path for each tensor.

The final section contains the actual tensor data, aligned to a specified boundary (typically 32 bytes) for efficient memory-mapped access. When loading a GGUF file, inference engines can memory-map this section directly rather than copying the data into newly allocated memory. This dramatically reduces loading time for large models, which can otherwise take 30-60 seconds just to read from disk.

Memory mapping allows the operating system to manage which portions of the model are resident in RAM. The OS's virtual memory system acts as a smart cache: when the inference engine needs a tensor, the OS fetches the corresponding pages from disk. If available RAM is limited, the OS evicts pages that have not been used recently. This means you can technically run a model that is larger than your available RAM, though with performance penalties as the OS continually pages data in and out.

The tensor data is stored in the quantized format specified in the tensor information section. Different tensors can use different quantization formats within the same file, enabling mixed-precision approaches where attention weights might use higher precision than feed-forward layers. This flexibility is essential for optimizing the quality-size trade-off, as different parts of a model have different sensitivities to quantization error. Embedding tables, for instance, are often kept at higher precision because they directly map token identities to representations, while the middle layers of deep feed-forward networks tolerate more aggressive quantization.