GGUF Format › Worked Example: Interpreting a Real GGUF Model
Language AI HandbookAll chapters GGUF Format › Worked Example: Interpreting a Real GGUF ModelOne section · Sections in this chapter7 / 14 ↓
Reading tools Definitions & inline links
Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.
Article links
Make inline references clickable
To make the file structure concrete, let us trace through what we would see inspecting a real 7B LLaMA model stored as GGUF. We will not load an actual model (too large to download during building), but we will walk through the numbers that a real file would contain and what each value means.
A Llama-2-7B-Q4_K_M.gguf file begins with the 24-byte header. Reading the first four bytes gives 0x47 0x47 0x55 0x46, which is "GGUF" in ASCII. The next four bytes are 0x03 0x00 0x00 0x00 (little-endian), confirming version 3. The next eight bytes give the tensor count: 291 tensors in total. The final eight bytes of the header give the metadata count: typically around 30-50 key-value pairs depending on the model variant.
The metadata section then begins. Parsing the key-value pairs reveals entries like:
general.architecture: [string] "llama"
general.name: [string] "LLaMA v2"
llama.context_length: [uint32] 4096
llama.embedding_length: [uint32] 4096
llama.block_count: [uint32] 32
llama.feed_forward_length: [uint32] 11008
llama.attention.head_count: [uint32] 32
llama.attention.head_count_kv: [uint32] 32
llama.attention.layer_norm_rms_epsilon: [float32] 1e-05
llama.rope.dimension_count: [uint32] 128
tokenizer.ggml.model: [string] "llama"
tokenizer.ggml.tokens: [array] (32000 items)
tokenizer.ggml.scores: [array] (32000 items)
tokenizer.ggml.token_type: [array] (32000 items)
Notice that the full tokenizer vocabulary of 32,000 entries is embedded directly in the file. Each token is a string, each score is a float32 used by the SentencePiece-based tokenizer to resolve ambiguous segmentations, and each type is an integer indicating whether the token is a normal piece, a byte fallback, a special marker like <s> or <unk>, or an unused slot. This embedded tokenizer is one of GGUF's most important contributions: it eliminates the need for a separate tokenizer.model file and ensures the tokenizer behavior is always exactly matched to the weights.
After the metadata, the tensor information section lists all 291 tensors. The first few entries look like:
token_embd.weight: shape=[4096, 32000], type=Q4_K, offset=0
output_norm.weight: shape=[4096], type=F32, offset=<X bytes>
output.weight: shape=[4096, 32000], type=Q6_K, offset=<Y bytes>
blk.0.attn_norm.weight: shape=[4096], type=F32, offset=<Z bytes>
blk.0.attn_q.weight: shape=[4096, 4096], type=Q4_K, offset=<W bytes>
...
The key insight is that different layers use different quantization types. The embedding and output projection tensors use Q6_K rather than Q4_K because they are the interface between the discrete token vocabulary and the continuous weight space. Small errors in these layers directly corrupt token representations. Normalization weights (ending in _norm.weight) are stored as F32 because they are only 4,096 numbers and their precise values matter for numerical stability. The bulk of the attention and feed-forward weight matrices use Q4_K, where the compression is highest and the quality impact lowest.
The total file size works out to approximately 4.1 GB. We can verify this independently: the embedding matrix is 4096 times 32000 values. At Q4_K's roughly 4.5 bits per weight including overhead, that is 4096 times 32000 times 4.5 / 8 bytes, which is about 74 MB. The 32 transformer blocks each contain attention weight matrices (Q, K, V, output projections: 4 matrices of 4096 times 4096 at Q4_K) and feed-forward matrices (gate, up, down: 3 matrices of 4096 times 11008 at Q4_K). Running these calculations, the attention weights per layer are roughly 37 MB, the feed-forward weights roughly 74 MB, and the normalization weights a negligible 0.03 MB. Summing over 32 layers and adding the embedding and output tensors gives approximately 3.6 to 4.1 GB, matching the expected size.