Michael Brenndoerfer | Data & AI, Private Equity, Technology

Michael Brenndoerfer

GGUF Format › Converting Models to GGUF

Michael BrenndoerferJanuary 15, 20263 min read

Language AI HandbookAll chapters

Part 42 of 67Inference Optimization

Chapter 322GGUF Format › Converting Models to GGUF

Reading section 8 of 14 · Converting Models to GGUF · 3 minSection 8 of 14 · 3 min

Section 8 of 14 · 3 min read

GGUF Format › Converting Models to GGUFOne section · Sections in this chapter8 / 14 ↓

Sections in this chapter14

Reading tools Definitions & inline links

Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.

Article links

Make inline references clickable

A common workflow is converting a Hugging Face model to GGUF format. The llama.cpp repository includes conversion scripts for this purpose, and understanding the conversion process helps when the official scripts do not yet support a new model or when you need to debug a conversion problem.

The conversion process involves two main steps: first converting from Hugging Face's SafeTensors format to a full-precision GGUF file (FP16 or FP32), and then quantizing that GGUF file to the desired precision. Separating these steps is deliberate. The FP16 GGUF file is a clean starting point from which you can generate any quantization variant without re-downloading the original weights.

In[2]:

Code

# Conceptual conversion workflow (not executed)
# This shows the typical steps when using llama.cpp's convert scripts

# Step 1: Download model from Hugging Face
# huggingface-cli download meta-llama/Llama-2-7b-hf --local-dir ./llama-2-7b

# Step 2: Convert to GGUF (FP16)
# python convert_hf_to_gguf.py ./llama-2-7b --outfile llama-2-7b-f16.gguf

# Step 3: Quantize to desired format
# ./quantize llama-2-7b-f16.gguf llama-2-7b-q4_k_m.gguf Q4_K_M

The conversion process reads the model weights from Hugging Face's SafeTensors format, reorganizes them according to GGUF conventions, and writes out the new file with all necessary metadata extracted from the model's config.json and tokenizer.model. The tensor renaming step is required: Hugging Face uses a naming convention like model.layers.0.self_attn.q_proj.weight, while GGUF uses a standardized convention like blk.0.attn_q.weight. The conversion script maintains a mapping table between these conventions:

In[13]:

Code

# Mapping example: HuggingFace names -> GGUF names for LLaMA
LLAMA_TENSOR_MAP = {
    "model.embed_tokens.weight": "token_embd.weight",
    "model.layers.{}.self_attn.q_proj.weight": "blk.{}.attn_q.weight",
    "model.layers.{}.self_attn.k_proj.weight": "blk.{}.attn_k.weight",
    "model.layers.{}.self_attn.v_proj.weight": "blk.{}.attn_v.weight",
    "model.layers.{}.self_attn.o_proj.weight": "blk.{}.attn_output.weight",
    "model.layers.{}.mlp.gate_proj.weight": "blk.{}.ffn_gate.weight",
    "model.layers.{}.mlp.up_proj.weight": "blk.{}.ffn_up.weight",
    "model.layers.{}.mlp.down_proj.weight": "blk.{}.ffn_down.weight",
    "model.layers.{}.input_layernorm.weight": "blk.{}.attn_norm.weight",
    "model.layers.{}.post_attention_layernorm.weight": "blk.{}.ffn_norm.weight",
    "model.norm.weight": "output_norm.weight",
    "lm_head.weight": "output.weight",
}


def convert_name(hf_name, n_layers=32):
    """Convert HuggingFace tensor name to GGUF name."""
    for layer_idx in range(n_layers):
        for hf_pattern, gguf_pattern in LLAMA_TENSOR_MAP.items():
            hf_test = hf_pattern.format(layer_idx)
            if hf_name == hf_test:
                return gguf_pattern.format(layer_idx)

    # Non-layer-specific names
    return LLAMA_TENSOR_MAP.get(hf_name, hf_name)

Out[14]:

Console

Name conversion examples:
  model.embed_tokens.weight
    -> token_embd.weight
  model.layers.0.self_attn.q_proj.weight
    -> blk.0.attn_q.weight
  model.layers.15.mlp.down_proj.weight
    -> blk.15.ffn_down.weight
  model.norm.weight
    -> output_norm.weight

The conversion examples show how Hugging Face tensor names map to GGUF's standardized naming conventions. Layer-specific weights transform from the nested model.layers.N.component structure to the flatter blk.N.component structure. Model-level tensors like the embedding table (token_embd.weight) and the output projection (output.weight) use fixed names that llama.cpp looks for by convention. This consistent naming allows llama.cpp to load any LLaMA-family model without architecture-specific lookup logic at inference time.