Michael Brenndoerfer | Data & AI, Private Equity, Technology

Michael Brenndoerfer

GGUF Format › Memory Requirements

Michael BrenndoerferJanuary 15, 20269 min read

Language AI HandbookAll chapters

Part 42 of 67Inference Optimization

Chapter 322GGUF Format › Memory Requirements

Reading section 10 of 14 · Memory Requirements · 9 minSection 10 of 14 · 9 min

Section 10 of 14 · 9 min read

GGUF Format › Memory RequirementsOne section · Sections in this chapter10 / 14 ↓

Sections in this chapter14

Reading tools Definitions & inline links

Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.

Article links

Make inline references clickable

Understanding memory requirements helps you choose the right quantization level for your hardware. The relevant constraint is the total memory footprint during inference, not simply whether the model weights fit in RAM. That footprint includes three components: the model weights (whose size depends on quantization), the KV cache (which grows with context length and batch size), and temporary buffers for intermediate activations during the forward pass.

Getting these estimates right before downloading a multi-gigabyte model is time well spent. A model that barely fits in RAM will use virtual memory, causing dramatic slowdowns as the OS pages data in from disk. A rough rule of thumb is to have 20-30% headroom beyond your estimated peak memory to avoid this situation.

In[15]:

Code

def estimate_model_size(params_billion, quant_type):
    """Estimate model size in GB for different quantization types."""
    params = params_billion * 1e9

    bits_per_weight = {
        "F32": 32.0,
        "F16": 16.0,
        "Q8_0": 8.5,
        "Q6_K": 6.6,
        "Q5_K_M": 5.7,
        "Q5_0": 5.5,
        "Q4_K_M": 4.8,
        "Q4_0": 4.5,
        "Q3_K_M": 3.9,
        "Q2_K": 3.4,
        "IQ2_M": 2.7,
    }

    bits = bits_per_weight.get(quant_type, 4.8)
    bytes_needed = (params * bits) / 8
    gb = bytes_needed / (1024**3)

    return gb

Out[16]:

Console

Model Size Estimates (weights only)
==================================================

7B parameter model:
  F16     :   13.0 GB
  Q8_0    :    6.9 GB
  Q5_K_M  :    4.6 GB
  Q4_K_M  :    3.9 GB
  Q3_K_M  :    3.2 GB
  Q2_K    :    2.8 GB

13B parameter model:
  F16     :   24.2 GB
  Q8_0    :   12.9 GB
  Q5_K_M  :    8.6 GB
  Q4_K_M  :    7.3 GB
  Q3_K_M  :    5.9 GB
  Q2_K    :    5.1 GB

34B parameter model:
  F16     :   63.3 GB
  Q8_0    :   33.6 GB
  Q5_K_M  :   22.6 GB
  Q4_K_M  :   19.0 GB
  Q3_K_M  :   15.4 GB
  Q2_K    :   13.5 GB

70B parameter model:
  F16     :  130.4 GB
  Q8_0    :   69.3 GB
  Q5_K_M  :   46.4 GB
  Q4_K_M  :   39.1 GB
  Q3_K_M  :   31.8 GB
  Q2_K    :   27.7 GB

Out[17]:

Visualization

Grouped bar chart showing model weight storage in GB across four model sizes and six quantization formats. — Model weight storage requirements across quantization formats and model sizes. A 7B model drops from 14 GB in F16 format to 4.1 GB with Q4_K_M quantization, achieving 3.4x compression. Red dashed reference lines at 8 GB, 16 GB, and 32 GB mark common consumer RAM capacities. At 70B parameters, Q3_K_M sits just above 32 GB, while only the more aggressive Q2_K option falls below that threshold.

The model size estimates show the practical impact of quantization. A 7B model in FP16 requires about 14 GB, placing it out of reach for most laptops and consumer desktops. Q4_K_M quantization reduces this to approximately 4.1 GB, a 3.4 times reduction that brings it comfortably within the range of an 8 GB laptop. Moving to more aggressive quantization like Q2_K further reduces size to 2.9 GB, at the cost of noticeable quality degradation. These estimates cover only the model weights and do not include the KV cache, which can be substantial for long contexts.

KV Cache Memory FormulaLink Copied

To understand how much memory the KV cache requires, we need to account for the number of layers storing caches, the size of each cache entry, the number of tokens being cached, and the precision of the cached values. During inference, the model stores attention keys and values for all tokens processed so far, allowing subsequent token generation to attend back to them without recomputation. This cache grows linearly with sequence length, making its size a critical factor in memory planning.

The KV cache memory requirement follows a clean formula:

\text{KV Cache (GB)} = \frac{2 \times L \times d \times C \times b_{\text{kv}}}{8 \times 1024^3}

where:

$L$ is the number of transformer layers. Each layer maintains its own independent key-value cache, storing the computed attention keys and values for all tokens processed so far.
$d$ is the hidden dimension size (embedding length). Every cached key or value is a vector of this dimensionality.
$C$ is the context length. It represents how many token positions need to be cached. The actual cache grows from 0 to $C$ tokens during inference.
$b_{\text{kv}}$ is the number of bits per KV value (16 for FP16, 8 for quantized cache, etc.). Lower precision reduces memory at some potential cost to attention quality.
$2$ accounts for both key and value caches: every attention layer stores keys used for similarity computation and values used for information retrieval.
$8$ converts bits to bytes.
$1024^3$ converts bytes to gigabytes.

The key insight is that every factor in the numerator contributes multiplicatively. Doubling any of $L$ , $d$ , or $C$ doubles the cache size. This linear scaling in context length $C$ is what makes long-context inference so memory-hungry: a model designed for 4K tokens but pushed to 128K tokens needs 32 times more KV cache memory.

Let us work through the full calculation step by step for a typical 7B model with:

$L = 32$ transformer layers
$d = 4096$ hidden dimensions
$C = 4096$ context length
$b_{\text{kv}} = 16$ bits (FP16 precision)

Substituting into the formula:

\begin{aligned} \text{KV Cache (GB)} &= \frac{2 \times 32 \times 4096 \times 4096 \times 16}{8 \times 1024^3} && \text{(substitute all values)} \\ &= \frac{17{,}179{,}869{,}184}{8{,}589{,}934{,}592} && \text{(evaluate numerator and denominator)} \\ &\approx 2.0 \text{ GB} && \text{(divide)} \end{aligned}

So the KV cache for a 7B model at 4K context requires approximately 2 GB of additional memory beyond the model weights. This means a Q4_K_M model needs roughly 4.1 + 2.0 = 6.1 GB total, fitting comfortably in 8 GB of RAM but leaving little headroom for the OS and other applications.

We can build intuition for this number by breaking the calculation into stages. First, the total number of scalar values that need caching:

\begin{aligned} \text{Total values} &= 2 \times 32 \times 4096 \times 4096 && \text{(2 caches, L layers, d dimensions, C tokens)} \\ &= 1{,}073{,}741{,}824 && \text{(just over 1 billion numbers)} \end{aligned}

Over a billion numbers must be stored in memory just to hold the attention state for a 4K-token context. Next, converting to bytes using FP16 (2 bytes per value):

\begin{aligned} \text{Total bytes} &= 1{,}073{,}741{,}824 \times 2 && \text{(values times bytes per FP16 value)} \\ &= 2{,}147{,}483{,}648 \text{ bytes} && \text{(about 2 billion bytes)} \end{aligned}

Finally, converting to gigabytes:

\begin{aligned} \text{Total GB} &= \frac{2{,}147{,}483{,}648}{1{,}073{,}741{,}824} && \text{(bytes divided by bytes per GB)} \\ &= 2.0 \text{ GB} && \text{(the final answer)} \end{aligned}

The scaling behavior has important practical implications. Since context length $C$ appears as a direct multiplier, extending the context window multiplies the KV cache requirement proportionally:

At 8K context: $2 \text{ GB} \times 2 = 4 \text{ GB}$
At 16K context: $2 \text{ GB} \times 4 = 8 \text{ GB}$
At 32K context: $2 \text{ GB} \times 8 = 16 \text{ GB}$
At 128K context: $2 \text{ GB} \times 32 = 64 \text{ GB}$

This makes KV cache management critical for long-context applications. Moving from 4K to 128K contexts requires 32 times more memory for the cache alone, which quickly exceeds even the most well-equipped consumer hardware. This scaling behavior explains why many long-context models employ specialized techniques like sliding window attention, sparse attention patterns, and quantized KV caches to limit memory growth.

Out[18]:

Visualization

Log-scale line chart showing KV cache memory in GB versus context length for 7B, 13B, and 70B model sizes. — KV cache memory requirements scale linearly with context length across three model sizes using FP16 precision. A 7B model requires 2 GB of cache at 4K context but reaches 16 GB at 32K context. The red dashed lines at 8 GB, 16 GB, and 32 GB mark common consumer RAM thresholds. The steep slopes illustrate why long-context inference rapidly exhausts available memory and motivates specialized techniques like sliding window attention and quantized KV caches.

The scaling chart makes the memory challenge visceral. A 70B model requires about 10 GB of KV cache at 4K context and exceeds 32 GB by 16K context. Even the modest 7B model hits the 8 GB threshold around 16K context, meaning that many laptop users running local models cannot use the full context window their model was trained with. Quantizing the KV cache (reducing $b_{\text{kv}}$ from 16 to 8 or even 4 bits) offers a way to cut these requirements in half or more, and llama.cpp supports KV cache quantization as a deployment option.