GGUF Format › Running Inference with llama.cpp Python Bindings

Michael BrenndoerferJanuary 15, 20263 min read
Language AI HandbookAll chapters
Part 42 of 67Inference Optimization
Chapter 322GGUF Format › Running Inference with llama.cpp Python Bindings
Reading section 9 of 14 · Running Inference with llama.cpp Python Bindings · 3 minSection 9 of 14 · 3 min

Book contents

Language AI Handbook67 parts · 513 chaptersView the whole book →
Active chapter 322 of 513

Section 9 of 14 · 3 min read

GGUF Format › Running Inference with llama.cpp Python BindingsOne section · Sections in this chapter9 / 14 ↓
Sections in this chapter14
Reading tools Definitions & inline links

Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.

Article links

Make inline references clickable

While llama.cpp is primarily a C/C++ project, Python bindings are available through the llama-cpp-python package. This package wraps the native library with a Python API that closely mirrors the OpenAI chat completions interface, making it easy to prototype with local models and swap between cloud and local inference with minimal code changes.

In[2]:
Code
# Example using llama-cpp-python (not executed due to model size)
from llama_cpp import Llama

# Load a GGUF model
llm = Llama(
    model_path="./models/llama-2-7b-q4_k_m.gguf",
    n_ctx=2048,  # Context window
    n_threads=4,  # CPU threads
    n_gpu_layers=0,  # Layers to offload to GPU (0 = CPU only)
)

# Generate text
output = llm(
    "The capital of France is",
    max_tokens=32,
    temperature=0.7,
    top_p=0.9,
    echo=True,
)

print(output["choices"][0]["text"])

The key parameters when loading a model are worth understanding in detail, as they significantly affect both performance and quality:

  • n_ctx: the context window size. Larger contexts require more memory for the KV cache, growing linearly with sequence length. Setting this too large wastes memory; setting it too small truncates long inputs.
  • n_threads: number of CPU threads for inference. Setting this to the number of physical cores (not logical/hyperthreaded cores) usually gives the best performance, since transformer inference is compute-bound rather than thread-bound.
  • n_gpu_layers: how many transformer layers to offload to GPU. Setting this to -1 offloads all layers if VRAM permits; partial offloading helps when VRAM is insufficient for the full model.
  • n_batch: batch size for prompt processing. Larger batches speed up prompt evaluation (the prefill phase) but require more memory for intermediate activations.

For GPU acceleration, llama.cpp supports multiple backends. The choice depends on your hardware and platform:

In[4]:
Code
# GPU offloading examples

# Full GPU offload (requires sufficient VRAM)
llm = Llama(
    model_path="model.gguf",
    n_gpu_layers=-1,  # -1 = all layers
)

# Partial offload (when VRAM is limited)
llm = Llama(
    model_path="model.gguf",
    n_gpu_layers=20,  # First 20 layers on GPU
)

# Split across multiple GPUs
llm = Llama(
    model_path="model.gguf",
    n_gpu_layers=-1,
    tensor_split=[0.5, 0.5],  # 50% on each of 2 GPUs
)

Partial GPU offloading is a uniquely powerful feature of llama.cpp. When you cannot fit the entire model in VRAM, you can still get significant speedups by putting the first N layers on GPU and running the rest on CPU. The layers that see the most computation time (typically the early and middle layers) get GPU acceleration, while the layers near the output run on CPU. Determining the optimal split requires some experimentation, but a common heuristic is to fill VRAM to about 80% capacity and see how many layers that accommodates.