Model Serving › Serving Frameworks

Several mature frameworks handle LLM serving at production scale. Each makes different trade-offs among deployment flexibility, inference performance, and operational complexity.

vLLM is one of the most widely used open-source LLM serving frameworks, built around the PagedAttention memory management technique. Instead of allocating contiguous memory blocks for KV caches, vLLM manages KV cache memory as fixed-size pages, similar to how an operating system manages virtual memory. This eliminates fragmentation and allows many more concurrent requests to share GPU memory efficiently.

vLLM exposes an OpenAI-compatible API endpoint, making it a drop-in replacement for applications built against the OpenAI API. It supports tensor parallelism across multiple GPUs, quantization for reducing memory requirements, and continuous batching for high throughput. The framework handles most low-level serving concerns automatically, letting engineers focus on deployment configuration rather than inference implementation.

vLLM also supports prefix caching: if multiple requests share a common prefix (such as a long system prompt), the KV cache for that prefix is computed once and shared across all requests. For applications where 80% of every request is identical system instructions, this can dramatically reduce both compute and memory usage. A 1000-token system prompt that appears in every request needs to be prefilled only once per hour (when the cached KV blocks expire) rather than once per request.

The preemption_mode parameter is worth a closer look. When the server runs out of KV cache memory while active requests are in progress, it must free memory from some requests so others can continue. Two strategies exist. With "recompute" mode, the server discards the KV cache of the preempted request entirely, forcing it to rerun its prefill phase from scratch when memory becomes available. With "swap" mode, the KV cache is copied to CPU memory (a slower, larger storage tier), freeing GPU memory, and later copied back when the request resumes. Recompute is simpler and requires no CPU memory budget but wastes compute. Swap preserves work but requires careful bandwidth management to avoid the swap itself becoming a bottleneck.

Text Generation Inference from Hugging Face offers tight integration with the Hugging Face model hub and the Transformers library. TGI provides production-grade serving with features like continuous batching, tensor parallelism, quantization, and token streaming. Its primary advantage is direct ecosystem integration: any model available on the Hugging Face hub can typically be deployed with TGI without custom loading code.

TGI exposes both a REST API and a gRPC interface. The REST API is compatible with many existing LLM client libraries. TGI also supports speculative decoding out of the box, pairing a small draft model with the main model to accelerate generation for common token sequences. The Hugging Face team maintains a Docker image that bundles TGI with optimized kernels, making deployment as straightforward as running a single docker run command with the model name as an argument.

TGI's tight integration with the Hugging Face hub also means that model updates are simple: changing the model name in the deployment configuration and restarting the container loads the new model automatically, without any changes to serving infrastructure code.

NVIDIA's TensorRT-LLM takes a compilation-based approach. Models are compiled into highly optimized CUDA inference engines that use custom kernels for attention, layer normalization, and other operations. The compilation step is slow and hardware-specific, but the resulting engines often achieve the highest raw inference throughput on NVIDIA GPUs.

TensorRT-LLM supports in-flight batching, INT8 and FP8 quantization, tensor parallelism, and speculative decoding. It integrates with NVIDIA Triton Inference Server for production deployment, giving load balancing, model versioning, and monitoring capabilities. The main trade-off is the setup complexity: compiling a new model or changing hardware requires rerunning the optimization pipeline, which can take tens of minutes for large models.

For organizations that standardize on NVIDIA hardware and prioritize maximum throughput over flexibility, TensorRT-LLM can deliver higher tokens-per-second than framework-agnostic alternatives. The compilation process applies hardware-specific optimizations that generic runtimes cannot perform at execution time: custom fused kernels that merge multiple operations into a single GPU pass, memory layouts that minimize data movement, and quantization-aware kernel selection that maps FP8 operations to the hardware's native accelerators. These optimizations compound, making TensorRT-LLM a strong choice for large-scale serving where hardware is homogeneous and setup cost is a one-time investment.

Ollama is designed for local and single-machine deployment rather than distributed serving at scale. It provides an easy-to-use interface for running open-source models locally, handling model downloads, quantization, and serving configuration automatically. Ollama uses llama.cpp as its backend, which supports CPU inference in addition to GPU acceleration and implements highly optimized quantization formats like GGUF.

For development and small-scale deployment, Ollama's simplicity is its primary advantage. A developer can run ollama run llama3 to download and serve a model in a single command. There is no YAML configuration file, no GPU driver management, and no serving framework to configure. For production serving at scale, dedicated serving frameworks like vLLM or TGI offer better throughput and configuration options.

Ollama's CPU support suits certain workloads. A quantized 7B model in 4-bit GGUF format fits in roughly 4 GB of RAM and can generate tokens at a few tokens per second on a modern CPU. This is too slow for interactive use but sufficient for batch processing tasks where cost matters more than latency and GPU hardware is unavailable.

No single framework is best for every situation. The right choice depends on your hardware, team capabilities, traffic patterns, and latency requirements.

vLLM is the default starting point for most teams because it combines strong throughput, straightforward configuration, and an OpenAI-compatible API. It handles the widest range of use cases adequately, even if it is not optimal for any single specific workload.

TGI is the natural choice for teams already using the Hugging Face ecosystem. If your model is on the Hugging Face hub and your inference pipeline uses Transformers, TGI adds the minimum possible friction.

TensorRT-LLM is worth the setup cost for teams running at very large scale on standardized NVIDIA hardware. The throughput advantage compounds at scale: if you serve billions of tokens per day, a 20% throughput improvement translates directly into 20% lower infrastructure costs.

Ollama serves development and small-scale deployment where ease of use is more important than performance. It is an excellent tool for testing models locally before deciding which production framework to use.