PEFT Motivation › Multi-Task Deployment Challenges

Beyond storage costs, serving multiple fine-tuned models creates operational complexity that impacts latency and throughput while raising infrastructure costs. Even if you can afford to train eight specialized models, deploying them in a way that provides low-latency, high-availability service to users introduces a second layer of engineering difficulty.

The core tension is between two competing pressures. On one side, users expect fast responses: interactive applications require responses in under a second, and even batch processing tasks benefit from high throughput. On the other side, large models demand significant hardware resources that cost money to provision continuously. Balancing these pressures becomes substantially harder when you need to serve multiple models rather than one.

When deploying multiple task-specific models, you face architectural decisions that trade off between resource usage and response latency:

The two dominant approaches to multi-model serving each carry serious drawbacks. Dedicated instances assign one GPU cluster per task-specific model, guaranteeing consistent low latency since models are always loaded and ready. However, most applications experience variable traffic, and a model that serves 1,000 requests per hour at peak but only 10 per hour overnight wastes its dedicated GPU resources for most of its operational life. At eight tasks, you need eight GPU clusters running continuously, with costs that scale linearly regardless of actual utilization.

Shared infrastructure with model swapping loads models on-demand, improving hardware utilization during variable traffic patterns. The problem is that loading a 130 GB model from disk or system RAM into GPU memory is not instantaneous. For large models, this transfer takes 30-60 seconds, which is completely unacceptable latency for interactive applications. You could mitigate this by keeping models in system RAM for faster transfer, but system RAM has its own capacity limits and costs. Neither approach handles multi-task deployment gracefully when each task requires a complete model copy.

Modern GPU clusters typically provision 8-16 GPUs per node, with each GPU containing 40-80 GB of memory. The following visualization shows how quickly this fills up as we add task-specific models: