Prompt Tuning › Key Parameters
Michael BrenndoerferDecember 8, 20251 min read
Language AI HandbookAll chapters Part 35 of 67Parameter-Efficient Fine-tuning
Prompt Tuning › Key ParametersOne section · Sections in this chapter10 / 13 ↓
Sections in this chapter13
Reading tools Definitions & inline links
Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.
Article links
Make inline references clickable
Effective prompt tuning requires understanding the small set of design choices that control its behavior:
- num_prompt_tokens: The number of soft prompt tokens prepended to each input. This is the primary hyperparameter balancing expressiveness against context consumption. Values of 10-100 work well for most tasks; start with 20 and adjust based on task complexity and input lengths.
- init_from_vocab: Whether to initialize prompts from the model's existing vocabulary embeddings. Use vocabulary or class-label initialization by default, falling back to random initialization only when no vocabulary access is available.
- embedding_dim: The dimension of the prompt embeddings, which must match the model's hidden size. This is determined by the pretrained model and is not a tunable hyperparameter.
- learning_rate: Soft prompt embeddings often benefit from a higher learning rate than full fine-tuning would use, since fewer parameters receive gradient updates and each update must accomplish more. Learning rates in the range to often work well, compared to the to typical for fine-tuning.