In-Context Learning › Limitations of In-Context Learning

ICL has basic limitations that practitioners should understand.

The number of demonstrations is bounded by the model's context window. With 4,096 tokens, you might fit 10-20 examples depending on their length. This hard limit constrains what ICL can learn from a single prompt.

Long-context models (32K, 100K+ tokens) partially address this, but attention over very long contexts introduces its own challenges: slower inference, potential loss of focus, and quadratic memory scaling.

ICL is sensitive to aspects of the prompt that shouldn't matter:

• Formatting: Changing delimiters or spacing affects predictions
• Order: Shuffling examples changes accuracy
• Wording: Synonymous task descriptions yield different results
• Few irrelevant tokens: Adding noise degrades performance

This brittleness makes reliable deployment challenging. A prompt optimized on development data may fail unexpectedly on edge cases.

ICL provides only temporary task adaptation. Each new inference requires giving demonstrations again. The model doesn't retain any information between queries. For frequently repeated tasks, this overhead can be significant compared to a fine-tuned model that encodes the task permanently.

Some tasks are too complex to demonstrate in a few examples. Tasks requiring:

• Extensive domain knowledge
• Multi-step procedures with many steps
• Precise output formats (structured data, code)
• Consistency across a long document

often exceed what ICL can reliably achieve. Fine-tuning or more sophisticated architectures may be necessary.