Adapting LLMs: Context Injection vs. Fine-Tuning › Practical Examples: Three Approaches Compared

To solidify the concepts, let's walk through a scenario with three approaches. Imagine you want an AI assistant to answer questions about your company's internal policies. You have a policy document that employees might ask about (e.g., parental leave policy). ## Scenario: "What is the parental leave policy at our company?" #### **Off-the-Shelf Model** You take a base pre-trained model (no fine-tuning, no added context). The model, not having seen your company's policy in its training data, will likely respond with a generic answer or "I'm not sure". - **Response**: "Companies typically offer 12 weeks of parental leave" - but that's just a guess. - **Issue**: The off-the-shelf model has no knowledge of your specific company, so it either produces a generic policy or hallucinates details, which could be incorrect. #### **Context Injection (RAG/Prompt Engineering)** You use the context injection approach. You retrieve the actual policy text from your internal knowledge base. For example, the policy document says "Acme Corp provides 16 weeks of paid parental leave for primary caregivers." Your system prepends this to the model prompt: "[Policy Excerpt]: Acme Corp provides 16 weeks of paid parental leave for primary caregivers... [User's question]: What is our company's parental leave policy?" - **Response**: "Our company offers 16 weeks of paid parental leave for primary caregivers." - **Benefits**: This answer is accurate and company-specific, thanks to the retrieved context. The model is effectively reading from your data and formatting the answer in natural language. If the policy changes (say to 20 weeks next year), your RAG-based assistant will pick up the new text automatically - you just update the document in the database. #### **Fine-Tuned Model** You create a fine-tuning dataset with questions like "What is the parental leave policy?" paired with the correct answer from your policy docs. You fine-tune the model on hundreds of such Q&A pairs covering all your company policies. - **Response**: "We offer 16 weeks paid leave for primary caregivers, and 8 weeks for secondary caregivers," with the exact phrasing it learned. - **Benefits**: You don't need to provide the policy text at runtime - the info is now in the model's weights. This fine-tuned model will likely respond faster (no retrieval step) and in a style consistent with your training data. - **Limitations**: If the policy changes to 20 weeks, the fine-tuned model will still say 16 weeks until you update your training data and re-train. The model won't cite a source or show how it knows the answer - it just "knows". ## Example Summary - Off-the-Shelf, our model didn't know the answer. - With context (RAG), the model gave the right answer because we fed it the info on the fly. - With fine-tuning, the model learned the answer and could respond on its own, but was locked to the info it was trained on (requiring re-training for updates). Each approach has a place: if you only have a handful of well-defined queries and they rarely change, fine-tuning can be great. If you have lots of documents and need any arbitrary question answered from them, RAG is more flexible. And if you're just exploring general capabilities or don't have custom data yet, using the base model with a bit of prompt engineering is the quickest way to get started. To illustrate the difference in outputs, consider a concrete Q&A example with a fictitious scenario: **User question**: "How do I reset my VPN password in our company network?" - **Base Model (no context)**: "I'm sorry, I don't have information on that." (It doesn't know company-specific procedures and might apologize or give a generic suggestion that could be incorrect.) - **With RAG context**: "According to our IT policy: Go to the VPN portal, click 'Forgot Password', then enter your work email. You'll receive a reset link via email to set a new password.” (Here the assistant pulled the exact steps from the internal IT document and presented them. This answer is accurate and complete, reflecting current internal procedures.) - **Fine-Tuned model**: "To reset your VPN password, go to the company's VPN login page and use the 'Forgot Password' option. Follow the emailed link to set a new password. Contact IT support if you run into issues." (This answer was generated from the model's trained knowledge. It's very similar to the RAG answer because we trained the model on that IT policy. It's correct - assuming the policy hasn't changed. The style might be a bit more streamlined since the model learned how such answers typically look. But if the process changes, this answer would not automatically update.) In practice, building a robust solution might involve a combination: you could fine-tune the model on general company Q&A and conversation style (so it knows how to answer questions politely and in-depth), and use RAG to inject specific facts that change often or weren't in the fine-tuning data. This hybrid approach is common in production systems.