# Fine-Tuning or Retrieval? Comparing Knowledge Injection in LLMs

**Authors:** Oded Ovadia, Menachem Brief, Moshik Mishaeli, Oren Elisha  
**arXiv:** [2312.05934](https://arxiv.org/abs/2312.05934)  
**Date:** 2023-12-10  
**Summary page:** https://www.summarizepaper.com/en/paper/2312.05934v1/

## Key points

- Comparison of fine-tuning and retrieval-augmented generation (RAG) approaches for large language models (LLMs)
- LLMs encapsulate factual information but are limited by training data
- RAG consistently outperforms fine-tuning for existing and new knowledge
- Exposing LLMs to variations of facts during training improves learning of new information
- Task creation and paraphrase generation using GPT-4 for current events and paraphrasing tasks
- Study suggests LLMs can effectively learn new facts through exposure to variations during training instead of relying solely on fine-tuning methods.

## Summary

In this study, the authors compare two common approaches, fine-tuning and retrieval-augmented generation (RAG), to incorporate new information or refine the capabilities of large language models (LLMs). LLMs are known to encapsulate a vast amount of factual information within their pre-trained weights, allowing them to answer diverse questions across different domains. However, this knowledge is limited and relies heavily on the characteristics of the training data. The researchers evaluate both fine-tuning and RAG on various knowledge-intensive tasks across different topics. Their findings reveal that while fine-tuning offers some improvement, RAG consistently outperforms it for both existing knowledge encountered during training and entirely new knowledge. This suggests that LLMs struggle to learn new factual information through fine-tuning alone. To investigate further, the authors explore exposing LLMs to numerous variations of the same fact during training. They find that this approach could alleviate the problem of LLMs struggling to learn new factual information through fine-tuning. Additionally, the study includes details about task creation and paraphrase generation using GPT-4. For current events task creation, relevant chunks from Wikipedia were collected and used as input for GPT-4 to generate highly specific multiple-choice questions with one correct answer. A total of 910 new questions were created through this process. For paraphrase generation, GPT-4 was utilized to provide reworded versions of input data while retaining the information. Different seeds were used for each paraphrasing iteration to ensure variety. 240 chunks were randomly selected for each task, resulting in two paraphrases per chunk. Overall, this expanded summary provides more context on how the study was conducted and highlights key findings related to fine-tuning, retrieval augmented generation (RAG), and exposing LLMs to variations of facts during training which suggest that they can effectively learn new factual information when exposed to numerous variations during training rather than relying solely on fine tuning methods alone.