# SimRAG: Self-Improving Retrieval-Augmented Generation for Adapting Large Language Models to Specialized Domains

**Authors:** Ran Xu, Hui Liu, Sreyashi Nag, Zhenwei Dai, Yaochen Xie, Xianfeng Tang, Chen Luo, Yang Li, Joyce C. Ho, Carl Yang, Qi He  
**arXiv:** [2410.17952](https://arxiv.org/abs/2410.17952)  
**Date:** 2024-10-23  
**Summary page:** https://www.summarizepaper.com/en/paper/2410.17952v1/

## Key points

- Retrieval-augmented generation (RAG) enhances capabilities of large language models (LLMs) by incorporating external knowledge
- Adapting general-purpose RAG systems to specialized fields like science and medicine poses unique challenges
- SimRAG is a self-training method that equips LLMs with question answering and generation abilities for domain adaptation
- SimRAG fine-tunes LLM on various data types, prompts it to generate domain-relevant questions, filters high-quality synthetic examples, and improves performance on domain-specific RAG tasks
- Experimental results show SimRAG outperforms existing baselines by 1.2% to 8.6%, with comparisons to off-the-shelf domain-specific LLMs and retrieval-augmented LLMs
- Case studies demonstrate the effectiveness of SimRAG in generating accurate pseudo-labeled QA pairs for tasks like claim verification in textbooks and short-span QA in medical subsets from Wikipedia

## Summary

, , , , 

In the realm of question-answering (QA) tasks, retrieval-augmented generation (RAG) has proven to enhance the capabilities of large language models (LLMs) by incorporating external knowledge. However, adapting these general-purpose RAG systems to specialized fields like science and medicine poses unique challenges due to distribution shifts and limited access to domain-specific data. To address this issue, a novel approach called SimRAG has been proposed. SimRAG is a self-training method that equips LLMs with the joint abilities of both question answering and question generation for effective domain adaptation. The process begins with fine-tuning the LLM on various types of data related to instruction-following, question-answering, and search tasks. Subsequently, the same LLM is prompted to generate a diverse set of domain-relevant questions from unlabeled corpora. A filtering strategy is then applied to retain high-quality synthetic examples generated by the model. By leveraging these synthetic examples, the LLM can significantly enhance its performance on domain-specific RAG tasks. Experimental results conducted on 11 datasets across different domains and backbone sizes demonstrate that SimRAG outperforms existing baselines by a margin ranging from 1.2% to 8.6%. The study also includes comparisons with various off-the-shelf domain-specific LLMs as well as general and domain-specific retrieval-augmented LLMs in order to provide a comprehensive evaluation framework. Moreover, additional insights are provided through case studies showcasing the effectiveness of SimRAG in generating accurate pseudo-labeled QA pairs compared to baseline models like Llama3-8B-it. These case studies highlight how SimRAG excels in tasks such as claim verification in textbooks and short-span QA in medical subsets from Wikipedia. Overall, SimRAG emerges as a promising approach for enhancing the adaptability of LLMs in specialized domains through its innovative combination of question answering and generation capabilities.