RAFT: Adapting Language Model to Domain Specific RAG

Ranked by how close their title and abstract are to this paper’s. Select some to read them together.

Closest papers

1. Very close Cites this paper Not relevant

   Efficient In-Domain Question Answering for Resource-Constrained Environments

   Isaac Chung, Phat T. Vo, Arman Kizilkale et al. · 2024 · arXiv.org · preprint, not peer reviewed · cited 1 time · Not summarized yet

   Abstract

   Retrieval Augmented Generation (RAG) is a common method for integrating external knowledge into pretrained Large Language Models (LLMs) to enhance accuracy and relevancy in question answering (QA) tasks. However, prompt engineering and resource efficiency remain significant bottlenecks in developing optimal and robust RAG solutions for real-world QA applications. Recent studies have shown success in using fine tuning to address these problems; in particular, Retrieval Augmented Fine Tuning (RAFT) applied to smaller 7B models has demonstrated superior performance compared to RAG setups with much larger models such as GPT-3.5. The combination of RAFT with parameter-efficient fine tuning (PEFT)…

2. Very close Cites this paper Uses this paper in depth Not relevant

   Fine-Tuning Pre-Trained Language Models for Improved Retrieval in RAG Systems for Domain-Specific Use

   Syed Arham Akheel · 2024 · International Journal For Multidisciplinary Research · cited 1 time · Not summarized yet

   Abstract

   Large Language Models (LLMs) have significantly advanced natural language understanding and generation capabilities, but domain-specific applications often necessitate supplementation with current, external information to mitigate knowledge gaps and reduce hallucinations. Retrieval-Augmented Generation (RAG) has emerged as an effective solution, dynamically integrating up-to-date information through retrieval mechanisms. Fine-tuning pre-trained LLMs with domain-specific data to optimize retrieval queries has become an essential strategy to enhance RAG systems, especially in ensuring the retrieval of highly relevant information from vector databases for response generation. This paper provide…

3. Very close Cites this paper Not relevant

   Enhancing Code Transformation in Large Language Models Through Retrieval-Augmented Fine-Tuning

   Jing-Ming Guo, Po-Yang Liu, Yi-Chong Zeng et al. · 2025 · IEEE transactions on consumer electronics · cited 2 times · Not summarized yet

   Abstract

   Large language models (LLMs) have made substantial advancements in knowledge reasoning and are increasingly utilized in specialized domains such as code completion, legal analysis, and medical transcription, where accuracy is paramount. In such applications, document-specific precision is more critical than general reasoning capabilities. This paper proposes a novel approach based on Retrieval-Augmented Fine-Tuning (RAFT) to enhance model-generated outputs, particularly in code transformation tasks. RAFT integrates domain-specific knowledge, optimizing in-domain retrieval-augmented generation by training the model to discern the relationship between prompts, retrieved documents, and target o…

4. Very close Cites this paper Uses this paper in depth Not relevant

   SimRAG: Self-Improving Retrieval-Augmented Generation for Adapting Large Language Models to Specialized Domains

   Ran Xu, Hui Liu, Sreyashi Nag et al. · 2024 · Computation and Language · cited 56 times · AI summary available

   Abstract

   Retrieval-augmented generation (RAG) enhances the question-answering (QA) abilities of large language models (LLMs) by integrating external knowledge. However, adapting general-purpose RAG systems to specialized fields such as science and medicine poses unique challenges due to distribution shifts and limited access to domain-specific data. To tackle this, we propose SimRAG, a self-training approach that equips the LLM with joint capabilities of question answering and question generation for domain adaptation. Our method first fine-tunes the LLM on instruction-following, question-answering, and search-related data. Then, it prompts the same LLM to generate diverse domain-relevant questions f…

5. Very close Cites this paper Not relevant

   TriageRAG: Failure-Aware Self-Training for Adapting Retrieval-Augmented Language Models to Specialized Domains

   Samarth Varshney, R. Yadav · 2026 · Not summarized yet

   Abstract

   Retrieval-augmented generation (RAG) lets large language models (LLMs) answer knowledge-intensive questions from retrieved evidence, but adapting a general-purpose RAG system to a specialized domain such as medicine is difficult when only an unlabeled corpus is available. Self-training approaches such as SimRAG address this by letting the LLM write its own question-answer pairs from domain passages and fine-tuning on the pairs that survive a round-trip consistency filter. We observe that these approaches treat every retained example identically: they never ask whether the student already knows the answer, already extracts it correctly from the retrieved context, or fails - and if so, whether…

6. Very close Not relevant

   Enhancing Large Language Models for Specialized Domains: A Two-Stage Framework with Parameter-Sensitive LoRA Fine-Tuning and Chain-of-Thought RAG

   Yao He, Xuan-Bing Zhu, Dong-Han Li et al. · 2025 · Electronics · cited 15 times · Not summarized yet

   Abstract

   Large language models (LLMs) have shown impressive general-purpose language capabilities, but their application in specialized domains such as healthcare and law remains limited due to two major challenges, namely, a lack of deep domain-specific knowledge and the inability to incorporate real-time information updates. This paper focuses on addressing these challenges by introducing parameter-sensitive low-rank adaptation (LoRA) and retrieval-augmented generation (RAG), named SensiLoRA-RAG, a two-stage framework designed to enhance LLM performance in domain-specific question-answering tasks. In the first stage, we propose a parameter-sensitive LoRA fine-tuning method that efficiently adapts L…

7. Very close Cites this paper Not relevant

   Systematic Knowledge Injection into Large Language Models via Diverse Augmentation for Domain-Specific RAG

   Kushagra Bhushan, Yatin Nandwani, Dinesh Khandelwal et al. · 2025 · North American Chapter of the Association for Computational Linguistics · cited 16 times · Not summarized yet

   Abstract

   Retrieval-Augmented Generation (RAG) has emerged as a prominent method for incorporating domain knowledge into Large Language Models (LLMs). While RAG enhances response relevance by incorporating retrieved domain knowledge in the context, retrieval errors can still lead to hallucinations and incorrect answers. To recover from retriever failures, domain knowledge is injected by fine-tuning the model to generate the correct response, even in the case of retrieval errors. However, we observe that without systematic knowledge augmentation, fine-tuned LLMs may memorize new information but still fail to extract relevant domain knowledge, leading to poor performance. In this work, we present a nove…

8. Very close Cites this paper Uses this paper in depth Not relevant

   Divide-Then-Align: Honest Alignment based on the Knowledge Boundary of RAG

   Xin Sun, Jia-Nan Xie, Zhong-Qi Chen et al. · 2025 · Annual Meeting of the Association for Computational Linguistics · cited 19 times · Not summarized yet

   Abstract

   Large language models (LLMs) augmented with retrieval systems have significantly advanced natural language processing tasks by integrating external knowledge sources, enabling more accurate and contextually rich responses. To improve the robustness of such systems against noisy retrievals, Retrieval-Augmented Fine-Tuning (RAFT) has emerged as a widely adopted method. However, RAFT conditions models to generate answers even in the absence of reliable knowledge. This behavior undermines their reliability in high-stakes domains, where acknowledging uncertainty is critical. To address this issue, we propose Divide-Then-Align (DTA), a post-training approach designed to endow RAG systems with the …

Prior work it builds on

1. Close Not relevant

   Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks

   Patrick Lewis, Ethan Perez, Aleksandara Piktus et al. · 2020 · Computation and Language · cited 20,501 times · AI summary available

   Abstract

   Large pre-trained language models have been shown to store factual knowledge in their parameters, and achieve state-of-the-art results when fine-tuned on downstream NLP tasks. However, their ability to access and precisely manipulate knowledge is still limited, and hence on knowledge-intensive tasks, their performance lags behind task-specific architectures. Additionally, providing provenance for their decisions and updating their world knowledge remain open research problems. Pre-trained models with a differentiable access mechanism to explicit non-parametric memory can overcome this issue, but have so far been only investigated for extractive downstream tasks. We explore a general-purpose …

2. Close Not relevant

   Self-RAG: Learning to Retrieve, Generate, and Critique through Self-Reflection

   Akari Asai, Zeqiu Wu, Yizhong Wang et al. · 2023 · Computation and Language · cited 3,017 times · AI summary available

   Abstract

   Despite their remarkable capabilities, large language models (LLMs) often produce responses containing factual inaccuracies due to their sole reliance on the parametric knowledge they encapsulate. Retrieval-Augmented Generation (RAG), an ad hoc approach that augments LMs with retrieval of relevant knowledge, decreases such issues. However, indiscriminately retrieving and incorporating a fixed number of retrieved passages, regardless of whether retrieval is necessary, or passages are relevant, diminishes LM versatility or can lead to unhelpful response generation. We introduce a new framework called Self-Reflective Retrieval-Augmented Generation (Self-RAG) that enhances an LM's quality and fa…

3. Close A method it uses Not relevant

   Improving language models by retrieving from trillions of tokens

   Sebastian Borgeaud, Arthur Mensch, Jordan Hoffmann et al. · 2021 · International Conference on Machine Learning · cited 2,016 times · Not summarized yet

   Abstract

   We enhance auto-regressive language models by conditioning on document chunks retrieved from a large corpus, based on local similarity with preceding tokens. With a $2$ trillion token database, our Retrieval-Enhanced Transformer (RETRO) obtains comparable performance to GPT-3 and Jurassic-1 on the Pile, despite using 25$\times$ fewer parameters. After fine-tuning, RETRO performance translates to downstream knowledge-intensive tasks such as question answering. RETRO combines a frozen Bert retriever, a differentiable encoder and a chunked cross-attention mechanism to predict tokens based on an order of magnitude more data than what is typically consumed during training. We typically train RETR…

4. Close Not relevant

   RA-DIT: Retrieval-Augmented Dual Instruction Tuning

   Xi Victoria Lin, Xilun Chen, Mingda Chen et al. · 2023 · Computation and Language · cited 273 times · AI summary available

   Abstract

   Retrieval-augmented language models (RALMs) improve performance by accessing long-tail and up-to-date knowledge from external data stores, but are challenging to build. Existing approaches require either expensive retrieval-specific modifications to LM pre-training or use post-hoc integration of the data store that leads to suboptimal performance. We introduce Retrieval-Augmented Dual Instruction Tuning (RA-DIT), a lightweight fine-tuning methodology that provides a third option by retrofitting any LLM with retrieval capabilities. Our approach operates in two distinct fine-tuning steps: (1) one updates a pre-trained LM to better use retrieved information, while (2) the other updates the retr…

5. Close A method it uses Not relevant

   Gorilla: Large Language Model Connected with Massive APIs

   Shishir G. Patil, Tianjun Zhang, Xin Wang et al. · 2023 · Neural Information Processing Systems · cited 1,838 times · AI summary available

   Abstract

   Large Language Models (LLMs) have seen an impressive wave of advances recently, with models now excelling in a variety of tasks, such as mathematical reasoning and program synthesis. However, their potential to effectively use tools via API calls remains unfulfilled. This is a challenging task even for today's state-of-the-art LLMs such as GPT-4, largely due to their inability to generate accurate input arguments and their tendency to hallucinate the wrong usage of an API call. We release Gorilla, a finetuned LLaMA-based model that surpasses the performance of GPT-4 on writing API calls. When combined with a document retriever, Gorilla demonstrates a strong capability to adapt to test-time d…

Papers that cite it

1. Very close Uses this paper in depth Not relevant

   Adapting Large Language Models for Multi-Domain Retrieval-Augmented-Generation

   Alexandre Misrahi, Nadezhda Chirkova, Maxime Louis et al. · 2025 · arXiv.org · preprint, not peer reviewed · cited 3 times · Not summarized yet

   Abstract

   Retrieval-Augmented Generation (RAG) enhances LLM factuality, but multi-domain applications face challenges like lack of diverse benchmarks and poor out-of-domain generalization. The first contribution of this work is to introduce a diverse benchmark comprising a variety of question-answering tasks from 8 sources and covering 13 domains. Our second contribution consists in systematically testing out-of-domain generalization for typical RAG tuning strategies. While our findings reveal that standard fine-tuning fails to generalize effectively, we show that sequence-level distillation with teacher-generated labels improves out-of-domain performance by providing more coherent supervision. Our fi…

2. Very close Not relevant

   Long-Context LLMs Meet RAG: Overcoming Challenges for Long Inputs in RAG

   Bowen Jin, Jinsung Yoon, Jiawei Han et al. · 2024 · Computation and Language · cited 188 times · AI summary available

   Abstract

   Retrieval-augmented generation (RAG) empowers large language models (LLMs) to utilize external knowledge sources. The increasing capacity of LLMs to process longer input sequences opens up avenues for providing more retrieved information, to potentially enhance the quality of generated outputs. It is plausible to assume that a larger retrieval set would contain more relevant information (higher recall), that might result in improved performance. However, our empirical findings demonstrate that for many long-context LLMs, the quality of generated output initially improves first, but then subsequently declines as the number of retrieved passages increases. This paper investigates this phenomen…

3. Close Uses this paper in depth Not relevant

   Towards Agentic RAG with Deep Reasoning: A Survey of RAG-Reasoning Systems in LLMs

   Yangning Li, Weizhi Zhang, Yuyao Yang et al. · 2025 · Computation and Language · preprint, not peer reviewed · cited 62 times · AI summary available

   Abstract

   Retrieval-Augmented Generation (RAG) lifts the factuality of Large Language Models (LLMs) by injecting external knowledge, yet it falls short on problems that demand multi-step inference; conversely, purely reasoning-oriented approaches often hallucinate or mis-ground facts. This survey synthesizes both strands under a unified reasoning-retrieval perspective. We first map how advanced reasoning optimizes each stage of RAG (Reasoning-Enhanced RAG). Then, we show how retrieved knowledge of different type supply missing premises and expand context for complex inference (RAG-Enhanced Reasoning). Finally, we spotlight emerging Synergized RAG-Reasoning frameworks, where (agentic) LLMs iteratively …

4. Close Uses this paper in depth Not relevant

   Domain-Specific Data Generation Framework for RAG Adaptation

   Chris Xing Tian, Weihao Xie, Zhen Chen et al. · 2025 · Annual Meeting of the Association for Computational Linguistics · cited 3 times · Not summarized yet

   Abstract

   Retrieval-Augmented Generation (RAG) combines the language understanding and reasoning power of large language models (LLMs) with external retrieval to enable domain-grounded responses. Effectively adapting RAG systems to domain-specific settings requires specialized, context-rich training data beyond general-purpose question-answering. Here, we propose RAGen, a scalable and modular framework for generating domain-grounded question-answer-context (QAC) triples tailored to diverse RAG adaptation approaches. RAGen produces these QAC triples by identifying key concepts in documents, generating diverse questions guided by Bloom's Taxonomy-inspired principles, and pairing them with precise answer…

5. Close Not relevant

   Open-RAG: Enhanced Retrieval-Augmented Reasoning with Open-Source Large Language Models

   Shayekh Bin Islam, Md Asib Rahman, K. Hossain et al. · 2024 · Conference on Empirical Methods in Natural Language Processing · cited 62 times · Not summarized yet

   Abstract

   Retrieval-Augmented Generation (RAG) has been shown to enhance the factual accuracy of Large Language Models (LLMs), but existing methods often suffer from limited reasoning capabilities in effectively using the retrieved evidence, particularly when using open-source LLMs. To mitigate this gap, we introduce a novel framework, Open-RAG, designed to enhance reasoning capabilities in RAG with open-source LLMs. Our framework transforms an arbitrary dense LLM into a parameter-efficient sparse mixture of experts (MoE) model capable of handling complex reasoning tasks, including both single- and multi-hop queries. Open-RAG uniquely trains the model to navigate challenging distractors that appear re…