Seven Failure Points When Engineering a Retrieval Augmented Generation System

Classés selon la proximité de leur titre et de leur résumé avec ceux de cet article. Sélectionnez-en pour les lire ensemble.

Les articles les plus proches

1. Très proche Même question Cite cet article Pas pertinent

   Towards Understanding Retrieval Accuracy and Prompt Quality in RAG Systems

   Shengming Zhao, Yuheng Huang, Jiayang Song et al. · 2024 · Software Engineering · cité 13 fois · Résumé IA disponible

   Cette étude explore comment les choix de conception influencent la précision de récupération et la qualité des prompts dans les systèmes RAG, tandis que l’article décrit leurs points de défaillance.

   Résumé scientifique

   Retrieval-Augmented Generation (RAG) is a pivotal technique for enhancing the capability of large language models (LLMs) and has demonstrated promising efficacy across a diverse spectrum of tasks. While LLM-driven RAG systems show superior performance, they face unique challenges in stability and reliability. Their complexity hinders developers' efforts to design, maintain, and optimize effective RAG systems. Therefore, it is crucial to understand how RAG's performance is impacted by its design. In this work, we conduct an early exploratory study toward a better understanding of the mechanism of RAG systems, covering three code datasets, three QA datasets, and two LLMs. We focus on four desi…

2. Très proche Cite cet article Pas pertinent

   Retrieval-Augmented Generation: A Comprehensive Survey of Foundations, Evolution, Applications, Limitations, and Future Directions

   Jun-Zhe Feng, M. Ariannejad, Intan Izafina Idrus et al. · 2025 · International Conference on Recent Advances and Innovations in Engineering · cité 2 fois · Pas encore résumé

   Résumé scientifique

   Retrieval-augmented generation (RAG) is one of the best methods for performing language tasks these days. RAG uses large language models with external tools for better accuracy, correctness, and contextual understanding. The review discusses key concepts in RAG, how RAG differs from previous systems that retrieved or generated content, how RAG is utilized in question answering, knowledge management, and content generation, and it discusses some limitations, including quality of retrieval, requirement for high-performance computers, complicated integration, as well as data privacy problems, and presents new approaches to address these limitations. The review examines over a quarter-century of…

3. Très proche Cite cet article Pas pertinent

   Optimizing and Evaluating Enterprise Retrieval-Augmented Generation (RAG): A Content Design Perspective

   Sarah Packowski, Inge Halilovic, Jenifer Schlotfeldt et al. · 2024 · International Conference on Advances in Artificial Intelligence · cité 20 fois · Pas encore résumé

   Résumé scientifique

   Retrieval-augmented generation (RAG) is a popular technique for using large language models (LLMs) to build customer-support, question-answering solutions. In this paper, we share our team’s practical experience building and maintaining enterprise-scale RAG solutions that answer users’ questions about our software based on product documentation. Our experience has not always matched the most common patterns in the RAG literature. This paper focuses on solution strategies that are modular and model-agnostic. For example, our experience over the past few years - using different search methods and LLMs, and many knowledge base collections - has been that simple changes to the way we create know…

4. Très proche Pas pertinent

   Mindful-RAG: A Study of Points of Failure in Retrieval Augmented Generation

   Garima Agrawal, Tharindu Kumarage, Zeyad Alghamdi et al. · 2024 · Information Retrieval · cité 28 fois · Résumé IA disponible

   Résumé scientifique

   Large Language Models (LLMs) are proficient at generating coherent and contextually relevant text but face challenges when addressing knowledge-intensive queries in domain-specific and factual question-answering tasks. Retrieval-augmented generation (RAG) systems mitigate this by incorporating external knowledge sources, such as structured knowledge graphs (KGs). However, LLMs often struggle to produce accurate answers despite access to KG-extracted information containing necessary facts. Our study investigates this dilemma by analyzing error patterns in existing KG-based RAG methods and identifying eight critical failure points. We observed that these errors predominantly occur due to insuf…

5. Très proche Pas pertinent

   Harnessing Retrieval-Augmented Generation (RAG) for Uncovering Knowledge Gaps

   Joan Figuerola Hurtado · 2023 · Information Retrieval · preprint, non relu par les pairs · cité 2 fois · Résumé IA disponible

   Résumé scientifique

   The paper presents a methodology for uncovering knowledge gaps on the internet using the Retrieval Augmented Generation (RAG) model. By simulating user search behaviour, the RAG system identifies and addresses gaps in information retrieval systems. The study demonstrates the effectiveness of the RAG system in generating relevant suggestions with a consistent accuracy of 93%. The methodology can be applied in various fields such as scientific discovery, educational enhancement, research development, market analysis, search engine optimisation, and content development. The results highlight the value of identifying and understanding knowledge gaps to guide future endeavours.

6. Très proche Cite cet article Pas pertinent

   Enhancing the Precision and Interpretability of Retrieval-Augmented Generation (RAG) in Legal Technology: A Survey

   Mahd Hindi, Linda Mohammed, Ommama Maaz et al. · 2025 · IEEE Access · cité 74 fois · Pas encore résumé

   Résumé scientifique

   Retrieval-Augmented Generation (RAG) is a promising solution that can enhance the capabilities of large language model (LLM) applications in critical domains, including legal technology, by retrieving knowledge from external databases. Implementing RAG pipelines requires careful attention to the techniques and methods implemented in the different stages of the RAG process. However, robust RAG can enhance LLM generation with faithfulness and few hallucinations in responses. In this paper, we discuss the application of RAG in the legal domain. First, we present an overview of the main RAG methods, stages, techniques, and applications in the legal domain. We then briefly discuss the different i…

7. Très proche Pas pertinent

   Searching for Best Practices in Retrieval-Augmented Generation

   Xiaohua Wang, Zhenghua Wang, Xuan Gao et al. · 2024 · Computation and Language · cité 218 fois · Résumé IA disponible

   Résumé scientifique

   Retrieval-augmented generation (RAG) techniques have proven to be effective in integrating up-to-date information, mitigating hallucinations, and enhancing response quality, particularly in specialized domains. While many RAG approaches have been proposed to enhance large language models through query-dependent retrievals, these approaches still suffer from their complex implementation and prolonged response times. Typically, a RAG workflow involves multiple processing steps, each of which can be executed in various ways. Here, we investigate existing RAG approaches and their potential combinations to identify optimal RAG practices. Through extensive experiments, we suggest several strategie…

8. Très proche Pas pertinent

   Corrective Retrieval Augmented Generation

   Shi-Qi Yan, Jia-Chen Gu, Yun Zhu et al. · 2024 · Computation and Language · preprint, non relu par les pairs · cité 345 fois · Résumé IA disponible

   Résumé scientifique

   Large language models (LLMs) inevitably exhibit hallucinations since the accuracy of generated texts cannot be secured solely by the parametric knowledge they encapsulate. Although retrieval-augmented generation (RAG) is a practicable complement to LLMs, it relies heavily on the relevance of retrieved documents, raising concerns about how the model behaves if retrieval goes wrong. To this end, we propose the Corrective Retrieval Augmented Generation (CRAG) to improve the robustness of generation. Specifically, a lightweight retrieval evaluator is designed to assess the overall quality of retrieved documents for a query, returning a confidence degree based on which different knowledge retriev…

Les travaux antérieurs sur lesquels il s’appuie

1. Proche Pas pertinent

   Improving the Domain Adaptation of Retrieval Augmented Generation (RAG) Models for Open Domain Question Answering

   Shamane Siriwardhana, Rivindu Weerasekera, Elliott Wen et al. · 2022 · Computation and Language · cité 404 fois · Résumé IA disponible

   Résumé scientifique

   Retrieval Augment Generation (RAG) is a recent advancement in Open-Domain Question Answering (ODQA). RAG has only been trained and explored with a Wikipedia-based external knowledge base and is not optimized for use in other specialized domains such as healthcare and news. In this paper, we evaluate the impact of joint training of the retriever and generator components of RAG for the task of domain adaptation in ODQA. We propose \textit{RAG-end2end}, an extension to RAG, that can adapt to a domain-specific knowledge base by updating all components of the external knowledge base during training. In addition, we introduce an auxiliary training signal to inject more domain-specific knowledge. T…

2. Proche Pas pertinent

   Benchmarking Large Language Models in Retrieval-Augmented Generation

   Jiawei Chen, Hong-Yu Lin, Xian-Pei Han et al. · 2023 · AAAI Conference on Artificial Intelligence · cité 729 fois · Résumé IA disponible

   Résumé scientifique

   Retrieval-Augmented Generation (RAG) is a promising approach for mitigating the hallucination of large language models (LLMs). However, existing research lacks rigorous evaluation of the impact of retrieval-augmented generation on different large language models, which make it challenging to identify the potential bottlenecks in the capabilities of RAG for different LLMs. In this paper, we systematically investigate the impact of Retrieval-Augmented Generation on large language models. We analyze the performance of different large language models in 4 fundamental abilities required for RAG, including noise robustness, negative rejection, information integration, and counterfactual robustness…

3. Apparenté Utilisé en profondeur par cet article Une méthode qu’il utilise Pas pertinent

   Large Language Models for Information Retrieval: A Survey

   Yutao Zhu, Huaying Yuan, Shuting Wang et al. · 2023 · ACM Trans. Inf. Syst. · cité 693 fois · Résumé IA disponible

   Résumé scientifique

   As a primary means of information acquisition, information retrieval (IR) systems, such as search engines, have integrated themselves into our daily lives. These systems also serve as components of dialogue, question-answering, and recommender systems. The trajectory of IR has evolved dynamically from its origins in term-based methods to its integration with advanced neural models. While the neural models excel at capturing complex contextual signals and semantic nuances, they still face challenges such as data scarcity, interpretability, and the generation of contextually plausible yet potentially inaccurate responses. This evolution requires a combination of traditional methods (such as te…

4. Apparenté Pas pertinent

   FiD-Light: Efficient and Effective Retrieval-Augmented Text Generation

   Sebastian Hofstätter, Jie-Cao Chen, K. Raman et al. · 2022 · Annual International ACM SIGIR Conference on Research and Development in Information Retrieval · cité 119 fois · Pas encore résumé

   Résumé scientifique

   Retrieval-augmented generation models offer many benefits over standalone language models: besides a textual answer to a given query they provide provenance items retrieved from an updateable knowledge base. However, they are also more complex systems and need to handle long inputs. In this work, we introduce FiD-Light to strongly increase the efficiency of the state-of-the-art retrieval-augmented FiD model, while maintaining the same level of effectiveness. Our FiD-Light model constrains the information flow from the encoder (which encodes passages separately) to the decoder (using concatenated encoded representations). Furthermore, we adapt FiD-Light with re-ranking capabilities through te…

5. Apparenté Pas pertinent

   Leveraging Passage Retrieval with Generative Models for Open Domain Question Answering

   Gautier Izacard, E. Grave · 2020 · Conference of the European Chapter of the Association for Computational Linguistics · cité 2 207 fois · Résumé IA disponible

   Résumé scientifique

   Generative models for open domain question answering have proven to be competitive, without resorting to external knowledge. While promising, this approach requires to use models with billions of parameters, which are expensive to train and query. In this paper, we investigate how much these models can benefit from retrieving text passages, potentially containing evidence. We obtain state-of-the-art results on the Natural Questions and TriviaQA open benchmarks. Interestingly, we observe that the performance of this method significantly improves when increasing the number of retrieved passages. This is evidence that sequence-to-sequence models offers a flexible framework to efficiently aggreg…

Les articles qui le citent

1. Très proche Pas pertinent

   Retrieval-Augmented Generation for AI-Generated Content: A Survey

   Penghao Zhao, Hailin Zhang, Qinhan Yu et al. · 2024 · Computer Vision and Pattern Recognition · cité 243 fois · Résumé IA disponible

   Résumé scientifique

   Advancements in model algorithms, the growth of foundational models, and access to high-quality datasets have propelled the evolution of Artificial Intelligence Generated Content (AIGC). Despite its notable successes, AIGC still faces hurdles such as updating knowledge, handling long-tail data, mitigating data leakage, and managing high training and inference costs. Retrieval-Augmented Generation (RAG) has recently emerged as a paradigm to address such challenges. In particular, RAG introduces the information retrieval process, which enhances the generation process by retrieving relevant objects from available data stores, leading to higher accuracy and better robustness. In this paper, we c…

2. Très proche Pas pertinent

   Engineering RAG Systems for Real-World Applications: Design, Development, and Evaluation

   M. Hasan, Muhammad Waseem, Kai-Kristian Kemell et al. · 2025 · EUROMICRO Conference on Software Engineering and Advanced Applications · cité 12 fois · Pas encore résumé

   Résumé scientifique

   Retrieval-Augmented Generation (RAG) systems are emerging as a key approach for grounding Large Language Models (LLMs) in external knowledge, addressing limitations in factual accuracy and contextual relevance. However, there is a lack of empirical studies that report on the development of RAG-based implementations grounded in real-world use cases, evaluated through general user involvement, and accompanied by systematic documentation of lessons learned. This paper presents five domain-specific RAG applications developed for real-world scenarios across governance, cybersecurity, agriculture, industrial research, and medical diagnostics. Each system incorporates multilingual OCR, semantic ret…

3. Très proche Pas pertinent

   A Methodology for Evaluating RAG Systems: A Case Study On Configuration Dependency Validation

   Sebastian Simon, Alina Mailach, Johannes Dorn et al. · 2024 · arXiv.org · preprint, non relu par les pairs · cité 16 fois · Pas encore résumé

   Résumé scientifique

   Retrieval-augmented generation (RAG) is an umbrella of different components, design decisions, and domain-specific adaptations to enhance the capabilities of large language models and counter their limitations regarding hallucination and outdated and missing knowledge. Since it is unclear which design decisions lead to a satisfactory performance, developing RAG systems is often experimental and needs to follow a systematic and sound methodology to gain sound and reliable results. However, there is currently no generally accepted methodology for RAG evaluation despite a growing interest in this technology. In this paper, we propose a first blueprint of a methodology for a sound and reliable e…

4. Très proche Pas pertinent

   Retrieval-augmented generation for educational application: A systematic survey

   Zongxi Li, Zijian Wang, Weiming Wang et al. · 2025 · Computers and Education Artificial Intelligence · cité 151 fois · Pas encore résumé

   Résumé scientifique

   Advancements in large language models (LLMs) have transformed AI-driven education, enabling innovative applications across various learning and teaching domains. However, LLMs still face several challenges, including hallucination and static internal knowledge, which hinder their reliability in educational settings. Retrieval-Augmented Generation (RAG) enhances LLMs by retrieving relevant information from an external knowledge base and incorporating it into the LLM's generation process. This approach improves factual accuracy and enables dynamic knowledge updates, making LLMs particularly suitable for educational applications. In this paper, we comprehensively review existing research that i…

5. Proche Utilise cet article en profondeur Pas pertinent

   RAGProbe: Breaking RAG Pipelines with Evaluation Scenarios

   Shangeetha Sivasothy, Scott Barnett, Stefanus Kurniawan et al. · 2025 · 2025 IEEE/ACM 4th International Conference on AI Engineering – Software Engineering for AI (CAIN) · cité 2 fois · Pas encore résumé

   Résumé scientifique

   Retrieval Augmented Generation (RAG) is increasingly employed in building Generative AI applications, yet their evaluation often relies on manual, trial-and-error processes. Automating this evaluation process involves generating test data to trigger failures involving context comprehension, data formatting, specificity, and content completeness. Random question-answer generation is insufficient. However, prior works rely on standard QA datasets, benchmarks and tactics that are not tailored to the specific domain requirements. Hence, current approaches and datasets do not trigger sufficiently broad and context-specific failures. In this paper, we introduce evaluation scenarios that describe t…