# RAG Foundry: A Framework for Enhancing LLMs for Retrieval Augmented Generation

**Authors:** Daniel Fleischer, Moshe Berchansky, Moshe Wasserblat, Peter Izsak  
**arXiv:** [2408.02545](https://arxiv.org/abs/2408.02545)  
**Date:** 2024-08-05  
**Summary page:** https://www.summarizepaper.com/en/paper/2408.02545v1/

## Key points

- Large Language Models (LLMs) have shown remarkable capabilities in various tasks traditionally requiring human intelligence.
- LLMs can produce incorrect answers and struggle with factual accuracy due to lack of access to up-to-date information.
- Retrieval-Augmented Generation (RAG) systems aim to integrate external information using retrieval mechanisms to enhance LLM performance.
- Implementing RAG systems involves key design decisions such as text embedding, indexing parameters, retrieval algorithms, query building, and prompt design.
- Reproducibility is a challenge due to variations in training data and model configurations affecting performance consistency.
- RAG Foundry is an open-source python framework supporting the development of sophisticated retrieval-augmented LLMs by providing tools for data selection, aggregation, filtering, retrieval mechanisms, text processing, document ranking, few-shot generation, prompt design using templates, fine-tuning models for specific tasks, inference processes and evaluation metrics.
- RAG Foundry functions as an end-to-end experimentation environment with modules for data creation, training, inference, and evaluation controlled by configuration files for compatibility across different stages of the workflow.
- By leveraging RAG Foundry's capabilities on knowledge-intensive datasets like Llama-3 and Phi-3 models, consistent improvements in performance are demonstrated.
- The framework enables easy dataset generation from internal or specialized knowledge sources for training large language models in RAG settings.

## Summary

In the rapidly evolving field of artificial intelligence, Large Language Models (LLMs) have demonstrated remarkable capabilities in performing a wide range of tasks that traditionally required human intelligence. However, these models are not without limitations. They can produce incorrect or nonsensical answers and struggle with factual accuracy due to their lack of access to up-to-date information. To address these limitations, Retrieval-Augmented Generation (RAG) systems aim to integrate external information using retrieval mechanisms and enhance the performance of LLMs. Implementing RAG systems is a complex process that requires a deep understanding of data, use cases, and intricate design decisions. Key design decisions include text embedding, indexing parameters, retrieval algorithms, query building, and prompt design. Reproducibility is also a challenge in this domain as variations in training data and model configurations can lead to discrepancies in performance. To facilitate the development of sophisticated retrieval-augmented LLMs for RAG use cases, we introduce RAG Foundry - an open-source python framework. This framework supports researchers and practitioners in enhancing the capabilities of LLMs by providing tools for data selection, aggregation and filtering, retrieval mechanisms, text processing, document ranking, few-shot generation, prompt design using templates, fine-tuning models for specific tasks, inference processes and evaluation metrics. RAG Foundry is designed to function as an end-to-end experimentation environment with four distinct modules: data creation,
training,
inference,
and evaluation. Each module is controlled by a configuration file to ensure compatibility between different stages of the workflow. This modular approach allows for rapid prototyping and experimentation with various RAG techniques while maintaining consistency across different datasets and tasks. By leveraging RAG Foundry's capabilities to augment and fine-tune LLMs with diverse configurations on knowledge-intensive datasets like Llama-3 and Phi-3 models, consistent improvements in performance are shown. The framework enables users to easily generate datasets from internal or specialized knowledge sources for training large language models in RAG settings. The code for RAG Foundry is available as open-source on GitHub (https://github.com/IntelLabs/RAGFoundry), providing a valuable resource for researchers looking to advance the field of retrieval-augmented generation systems.