Running summarizing tools on a new article

Abstract: Retrieval-Augmented Generation (RAG) has become one of the key functionalities in current large language model (LLM) applications. It effectively mitigates hallucination issues in LLMs and provides users with the latest domain-specific information by accessing knowledge bases, serving as a costefficient method for LLM capability migration. However, users primarily interact with LLMs through application systems like OpenWebUI, where RAG functionality is typically embedded as a “black box.” This makes large-scale automated evaluation challenging for existing RAG assessment systems. To address this, this paper proposes an universal end-to-end automated evaluation framework. This framework can evaluate the RAG functionality embedded within various LLM application systems and output quantitative scores. We conducted experimental validation on widely adopted LLM application systems including OpenWebUI, AnythingLLM, and RAGFlow. The results demonstrate that the framework can effectively evaluate RAG performance across different LLM application systems and their configurations.