Few-shot Learning with Retrieval Augmented Language Model

Classés selon la proximité de leur titre et de leur résumé avec ceux de cet article. Sélectionnez-en pour les lire ensemble.

Les articles les plus proches

1. Très proche Cite cet article Pas pertinent

   Self-Knowledge Guided Retrieval Augmentation for Large Language Models

   Yile Wang, Peng Li, Maosong Sun et al. · 2023 · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · cité 62 fois · Pas encore résumé

   Résumé scientifique

   Large language models (LLMs) have shown superior performance without task-specific finetuning.Despite the success, the knowledge stored in the parameters of LLMs could still be incomplete and difficult to update due to the computational costs.As complementary, retrieval-based methods can offer nonparametric world knowledge and improve the performance on tasks such as question answering.However, we find that the retrieved knowledge does not always help and even has a negative impact on original responses occasionally.To better make use of both internal knowledge and external world knowledge, we investigate eliciting the model's ability to recognize what they know and do not know (which is als…

2. Proche Pas pertinent

   Investigating the Factual Knowledge Boundary of Large Language Models with Retrieval Augmentation

   Ruiyang Ren, Yuhao Wang, Yingqi Qu et al. · 2023 · Computation and Language · cité 213 fois · Résumé IA disponible

   Résumé scientifique

   Knowledge-intensive tasks (e.g., open-domain question answering (QA)) require a substantial amount of factual knowledge and often rely on external information for assistance. Recently, large language models (LLMs) (e.g., ChatGPT), have demonstrated impressive prowess in solving a wide range of tasks with world knowledge, including knowledge-intensive tasks. However, it remains unclear how well LLMs are able to perceive their factual knowledge boundaries, particularly how they behave when incorporating retrieval augmentation. In this study, we present an initial analysis of the factual knowledge boundaries of LLMs and how retrieval augmentation affects LLMs on open-domain QA. Specially, we fo…

3. Proche Cité par cet article Pas pertinent

   REALM: Retrieval-Augmented Language Model Pre-Training

   Kelvin Guu, Kenton Lee, Zora Tung et al. · 2020 · International Conference on Machine Learning · cité 3 595 fois · Pas encore résumé

   Résumé scientifique

   Language model pre-training has been shown to capture a surprising amount of world knowledge, crucial for NLP tasks such as question answering. However, this knowledge is stored implicitly in the parameters of a neural network, requiring ever-larger networks to cover more facts. To capture knowledge in a more modular and interpretable way, we augment language model pre-training with a latent knowledge retriever, which allows the model to retrieve and attend over documents from a large corpus such as Wikipedia, used during pre-training, fine-tuning and inference. For the first time, we show how to pre-train such a knowledge retriever in an unsupervised manner, using masked language modeling a…

4. Proche Pas pertinent

   Augmentation-Adapted Retriever Improves Generalization of Language Models as Generic Plug-In

   Zichun Yu, Chenyan Xiong, Shi Yu et al. · 2023 · Computation and Language · cité 107 fois · Pas encore résumé

   Résumé scientifique

   Retrieval augmentation can aid language models (LMs) in knowledge-intensive tasks by supplying them with external information. Prior works on retrieval augmentation usually jointly fine-tune the retriever and the LM, making them closely coupled. In this paper, we explore the scheme of generic retrieval plug-in: the retriever is to assist target LMs that may not be known beforehand or are unable to be fine-tuned together. To retrieve useful documents for unseen target LMs, we propose augmentation-adapted retriever (AAR), which learns LM's preferences obtained from a known source LM. Experiments on the MMLU and PopQA datasets demonstrate that our AAR trained with a small source LM is able to s…

5. Proche Pas pertinent

   Knowledge-Augmented Language Model Prompting for Zero-Shot Knowledge Graph Question Answering

   Jinheon Baek, Alham Fikri Aji, Amir Saffari · 2023 · MATCHING · cité 197 fois · Pas encore résumé

   Résumé scientifique

   Large Language Models (LLMs) are capable of performing zero-shot closed-book question answering tasks, based on their internal knowledge stored in parameters during pre-training. However, such internalized knowledge might be insufficient and incorrect, which could lead LLMs to generate factually wrong answers. Furthermore, fine-tuning LLMs to update their knowledge is expensive. To this end, we propose to augment the knowledge directly in the input of LLMs. Specifically, we first retrieve the relevant facts to the input question from the knowledge graph based on semantic similarities between the question and its associated facts. After that, we prepend the retrieved facts to the input questi…

6. Proche Cite cet article Pas pertinent

   Knowledge-Augmented Language Model Prompting for Zero-Shot Knowledge Graph Question Answering

   Jinheon Baek, Alham Fikri Aji, Amir Saffari · 2023 · NLRSE · cité 229 fois · Pas encore résumé

   Résumé scientifique

   Large Language Models (LLMs) are capable of performing zero-shot closed-book question answering tasks, based on their internal knowledge stored in parameters during pre-training. However, such internalized knowledge might be insufficient and incorrect, which could lead LLMs to generate factually wrong answers. Furthermore, fine-tuning LLMs to update their knowledge is expensive. To this end, we propose to augment the knowledge directly in the input of LLMs. Specifically, we first retrieve the relevant facts to the input question from the knowledge graph based on semantic similarities between the question and its associated facts. After that, we prepend the retrieved facts to the input questi…

7. Proche Cite cet article Pas pertinent

   Knowledge-Augmented Language Model Prompting for Zero-Shot Knowledge Graph Question Answering

   Jinheon Baek, Alham F. Aji, Amir Saffari · 2023 · cité 51 fois · Pas encore résumé

   Résumé scientifique

   Large Language Models (LLMs) are capable of performing zero-shot closed-book question answering tasks, based on their internal knowledge stored in parameters during pre-training.However, such internalized knowledge might be insufficient and incorrect, which could lead LLMs to generate factually wrong answers.Furthermore, fine-tuning LLMs to update their knowledge is expensive.To this end, we propose to augment the knowledge directly in the input of LLMs.Specifically, we first retrieve the relevant facts to the input question from the knowledge graph based on semantic similarities between the question and its associated facts.After that, we prepend the retrieved facts to the input question in…

8. Proche Pas pertinent

   Fine-Tuning or Retrieval? Comparing Knowledge Injection in LLMs

   Oded Ovadia, Menachem Brief, Moshik Mishaeli et al. · 2023 · Artificial Intelligence · cité 334 fois · Résumé IA disponible

   Résumé scientifique

   Large language models (LLMs) encapsulate a vast amount of factual information within their pre-trained weights, as evidenced by their ability to answer diverse questions across different domains. However, this knowledge is inherently limited, relying heavily on the characteristics of the training data. Consequently, using external datasets to incorporate new information or refine the capabilities of LLMs on previously seen information poses a significant challenge. In this study, we compare two common approaches: fine-tuning and retrieval-augmented generation (RAG). We evaluate both approaches on a variety of knowledge-intensive tasks across different topics. Our findings reveal that while f…

Les travaux antérieurs sur lesquels il s’appuie

1. Proche Utilisé en profondeur par cet article Une méthode qu’il utilise Pas pertinent

   Improving language models by retrieving from trillions of tokens

   Sebastian Borgeaud, Arthur Mensch, Jordan Hoffmann et al. · 2021 · Computation and Language · cité 2 014 fois · Résumé IA disponible

   Résumé scientifique

   We enhance auto-regressive language models by conditioning on document chunks retrieved from a large corpus, based on local similarity with preceding tokens. With a $2$ trillion token database, our Retrieval-Enhanced Transformer (RETRO) obtains comparable performance to GPT-3 and Jurassic-1 on the Pile, despite using 25$\times$ fewer parameters. After fine-tuning, RETRO performance translates to downstream knowledge-intensive tasks such as question answering. RETRO combines a frozen Bert retriever, a differentiable encoder and a chunked cross-attention mechanism to predict tokens based on an order of magnitude more data than what is typically consumed during training. We typically train RETR…

2. Proche Utilisé en profondeur par cet article Une méthode qu’il utilise Pas pertinent

   Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks

   Patrick Lewis, Ethan Perez, Aleksandra Piktus et al. · 2020 · Computation and Language · cité 20 462 fois · Résumé IA disponible

   Résumé scientifique

   Large pre-trained language models have been shown to store factual knowledge in their parameters, and achieve state-of-the-art results when fine-tuned on downstream NLP tasks. However, their ability to access and precisely manipulate knowledge is still limited, and hence on knowledge-intensive tasks, their performance lags behind task-specific architectures. Additionally, providing provenance for their decisions and updating their world knowledge remain open research problems. Pre-trained models with a differentiable access mechanism to explicit non-parametric memory can overcome this issue, but have so far been only investigated for extractive downstream tasks. We explore a general-purpose …

3. Apparenté Utilisé en profondeur par cet article Une méthode qu’il utilise Pas pertinent

   Leveraging Passage Retrieval with Generative Models for Open Domain Question Answering

   Gautier Izacard, E. Grave · 2020 · Conference of the European Chapter of the Association for Computational Linguistics · cité 2 206 fois · Résumé IA disponible

   Résumé scientifique

   Generative models for open domain question answering have proven to be competitive, without resorting to external knowledge. While promising, this approach requires to use models with billions of parameters, which are expensive to train and query. In this paper, we investigate how much these models can benefit from retrieving text passages, potentially containing evidence. We obtain state-of-the-art results on the Natural Questions and TriviaQA open benchmarks. Interestingly, we observe that the performance of this method significantly improves when increasing the number of retrieved passages. This is evidence that sequence-to-sequence models offers a flexible framework to efficiently aggreg…

4. Apparenté Utilisé en profondeur par cet article Une méthode qu’il utilise Pas pertinent

   PaLM: Scaling Language Modeling with Pathways

   A. Chowdhery, Sharan Narang, J. Devlin et al. · 2022 · Journal of machine learning research · cité 8 570 fois · Résumé IA disponible

   Résumé scientifique

   Large language models have been shown to achieve remarkable performance across a variety of natural language tasks using few-shot learning, which drastically reduces the number of task-specific training examples needed to adapt the model to a particular application. To further our understanding of the impact of scale on few-shot learning, we trained a 540-billion parameter, densely activated, Transformer language model, which we call Pathways Language Model PaLM. We trained PaLM on 6144 TPU v4 chips using Pathways, a new ML system which enables highly efficient training across multiple TPU Pods. We demonstrate continued benefits of scaling by achieving state-of-the-art few-shot learning resu…

5. Apparenté Utilisé en profondeur par cet article Une méthode qu’il utilise Pas pertinent

   Distilling Knowledge from Reader to Retriever for Question Answering

   Gautier Izacard, E. Grave · 2020 · International Conference on Learning Representations · cité 366 fois · Pas encore résumé

   Résumé scientifique

   The task of information retrieval is an important component of many natural language processing systems, such as open domain question answering. While traditional methods were based on hand-crafted features, continuous representations based on neural networks recently obtained competitive results. A challenge of using such methods is to obtain supervised data to train the retriever model, corresponding to pairs of query and support documents. In this paper, we propose a technique to learn retriever models for downstream tasks, inspired by knowledge distillation, and which does not require annotated pairs of query and documents. Our approach leverages attention scores of a reader model, used …

Les articles qui le citent

1. Proche Pas pertinent

   In-Context Retrieval-Augmented Language Models

   Ori Ram, Yoav Levine, Itay Dalmedigos et al. · 2023 · Computation and Language · cité 449 fois · Résumé IA disponible

   Résumé scientifique

   Retrieval-Augmented Language Modeling (RALM) methods, that condition a language model (LM) on relevant documents from a grounding corpus during generation, have been shown to significantly improve language modeling while also providing a natural source attribution mechanism. Existing RALM approaches focus on modifying the LM architecture in order to facilitate the incorporation of external information, significantly complicating deployment. This paper proposes an under-explored alternative, which we dub In-Context RALM: leaving the LM architecture unchanged and prepending grounding documents to the input. We show that in-context RALM which uses off-the-shelf general purpose retrievers provid…

2. Proche Pas pertinent

   Benchmarking Large Language Models in Retrieval-Augmented Generation

   Jiawei Chen, Hongyu Lin, Xianpei Han et al. · 2023 · Computation and Language · cité 455 fois · Résumé IA disponible

   Résumé scientifique

   Retrieval-Augmented Generation (RAG) is a promising approach for mitigating the hallucination of large language models (LLMs). However, existing research lacks rigorous evaluation of the impact of retrieval-augmented generation on different large language models, which make it challenging to identify the potential bottlenecks in the capabilities of RAG for different LLMs. In this paper, we systematically investigate the impact of Retrieval-Augmented Generation on large language models. We analyze the performance of different large language models in 4 fundamental abilities required for RAG, including noise robustness, negative rejection, information integration, and counterfactual robustness…

3. Proche Utilise cet article en profondeur Pas pertinent

   Retrieval‐Augmented Generation for Large Language Models: Evolution, Architectures, Applications, and Challenges (2020–2025)

   Arman Rasool Faridi, Faraz Masood, Keshvi et al. · 2026 · WIREs Data Mining and Knowledge Discovery · cité 1 fois · Pas encore résumé

   Résumé scientifique

   Large language models (LLMs) generate fluent text but remain vulnerable to hallucinations, outdated knowledge, weak provenance, and costly model updates in knowledge‐intensive settings. Retrieval‐augmented generation (RAG) addresses these limitations by grounding generation in external evidence retrieved at inference time. This Advanced Review presents a PRISMA‐guided synthesis of 91 studies published between January 2020 and December 2025. It traces the evolution of RAG from early retrieve‐then‐generate pipelines to a broader design space that includes sparse, dense, and hybrid retrieval; multi‐stage reranking; evidence fusion; retrieval‐aware pre‐training; non‐parametric memory; iterative …

4. Proche Pas pertinent

   QA-RAG: Exploring LLM Reliance on External Knowledge

   Aigerim Mansurova, Aiganym Mansurova, Aliya Nugumanova · 2024 · Big Data and Cognitive Computing · cité 61 fois · Pas encore résumé

   Résumé scientifique

   Large language models (LLMs) can store factual knowledge within their parameters and have achieved superior results in question-answering tasks. However, challenges persist in providing provenance for their decisions and keeping their knowledge up to date. Some approaches aim to address these challenges by combining external knowledge with parametric memory. In contrast, our proposed QA-RAG solution relies solely on the data stored within an external knowledge base, specifically a dense vector index database. In this paper, we compare RAG configurations using two LLMs—Llama 2b and 13b—systematically examining their performance in three key RAG capabilities: noise robustness, knowledge gap de…

5. Proche Pas pertinent

   Micro-Macro Retrieval: Reducing Long-Form Hallucination in Large Language Models

   Yu-Jie Feng, Jian Li, Zhi-Hang Zhou et al. · 2026 · arXiv.org · preprint, non relu par les pairs · Pas encore résumé

   Résumé scientifique

   Large Language Models (LLMs) achieve impressive performance across many tasks but remain prone to hallucination, especially in long-form generation where redundant retrieved contexts and lengthy reasoning chains amplify factual errors. Recent studies highlight a critical phenomenon: the closer key information appears to the model outputs, the higher the factual accuracy. However, existing retrieval-augmented language models (RALMs) lack effective mechanisms to ensure this proximity - external evidence is injected into reasoning via multi-turn retrieval, but this cannot ensure key information stays close to the outputs. We propose Micro-Macro Retrieval (M2R), a novel retrieve-while-generate f…