Demonstrate-Search-Predict: Composing retrieval and language models for knowledge-intensive NLP
AI-generated Key Points
⚠The license of the paper does not allow us to build upon its content and the key points are generated using the paper metadata rather than the full article.
- Natural Language Processing (NLP) has seen significant advancements in recent years, particularly in retrieval-augmented in-context learning.
- Existing work has been limited to simple "retrieve-then-read" pipelines, where the retrieval model (RM) retrieves passages that are inserted into the language model (LM) prompt.
- A new framework called Demonstrate-Search-Predict (DSP) has been proposed to fully realize the potential of frozen LMs and RMs.
- The DSP framework relies on passing natural language texts through sophisticated pipelines between an LM and an RM to express high-level programs that bootstrap pipeline-aware demonstrations, search for relevant passages, and generate grounded predictions by breaking down problems into small transformations that the LM and RM can handle more reliably.
- The authors have written novel DSP programs for answering questions in open-domain, multi-hop, and conversational settings.
- In early evaluations, DSP has demonstrated new state-of-the-art results in context learning with relative gains against vanilla LMs ranging from 37% to 200%, a standard retrieve–then–read pipeline from 8% to 40%, and a contemporaneous self–ask pipeline from 80% to 290%.
- The authors behind this research include Omar Khattab, Keshav Santhanam, Xiang Lisa Li, David Hall, Percy Liang, Christopher Potts and Matei Zaharia.
- Overall, DSP represents a powerful advancement in NLP by allowing for more sophisticated interactions between LMs and RMs.
Authors: Omar Khattab, Keshav Santhanam, Xiang Lisa Li, David Hall, Percy Liang, Christopher Potts, Matei Zaharia
Abstract: Retrieval-augmented in-context learning has emerged as a powerful approach for addressing knowledge-intensive tasks using frozen language models (LM) and retrieval models (RM). Existing work has combined these in simple "retrieve-then-read" pipelines in which the RM retrieves passages that are inserted into the LM prompt. To begin to fully realize the potential of frozen LMs and RMs, we propose Demonstrate-Search-Predict (DSP), a framework that relies on passing natural language texts in sophisticated pipelines between an LM and an RM. DSP can express high-level programs that bootstrap pipeline-aware demonstrations, search for relevant passages, and generate grounded predictions, systematically breaking down problems into small transformations that the LM and RM can handle more reliably. We have written novel DSP programs for answering questions in open-domain, multi-hop, and conversational settings, establishing in early evaluations new state-of-the-art in-context learning results and delivering 37-200%, 8-40%, and 80-290% relative gains against vanilla LMs, a standard retrieve-then-read pipeline, and a contemporaneous self-ask pipeline, respectively.
Ask questions about this paper to our AI assistant
You can also chat with multiple papers at once here.
⚠The license of the paper does not allow us to build upon its content and the AI assistant only knows about the paper metadata rather than the full article.
Welcome to our AI assistant! Here are some important things to keep in mind:
- The assistant will only answer questions related to this specific paper.
- Please note that this is not a bot for casual chatting.
- If you want the answer in a language other than the language you chose for navigating the website, simply add "TRANSLATE IN LANGUAGE L" at the end of your query (replace "LANGUAGE L" with the language of your choice).
- For example, you could ask "Can you extract the most important aspect of the paper? TRANSLATE IN SPANISH".
- If you want to keep the history of your questions/answers you should create an account.
Assess the quality of the AI-generated content by voting
Why do we need votes?
Votes are used to determine whether we need to re-run our summarizing tools. If the count reaches -10, our tools can be restarted.
Look for similar papers (in beta version)
By clicking on the button above, our algorithm will scan all papers in our database to find the closest based on the contents of the full papers and not just on metadata. Please note that it only works for papers that we have generated summaries for and you can rerun it from time to time to get a more accurate result while our database grows.
Disclaimer: The AI-based summarization tool and virtual assistant provided on this website may not always provide accurate and complete summaries or responses. We encourage you to carefully review and evaluate the generated content to ensure its quality and relevance to your needs.