Answering Questions by Meta-Reasoning over Multiple Chains of Thought

Auteurs : Ori Yoran, Tomer Wolfson, Ben Bogin, Uri Katz, Daniel Deutch, Jonathan Berant

Soumis à arXiv le 25 Avr. 2023
Licence : CC BY 4.0
Résumé scientifique

Modern systems for multi-hop question answering (QA) typically break questions into a sequence of reasoning steps, termed chain-of-thought (CoT), before arriving at a final answer. Often, multiple chains are sampled and aggregated through a voting mechanism over the final answers, but the intermediate steps themselves are discarded. While such approaches improve performance, they do not consider the relations between intermediate steps across chains and do not provide a unified explanation for the predicted answer. We introduce Multi-Chain Reasoning (MCR), an approach which prompts large language models to meta-reason over multiple chains of thought, rather than aggregating their answers. MCR examines different reasoning chains, mixes information between them and selects the most relevant facts in generating an explanation and predicting the answer. MCR outperforms strong baselines on 7 multi-hop QA datasets. Moreover, our analysis reveals that MCR explanations exhibit high quality, enabling humans to verify its answers.

Les articles autour de lui, par date et par proximité

Plus haut, c’est plus proche de cet article ; à gauche, c’est plus ancien. Les traits relient les articles proches entre eux : les groupes de traits sont des sous-thèmes. Sélectionnez un point pour voir pourquoi il est là. Retour au résumé de cet article

Comment lire cette carte ? Guide illustré

Les articles les plus proches Les travaux antérieurs sur lesquels il s’appuie Les articles qui le citent

Les articles autour de celui-ci

Classés selon la proximité de leur titre et de leur résumé avec ceux de cet article. Sélectionnez-en pour les lire ensemble.

Les articles les plus proches

  1. Très proche Autre approche du problème
    Multi-hop Question Answering via Reasoning Chains
    Jifan Chen, Shih-Ting Lin, Greg Durrett · 2019 · arXiv.org · preprint, non relu par les pairs · cité 89 fois · Pas encore résumé

    Multi-hop Question Answering via Reasoning Chains extrait des chaînes discrètes de phrases puis les transmet à BERT, tandis que MCR combine plusieurs chaînes générées.

    Résumé scientifique

    Multi-hop question answering requires models to gather information from different parts of a text to answer a question. Most current approaches learn to address this task in an end-to-end way with neural networks, without maintaining an explicit representation of the reasoning process. We propose a method to extract a discrete reasoning chain over the text, which consists of a series of sentences leading to the answer. We then feed the extracted chains to a BERT-based QA model to do final answer prediction. Critically, we do not rely on gold annotated chains or "supporting facts:" at training time, we derive pseudogold reasoning chains using heuristics based on named entity recognition and c…

  2. Très proche Autre approche du problème
    CoQ:AN Empirical Framework for Multi-hop Question Answering Empowered by Large Language Models
    Qiang Huang, Feng Huang, De-Hao Tao et al. · 2024 · IEEE International Conference on Acoustics, Speech, and Signal Processing · cité 3 fois · Pas encore résumé

    CoQ décompose la question en sous-questions pour récupérer des connaissances externes, alors que MCR met en relation plusieurs chaînes de raisonnement.

    Résumé scientifique

    Prompt-based Large Language Models(LLMs) are surprisingly powerful in generating natural language reasoning steps or Chains of Thoughts(CoT) for multi-hop question answering(QA). However, LLMs struggle when they lack access to necessary knowledge or when the knowledge within their parameters is outdated. Additionally, LLMs that rely solely on CoT tend to generate hallucinations during the reasoning process. To address these dilemmas, we propose the Chain of Question (CoQ) framework, a novel multi-hop QA approach. This approach decomposes a complex original question into multiple sub-questions according to a CoT to retrieve knowledge from an external knowledge base. It then answers the questi…

  3. Très proche Autre approche du problème Cite cet article
    Leveraging Structured Information for Explainable Multi-hop Question Answering and Reasoning
    Ruosen Li, Xin-Ya Du · 2023 · Conference on Empirical Methods in Natural Language Processing · cité 39 fois · Pas encore résumé

    Leveraging Structured Information utilise l’extraction d’entités, de relations et d’événements pour rendre le raisonnement multi-saut interprétable, contrairement au méta-raisonnement de MCR.

    Résumé scientifique

    Neural models, including large language models (LLMs), achieve superior performance on multi-hop question-answering. To elicit reasoning capabilities from LLMs, recent works propose using the chain-of-thought (CoT) mechanism to generate both the reasoning chain and the answer, which enhances the model's capabilities in conducting multi-hop reasoning. However, several challenges still remain: such as struggling with inaccurate reasoning, hallucinations, and lack of interpretability. On the other hand, information extraction (IE) identifies entities, relations, and events grounded to the text. The extracted structured information can be easily interpreted by humans and machines (Grishman, 2019…

  4. Très proche Autre approche du problème Cité par cet article Utilisé en profondeur par cet article Une méthode qu’il utilise
    Interleaving Retrieval with Chain-of-Thought Reasoning for Knowledge-Intensive Multi-Step Questions
    H. Trivedi, Niranjan Balasubramanian, Tushar Khot et al. · 2022 · Annual Meeting of the Association for Computational Linguistics · cité 1 510 fois · Pas encore résumé

    IRCoT alterne récupération de documents et raisonnement CoT pour les questions multi-étapes, tandis que MCR combine des chaînes de raisonnement.

    Résumé scientifique

    Prompting-based large language models (LLMs) are surprisingly powerful at generating natural language reasoning steps or Chains-of-Thoughts (CoT) for multi-step question answering (QA). They struggle, however, when the necessary knowledge is either unavailable to the LLM or not up-to-date within its parameters. While using the question to retrieve relevant text from an external knowledge source helps LLMs, we observe that this one-step retrieve-and-read approach is insufficient for multi-step QA. Here, what to retrieve depends on what has already been derived, which in turn may depend on what was previously retrieved. To address this, we propose IRCoT, a new approach for multi-step QA that i…

  5. Très proche Autre approche du problème
    Self-prompted Chain-of-Thought on Large Language Models for Open-domain Multi-hop Reasoning
    Jin-Yuan Wang, Junlong Li, Hai Zhao · 2023 · Conference on Empirical Methods in Natural Language Processing · cité 42 fois · Pas encore résumé

    Self-prompted Chain-of-Thought vise le raisonnement multi-saut en domaine ouvert avec des chaînes explicites, tandis que MCR méta-raisonne sur plusieurs chaînes.

    Résumé scientifique

    In open-domain question-answering (ODQA), most existing questions require single-hop reasoning on commonsense. To further extend this task, we officially introduce open-domain multi-hop reasoning (ODMR) by answering multi-hop questions with explicit reasoning steps in open-domain setting. Recently, large language models (LLMs) have found significant utility in facilitating ODQA without external corpus. Furthermore, chain-of-thought (CoT) prompting boosts the reasoning capability of LLMs to a greater extent with manual or automated paradigms. However, existing automated methods lack of quality assurance, while manual approaches suffer from limited scalability and poor diversity, hindering the…

  6. Très proche Article de revue Cite cet article
    Towards A Unified View of Answer Calibration for Multi-Step Reasoning
    Shu-Min Deng, Ningyu Zhang, Nay Oo et al. · 2023 · NLRSE · cité 9 fois · Pas encore résumé

    Towards A Unified View of Answer Calibration classe et évalue les stratégies de calibration des réponses après génération de chemins de raisonnement.

    Résumé scientifique

    Large Language Models (LLMs) employing Chain-of-Thought (CoT) prompting have broadened the scope for improving multi-step reasoning capabilities. We generally divide multi-step reasoning into two phases: *path generation* to generate the reasoning path(s); and *answer calibration* post-processing the reasoning path(s) to obtain a final answer. However, the existing literature lacks systematic analysis on different answer calibration approaches. In this paper, we summarize the taxonomy of recent answer calibration techniques and break them down into step-level and path-level strategies. We then conduct a thorough evaluation on these strategies from a unified view, systematically scrutinizing …

  7. Très proche Autre approche du problème
    Enhancing the Completeness of Rationales for Multi-Step Question Answering
    Shangzi Xue, Zhen-Ya Huang, Xin Lin et al. · 2024 · International Conference on Information and Knowledge Management · cité 6 fois · Pas encore résumé

    Enhancing the Completeness of Rationales planifie puis évalue les justifications pour éviter les étapes omises et les erreurs factuelles en QA multi-étapes.

    Résumé scientifique

    Learning to answer multi-step complex questions requires machines to perform like a human to think and reason step by step, which is one of the core abilities of a question answering system. Recent advancements have revealed that large language models exhibit remarkable reasoning capabilities by generating intermediate chain-of-thought rationales. However, the completeness of their rationales lacks assurance as they are susceptible to omitting steps and making factual errors. In this paper, drawing inspiration from human-like reasoning processes in answering multi-step questions, we explicitly plan the rationales to ensure their completeness. We propose a two-stage Decomposition-Evaluation (…

  8. Très proche Contexte plus large
    Direct Evaluation of Chain-of-Thought in Multi-hop Reasoning with Knowledge Graphs
    Minh-Vuong Nguyen, Lin-Hao Luo, Fatemeh Shiri et al. · 2024 · Annual Meeting of the Association for Computational Linguistics · cité 63 fois · Pas encore résumé

    Direct Evaluation of Chain-of-Thought mesure la justesse des explications CoT en QA multi-saut à l’aide de graphes de connaissances, plutôt que de produire une réponse par méta-raisonnement.

    Résumé scientifique

    Large language models (LLMs) demonstrate strong reasoning abilities when prompted to generate chain-of-thought (CoT) explanations alongside answers. However, previous research on evaluating LLMs has solely focused on answer accuracy, neglecting the correctness of the generated CoT. In this paper, we delve deeper into the CoT reasoning capabilities of LLMs in multi-hop question answering by utilizing knowledge graphs (KGs). We propose a novel discriminative and generative CoT evaluation paradigm to assess LLMs' knowledge of reasoning and the accuracy of the generated CoT. Through experiments conducted on 5 different families of LLMs across 2 multi-hop question-answering datasets, we find that…

Les travaux antérieurs sur lesquels il s’appuie

  1. Proche Autre approche du problème Utilisé en profondeur par cet article Une méthode qu’il utilise
    Self-Consistency Improves Chain of Thought Reasoning in Language Models
    Xuezhi Wang, Jason Wei, Dale Schuurmans et al. · 2022 · Computation and Language · cité 8 219 fois · Résumé IA disponible

    Self-Consistency échantillonne plusieurs chemins puis choisit la réponse la plus cohérente, tandis que MCR exploite aussi les étapes intermédiaires pour construire une explication.

    Résumé scientifique

    Chain-of-thought prompting combined with pre-trained large language models has achieved encouraging results on complex reasoning tasks. In this paper, we propose a new decoding strategy, self-consistency, to replace the naive greedy decoding used in chain-of-thought prompting. It first samples a diverse set of reasoning paths instead of only taking the greedy one, and then selects the most consistent answer by marginalizing out the sampled reasoning paths. Self-consistency leverages the intuition that a complex reasoning problem typically admits multiple different ways of thinking leading to its unique correct answer. Our extensive empirical evaluation shows that self-consistency boosts the …

  2. Proche Données, outil ou méthode en commun Utilisé en profondeur par cet article Une méthode qu’il utilise
    ♫ MuSiQue: Multihop Questions via Single-hop Question Composition
    H. Trivedi, Niranjan Balasubramanian, Tushar Khot et al. · 2021 · Transactions of the Association for Computational Linguistics · cité 1 376 fois · Pas encore résumé

    MuSiQue construit un jeu de données de questions multi-sauts composées de questions à un saut, un type de ressource utile à l’évaluation de MCR.

    Résumé scientifique

    Multihop reasoning remains an elusive goal as existing multihop benchmarks are known to be largely solvable via shortcuts. Can we create a question answering (QA) dataset that, by construction, requires proper multihop reasoning? To this end, we introduce a bottom–up approach that systematically selects composable pairs of single-hop questions that are connected, that is, where one reasoning step critically relies on information from another. This bottom–up methodology lets us explore a vast space of questions and add stringent filters as well as other mechanisms targeting connected reasoning. It provides fine-grained control over the construction process and the properties of the resulting …

  3. Très proche Contexte plus large Une méthode qu’il utilise
    Entailer: Answering Questions with Faithful and Truthful Chains of Reasoning
    O. Tafjord, Bhavana Dalvi Mishra, Peter Clark · 2022 · Conference on Empirical Methods in Natural Language Processing · cité 69 fois · Pas encore résumé

    Entailer génère des chaînes multi-étapes vérifiées comme fidèles et véridiques, alors que MCR combine des chaînes pour répondre aux questions multi-sauts.

    Résumé scientifique

    Our goal is a question-answering (QA) system that can show how its answers are implied by its own internal beliefs via a systematic chain of reasoning. Such a capability would allow better understanding of why a model produced the answer it did. Our approach is to recursively combine a trained backward-chainingmodel, capable of generating a set of premises entailing an answer hypothesis, with a verifier that checks that the model itself believes those premises (and the entailment itself) through self-querying. To our knowledge, this is the first system to generate multistep chains that are both faithful (the answer follows from the reasoning) and truthful (the chain reflects the system’s own…

  4. Proche Données, outil ou méthode en commun Utilisé en profondeur par cet article Une méthode qu’il utilise
    Did Aristotle Use a Laptop? A Question Answering Benchmark with Implicit Reasoning Strategies
    Mor Geva, Daniel Khashabi, Elad Segal et al. · 2021 · Transactions of the Association for Computational Linguistics · cité 1 249 fois · Pas encore résumé

    Did Aristotle Use a Laptop? présente StrategyQA, un benchmark de questions dont les étapes de raisonnement sont implicites.

    Résumé scientifique

    Abstract A key limitation in current datasets for multi-hop reasoning is that the required steps for answering the question are mentioned in it explicitly. In this work, we introduce StrategyQA, a question answering (QA) benchmark where the required reasoning steps are implicit in the question, and should be inferred using a strategy. A fundamental challenge in this setup is how to elicit such creative questions from crowdsourcing workers, while covering a broad range of potential strategies. We propose a data collection procedure that combines term-based priming to inspire annotators, careful control over the annotator population, and adversarial filtering for eliminating reasoning shortcut…

  5. Proche Contexte plus large Utilisé en profondeur par cet article Une méthode qu’il utilise
    Measuring and Narrowing the Compositionality Gap in Language Models
    Ofir Press, Muru Zhang, Sewon Min et al. · 2022 · Computation and Language · cité 1 608 fois · Résumé IA disponible

    Measuring and Narrowing the Compositionality Gap mesure l’écart entre réussite aux sous-problèmes et résolution de questions nécessitant de composer plusieurs faits.

    Résumé scientifique

    We investigate the ability of language models to perform compositional reasoning tasks where the overall solution depends on correctly composing the answers to sub-problems. We measure how often models can correctly answer all sub-problems but not generate the overall solution, a ratio we call the compositionality gap. We evaluate this ratio by asking multi-hop questions with answers that require composing multiple facts unlikely to have been observed together during pretraining. In the GPT-3 family of models, as model size increases we show that the single-hop question answering performance improves faster than the multi-hop performance does, therefore the compositionality gap does not decr…

Les articles qui le citent

  1. Très proche Autre approche du problème
    Hierarchical Retrieval-Augmented Generation Model with Rethink for Multi-hop Question Answering
    Xiaoming Zhang, Ming Wang, Xiaocui Yang et al. · 2024 · arXiv.org · preprint, non relu par les pairs · cité 20 fois · Pas encore résumé

    HiRAG combine décomposition, récupération hiérarchique et synthèse pour le QA multi-saut, tandis que MCR meta-raisonne sur plusieurs chaînes.

    Résumé scientifique

    Multi-hop Question Answering (QA) necessitates complex reasoning by integrating multiple pieces of information to resolve intricate questions. However, existing QA systems encounter challenges such as outdated information, context window length limitations, and an accuracy-quantity trade-off. To address these issues, we propose a novel framework, the Hierarchical Retrieval-Augmented Generation Model with Rethink (HiRAG), comprising Decomposer, Definer, Retriever, Filter, and Summarizer five key modules. We introduce a new hierarchical retrieval strategy that incorporates both sparse retrieval at the document level and dense retrieval at the chunk level, effectively integrating their strength…

  2. Proche Autre approche du problème Utilise cet article en profondeur
    Graph Elicitation for Guiding Multi-Step Reasoning in Large Language Models
    Jinyoung Park, Ameen Patel, Omar Zia Khan et al. · 2023 · cité 5 fois · Pas encore résumé

    GE-Reasoning forme un graphe de triplets pour guider la création de sous-questions et leurs réponses, au lieu de mélanger des chaînes de pensée.

    Résumé scientifique

    Chain-of-Thought (CoT) prompting along with sub-question generation and answering has enhanced multi-step reasoning capabilities of Large Language Models (LLMs). However, prompting the LLMs to directly generate sub-questions is suboptimal since they sometimes generate redundant or irrelevant questions. To deal with them, we propose a GE-Reasoning method, which directs LLMs to generate proper sub-questions and corresponding answers. Concretely, given an input question, we first prompt the LLM to generate knowledge triplets, forming a graph representation of the question. Unlike conventional knowledge triplets, our approach allows variables as head or tail entities, effectively representing a …

  3. Très proche Contexte plus large
    Can We Verify Step by Step for Incorrect Answer Detection?
    Xin Xu, Shi-Zhe Diao, Can Yang et al. · 2024 · International Joint Conference on Artificial Intelligence · cité 16 fois · Pas encore résumé

    Can We Verify Step by Step évalue si l’examen des chaînes de raisonnement permet de détecter les réponses incorrectes, plutôt que de générer une réponse par méta-raisonnement.

    Résumé scientifique

    Chain-of-Thought (CoT) prompting has marked a significant advancement in enhancing the reasoning capabilities of large language models (LLMs). Previous studies have developed various extensions of CoT, which focus primarily on enhancing end-task performance. In addition, there has been research on assessing the quality of reasoning chains in CoT. This raises an intriguing question: Is it possible to predict the accuracy of LLM outputs by scrutinizing the reasoning chains they generate? To answer this research question, we introduce a benchmark, R2PE, designed specifically to explore the relationship between reasoning chains and performance in various reasoning tasks spanning five different d…

  4. Proche Autre approche du problème Utilise cet article en profondeur
    Fine-Tuning on Diverse Reasoning Chains Drives Within-Inference CoT Refinement in LLMs
    Haritz Puerto, Tilek Chubakov, Xiaodan Zhu et al. · 2024 · Annual Meeting of the Association for Computational Linguistics · cité 19 fois · Pas encore résumé

    Fine-Tuning on Diverse Reasoning Chains entraîne les modèles à raffiner plusieurs chaînes au cours d’une inférence, tandis que MCR les combine par sollicitation du modèle.

    Résumé scientifique

    Requiring a large language model (LLM) to generate intermediary reasoning steps, known as Chain of Thought (CoT), has been shown to be an effective way of boosting performance. Previous approaches have focused on generating multiple independent CoTs, combining them through ensembling or other post-hoc strategies to enhance reasoning. In this work, we introduce a novel approach where LLMs are fine-tuned to generate a sequence of Diverse Chains of Thought (DCoT) within a single inference step, which is fundamentally different from prior work that primarily operate on parallel CoT generations. DCoT allows LLMs to gain the ability to perform within-inference refinement of reasoning chains withou…

  5. Proche Contexte plus large
    Meta-CoT: Generalizable Chain-of-Thought Prompting in Mixed-task Scenarios with Large Language Models
    An-Ni Zou, Zhuosheng Zhang, Hai Zhao et al. · 2023 · arXiv.org · preprint, non relu par les pairs · cité 6 fois · Pas encore résumé

    Meta-CoT catégorise les questions de tâches mixtes pour choisir des démonstrations de CoT généralisables, plutôt que de traiter plusieurs chaînes pour le QA multi-saut.

    Résumé scientifique

    Large language models (LLMs) have unveiled remarkable reasoning capabilities by exploiting chain-of-thought (CoT) prompting, which generates intermediate reasoning chains to serve as the rationale for deriving the answer. However, current CoT methods either simply employ general prompts such as Let's think step by step, or heavily rely on pre-defined task-specific demonstrations to attain preferable performances, thereby engendering an inescapable gap between performance and generalization. To bridge this gap, we propose GeM-CoT, a Generalizable CoT prompting mechanism in Mixed-task scenarios where the type of input questions is unknown. GeM-CoT first categorizes the question type and subseq…

Comment ces articles sont-ils trouvés ?

Nous comparons le titre et le résumé de cet article avec ceux de tous les articles résumés sur ce site, et avec les articles que Semantic Scholar recommande, ceux qu’il cite et ceux qui le citent (pour un article très cité, ses citations les plus récentes et, depuis OpenAlex, les plus citées). Parmi les travaux antérieurs et les suites, les articles qui s’utilisent en profondeur, une méthode qu’il utilise et les suites très citées remontent un peu. Les niveaux (très proche, proche, apparenté) viennent de la similarité des textes, pas d’un jugement de qualité ; * signale un article sans résumé, comparé par son titre. La phrase sous un article est écrite par une IA à partir des deux résumés : vérifiez-la dans les articles. Un article jugé hors sujet par trois lecteurs quitte la liste.

Mis à jour le 6 novembre 2026.

Le guide illustré des articles autour d’un article