Références vérifiées

Références

• devlin2019 · BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding · arXiv:1810.04805

  Publication finale disponible Cité comme preprint, mais publié dans Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers). 10.18653/v1/n19-1423
  Correction proposée — à relire avant utilisation

  Votre entrée

  @article{devlin2019,
    title={BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding},
    author={Devlin, Jacob and Chang, Ming-Wei and Lee, Kenton and Toutanova, Kristina},
    journal={arXiv preprint arXiv:1810.04805},
    year={2018}
  }

  Corrigée

  @article{devlin2019,
    title = {BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding},
    author = {Devlin, Jacob and Chang, Ming-Wei and Lee, Kenton and Toutanova, Kristina},
    journal = {Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)},
    year = {2019},
    pages = {4171-4186},
    doi = {10.18653/v1/n19-1423},
  }

  Garder cette correction dans le .bib téléchargé

• brown2020b · Language models are few-shot learners · 10.48550/arxiv.2005.14165

  Doublon Le même article que brown2020.
  Entrée incomplète Ajouté d’après la notice qui l’identifie : pages = arXiv:2005.14165.
  Correction proposée — à relire avant utilisation

  Votre entrée

  @inproceedings{brown2020b,
    title={Language models are few-shot learners},
    author={Brown, Tom and Mann, Benjamin and Ryder, Nick and others},
    booktitle={Advances in Neural Information Processing Systems},
    year={2020}
  }

  Corrigée

  @inproceedings{brown2020b,
    title = {Language models are few-shot learners},
    author = {Brown, Tom and Mann, Benjamin and Ryder, Nick and others},
    booktitle = {Advances in Neural Information Processing Systems},
    year = {2020},
    pages = {arXiv:2005.14165},
  }

  Garder cette correction dans le .bib téléchargé

• vaswani2017 · Attention is all you need

  Référence non résolue Aucune correspondance suffisamment fiable n’a été retrouvée dans les sources consultées. Cette référence n’est pas nécessairement incorrecte.

• karpukhin2020 · Dense Passage Retrieval for Open-Domain Question Answering

  Référence non résolue Aucune correspondance suffisamment fiable n’a été retrouvée dans les sources consultées. Cette référence n’est pas nécessairement incorrecte.
  Entrée incomplète Manque encore : journal.

Citations à vérifier

3 phrases avec des citations lues sur 4 : d’abord celles qui attribuent un résultat, une valeur ou une méthode à un article. Chaque phrase est lue avec les résumés de tous les articles qu’elle cite ; celles qui sont signalées sont relues dans le texte intégral des articles quand il peut être lu. Une citation signalée peut être juste : vérifiez-la dans l’article. Textes intégraux lus : 2 sur les 3 articles concernés (les autres sont fermés à la fouille de textes ou n’ont pas pu être récupérés).

• L’article dit autre chose cité pour un résultat · main.tex, ligne 8

  Lewis et al. found that adding retrieved passages made the generated answers less specific and less factual than those of a parametric-only model [lewis2020].

  The paper reports that retrieval-augmented models generate more specific and factual language, the opposite of the draft's claim.

  For language generation tasks, we find that RAG models generate more specific, diverse and factual language than a state-of-the-art parametric-only seq2seq baseline.

  lewis2020 · Abstract · Ouvrir l’article à ce passage

  lewis2020 · Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks

  Résumé scientifique

  Large pre-trained language models have been shown to store factual knowledge in their parameters, and achieve state-of-the-art results when fine-tuned on downstream NLP tasks. However, their ability to access and precisely manipulate knowledge is still limited, and hence on knowledge-intensive tasks, their performance lags behind task-specific architectures. Additionally, providing provenance for their decisions and updating their world knowledge remain open research problems. Pre-trained models with a differentiable access mechanism to explicit non-parametric memory can overcome this issue, but have so far been only investigated for extractive downstream tasks. We explore a general-purpose fine-tuning recipe for retrieval-augmented generation (RAG) -- models which combine pre-trained parametric and non-parametric memory for language generation. We introduce RAG models where the parametric memory is a pre-trained seq2seq model and the non-parametric memory is a dense vector index of Wikipedia, accessed with a pre-trained neural retriever. We compare two RAG formulations, one which conditions on the same retrieved passages across the whole generated sequence, the other can use different passages per token. We fine-tune and evaluate our models on a wide range of knowledge-intensive NLP tasks and set the state-of-the-art on three open domain QA tasks, outperforming parametric seq2seq models and task-specific retrieve-and-extract architectures. For language generation tasks, we f

• L’article dit autre chose cité comme contexte · main.tex, ligne 6

  Transformer language models were first introduced by Devlin et al. [devlin2019], building on attention [vaswani2017].

  The paper identifies OpenAI GPT as an existing Transformer-based language model, contradicting the attribution of the first such models to Devlin et al.

  For example, in OpenAI GPT, the authors use a left-to-right architecture, where every token can only attend to previous tokens in the self-attention layers of the Transformer Vaswani et al. (2017).

  devlin2019 · 1 Introduction · Ouvrir l’article à ce passage

  devlin2019 · BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding

  Résumé scientifique

  We introduce a new language representation model called BERT, which stands for Bidirectional Encoder Representations from Transformers. Unlike recent language representation models, BERT is designed to pre-train deep bidirectional representations from unlabeled text by jointly conditioning on both left and right context in all layers. As a result, the pre-trained BERT model can be fine-tuned with just one additional output layer to create state-of-the-art models for a wide range of tasks, such as question answering and language inference, without substantial task-specific architecture modifications. BERT is conceptually simple and empirically powerful. It obtains new state-of-the-art results on eleven natural language processing tasks, including pushing the GLUE score to 80.5% (7.7% point absolute improvement), MultiNLI accuracy to 86.7% (4.6% absolute improvement), SQuAD v1.1 question answering Test F1 to 93.2 (1.5 point absolute improvement) and SQuAD v2.0 Test F1 to 83.1 (5.1 point absolute improvement).