References checked

References

• devlin2019 · BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding · arXiv:1810.04805

  Final publication available Cited as a preprint, but published in Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers). 10.18653/v1/n19-1423
  Suggested corrected entry — review before using

  Your entry

  @article{devlin2019,
    title={BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding},
    author={Devlin, Jacob and Chang, Ming-Wei and Lee, Kenton and Toutanova, Kristina},
    journal={arXiv preprint arXiv:1810.04805},
    year={2018}
  }

  Corrected

  @article{devlin2019,
    title = {BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding},
    author = {Devlin, Jacob and Chang, Ming-Wei and Lee, Kenton and Toutanova, Kristina},
    journal = {Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)},
    year = {2019},
    pages = {4171-4186},
    doi = {10.18653/v1/n19-1423},
  }

  Keep this correction in the downloaded .bib

• brown2020b · Language models are few-shot learners · 10.48550/arxiv.2005.14165

  Duplicate The same paper as brown2020.
  Incomplete entry Added from the record that identifies it: pages = arXiv:2005.14165.
  Suggested corrected entry — review before using

  Your entry

  @inproceedings{brown2020b,
    title={Language models are few-shot learners},
    author={Brown, Tom and Mann, Benjamin and Ryder, Nick and others},
    booktitle={Advances in Neural Information Processing Systems},
    year={2020}
  }

  Corrected

  @inproceedings{brown2020b,
    title = {Language models are few-shot learners},
    author = {Brown, Tom and Mann, Benjamin and Ryder, Nick and others},
    booktitle = {Advances in Neural Information Processing Systems},
    year = {2020},
    pages = {arXiv:2005.14165},
  }

  Keep this correction in the downloaded .bib

• vaswani2017 · Attention is all you need

  Reference unresolved No sufficiently reliable matching record was found in the sources consulted. This reference is not necessarily incorrect.

• karpukhin2020 · Dense Passage Retrieval for Open-Domain Question Answering

  Reference unresolved No sufficiently reliable matching record was found in the sources consulted. This reference is not necessarily incorrect.
  Incomplete entry Still missing: journal.

Citations to check

3 sentences with citations read out of 4: those that attribute a result, a value or a method to a paper first. Each sentence is read with the abstracts of all the papers it cites; those flagged are read again against the full text of the papers when it may be read. A flagged citation may be right: check it in the paper. Full texts read: 2 of the 3 papers concerned (the others are closed to text mining or could not be fetched).

• The paper says otherwise cited for a result · main.tex, line 8

  Lewis et al. found that adding retrieved passages made the generated answers less specific and less factual than those of a parametric-only model [lewis2020].

  Les auteurs rapportent que RAG produit un langage plus spécifique et plus factuel que le modèle purement paramétrique, soit l’inverse de l’affirmation.

  For language generation tasks, we find that RAG models generate more specific, diverse and factual language than a state-of-the-art parametric-only seq2seq baseline.

  lewis2020 · Abstract · Open the paper at this passage

  lewis2020 · Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks

  Abstract

  Large pre-trained language models have been shown to store factual knowledge in their parameters, and achieve state-of-the-art results when fine-tuned on downstream NLP tasks. However, their ability to access and precisely manipulate knowledge is still limited, and hence on knowledge-intensive tasks, their performance lags behind task-specific architectures. Additionally, providing provenance for their decisions and updating their world knowledge remain open research problems. Pre-trained models with a differentiable access mechanism to explicit non-parametric memory can overcome this issue, but have so far been only investigated for extractive downstream tasks. We explore a general-purpose fine-tuning recipe for retrieval-augmented generation (RAG) -- models which combine pre-trained parametric and non-parametric memory for language generation. We introduce RAG models where the parametric memory is a pre-trained seq2seq model and the non-parametric memory is a dense vector index of Wikipedia, accessed with a pre-trained neural retriever. We compare two RAG formulations, one which conditions on the same retrieved passages across the whole generated sequence, the other can use different passages per token. We fine-tune and evaluate our models on a wide range of knowledge-intensive NLP tasks and set the state-of-the-art on three open domain QA tasks, outperforming parametric seq2seq models and task-specific retrieve-and-extract architectures. For language generation tasks, we f

• The paper says otherwise cited as background · main.tex, line 6

  Transformer language models were first introduced by Devlin et al. [devlin2019], building on attention [vaswani2017].

  Le texte décrit GPT comme utilisant déjà un Transformer avant BERT, ce qui contredit l’attribution de l’introduction des modèles de langage Transformer à Devlin et ses collègues.

  For example, in OpenAI GPT, the authors use a left-to-right architecture, where every token can only attend to previous tokens in the self-attention layers of the Transformer Vaswani et al. (2017).

  devlin2019 · 1 Introduction · Open the paper at this passage

  devlin2019 · BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding

  Abstract

  We introduce a new language representation model called BERT, which stands for Bidirectional Encoder Representations from Transformers. Unlike recent language representation models, BERT is designed to pre-train deep bidirectional representations from unlabeled text by jointly conditioning on both left and right context in all layers. As a result, the pre-trained BERT model can be fine-tuned with just one additional output layer to create state-of-the-art models for a wide range of tasks, such as question answering and language inference, without substantial task-specific architecture modifications. BERT is conceptually simple and empirically powerful. It obtains new state-of-the-art results on eleven natural language processing tasks, including pushing the GLUE score to 80.5% (7.7% point absolute improvement), MultiNLI accuracy to 86.7% (4.6% absolute improvement), SQuAD v1.1 question answering Test F1 to 93.2 (1.5 point absolute improvement) and SQuAD v2.0 Test F1 to 83.1 (5.1 point absolute improvement).