Copy Is All You Need

Classés selon la proximité de leur titre et de leur résumé avec ceux de cet article. Sélectionnez-en pour les lire ensemble.

Les articles les plus proches

1. Très proche Pas pertinent

   May the Force Be with Your Copy Mechanism: Enhanced Supervised-Copy Method for Natural Language Generation

   Sanghyuk Choi, J. Hwang, Hyungjong Noh et al. · 2021 · arXiv.org · preprint, non relu par les pairs · cité 4 fois · Pas encore résumé

   Résumé scientifique

   Recent neural sequence-to-sequence models with a copy mechanism have achieved remarkable progress in various text generation tasks. These models addressed out-of-vocabulary problems and facilitated the generation of rare words. However, the identification of the word which needs to be copied is difficult, as observed by prior copy models, which suffer from incorrect generation and lacking abstractness. In this paper, we propose a novel supervised approach of a copy network that helps the model decide which words need to be copied and which need to be generated. Specifically, we re-define the objective function, which leverages source sequences and target vocabularies as guidance for copying.…

2. Très proche Pas pertinent

   Deep Copycat Networks for Text-to-Text Generation

   Julia Ive, P. Madhyastha, Lucia Specia · 2019 · Conference on Empirical Methods in Natural Language Processing · cité 7 fois · Pas encore résumé

   Résumé scientifique

   Most text-to-text generation tasks, for example text summarisation and text simplification, require copying words from the input to the output. We introduce Copycat, a transformer-based pointer network for such tasks which obtains competitive results in abstractive text summarisation and generates more abstractive summaries. We propose a further extension of this architecture for automatic post-editing, where generation is conditioned over two inputs (source language and machine translation), and the model is capable of deciding where to copy information from. This approach achieves competitive performance when compared to state-of-the-art automated post-editing systems. More importantly, we…

3. Proche Pas pertinent

   Improving Copy-oriented Text Generation via EDU Copy Mechanism

   Tianxiang Wu, Han Chen, Luozheng Qin et al. · 2024 · International Conference on Language Resources and Evaluation · cité 1 fois · Pas encore résumé

   Résumé scientifique

   Many text generation tasks are copy-oriented. For instance, nearly 30% content of news summaries is copied. The copy rate is even higher in Grammatical Error Correction (GEC). However, existing generative models generate texts through word-by-word decoding, which may lead to factual inconsistencies and slow inference. While Elementary Discourse Units (EDUs) are outstanding extraction units, EDU-based extractive methods can alleviate the aforementioned problems. As a consequence, we propose EDUCopy, a framework that integrates the behavior of copying EDUs into generative models. The main idea of EDUCopy is to use special index tags to represent the copied EDUs during generation. Specifically,…

4. Proche Pas pertinent

   Table to text generation with accurate content copying

   Yang Yang, Juan Cao, Yu-Jun Wen et al. · 2021 · Scientific Reports · cité 15 fois · Pas encore résumé

   Résumé scientifique

   Generating fluent, coherent, and informative text from structured data is called table-to-text generation. Copying words from the table is a common method to solve the “out-of-vocabulary” problem, but it’s difficult to achieve accurate copying. In order to overcome this problem, we invent an auto-regressive framework based on the transformer that combines a copying mechanism and language modeling to generate target texts. Firstly, to make the model better learn the semantic relevance between table and text, we apply a word transformation method, which incorporates the field and position information into the target text to acquire the position of where to copy. Then we propose two auxiliary l…

5. Proche Cité par cet article Utilisé en profondeur par cet article Une méthode qu’il utilise Pas pertinent

   A Contrastive Framework for Neural Text Generation

   Yixuan Su, Tian Lan, Yan Wang et al. · 2022 · Neural Information Processing Systems · cité 355 fois · Pas encore résumé

   Résumé scientifique

   Text generation is of great importance to many natural language processing applications. However, maximization-based decoding methods (e.g. beam search) of neural language models often lead to degenerate solutions -- the generated text is unnatural and contains undesirable repetitions. Existing approaches introduce stochasticity via sampling or modify training objectives to decrease probabilities of certain tokens (e.g., unlikelihood training). However, they often lead to solutions that lack coherence. In this work, we show that an underlying reason for model degeneration is the anisotropic distribution of token representations. We present a contrastive solution: (i) SimCTG, a contrastive tr…

6. Proche Pas pertinent

   Generate rather than Retrieve: Large Language Models are Strong Context Generators

   Wenhao Yu, Dan Iter, Shuohang Wang et al. · 2022 · Computation and Language · cité 461 fois · Résumé IA disponible

   Résumé scientifique

   Knowledge-intensive tasks, such as open-domain question answering (QA), require access to a large amount of world or domain knowledge. A common approach for knowledge-intensive tasks is to employ a retrieve-then-read pipeline that first retrieves a handful of relevant contextual documents from an external corpus such as Wikipedia and then predicts an answer conditioned on the retrieved documents. In this paper, we present a novel perspective for solving knowledge-intensive tasks by replacing document retrievers with large language model generators. We call our method generate-then-read (GenRead), which first prompts a large language model to generate contextutal documents based on a given qu…

7. Proche Pas pertinent

   Text Generation with Exemplar-based Adaptive Decoding

   Hao Peng, Ankur P. Parikh, Manaal Faruqui et al. · 2019 · North American Chapter of the Association for Computational Linguistics · cité 59 fois · Pas encore résumé

   Résumé scientifique

   We propose a novel conditioned text generation model. It draws inspiration from traditional template-based text generation techniques, where the source provides the content (i.e., what to say), and the template influences how to say it. Building on the successful encoder-decoder paradigm, it first encodes the content representation from the given input text; to produce the output, it retrieves exemplar text from the training data as “soft templates,” which are then used to construct an exemplar-specific decoder. We evaluate the proposed model on abstractive text summarization and data-to-text generation. Empirical results show that this model achieves strong performance and outperforms compa…

8. Proche Cite cet article Pas pertinent

   Copy-Paste Language Models

   Nathan Godey, Yoav Artzi · 2026 · Pas encore résumé

   Résumé scientifique

   Transformers are intuitively considered to be efficient at copying from their inputs. We actually show that the ability of LLMs to copy in context can be improved through explicit modeling. We propose Copy-Paste Language Models (CPLMs), which add a lightweight Copy-Paste head to the conventional language modeling (LM) head. CPLMs improve training efficiency, downstream performance, and offer new operating points in long-context and retrieval setups. We validate our approach by pretraining models from 135M to 1.7B parameters, showing improvements across all scales and backbones with negligible training and inference overheads. We explain the success of CPLMs by showing that they partially mit…

Les travaux antérieurs sur lesquels il s’appuie

1. Apparenté Utilisé en profondeur par cet article Une méthode qu’il utilise Pas pertinent

   The Curious Case of Neural Text Degeneration

   Ari Holtzman, Jan Buys, Li Du et al. · 2019 · International Conference on Learning Representations · cité 4 749 fois · Pas encore résumé

   Résumé scientifique

   Despite considerable advancements with deep neural language models, the enigma of neural text degeneration persists when these models are tested as text generators. The counter-intuitive empirical observation is that even though the use of likelihood as training objective leads to high quality models for a broad range of language understanding tasks, using likelihood as a decoding objective leads to text that is bland and strangely repetitive. In this paper, we reveal surprising distributional differences between human text and machine text. In addition, we find that decoding strategies alone can dramatically effect the quality of machine text, even when generated from exactly the same neura…

2. Apparenté Utilisé en profondeur par cet article Une méthode qu’il utilise Pas pertinent

   Improving language models by retrieving from trillions of tokens

   Sebastian Borgeaud, Arthur Mensch, Jordan Hoffmann et al. · 2021 · International Conference on Machine Learning · cité 2 026 fois · Pas encore résumé

   Résumé scientifique

   We enhance auto-regressive language models by conditioning on document chunks retrieved from a large corpus, based on local similarity with preceding tokens. With a $2$ trillion token database, our Retrieval-Enhanced Transformer (RETRO) obtains comparable performance to GPT-3 and Jurassic-1 on the Pile, despite using 25$\times$ fewer parameters. After fine-tuning, RETRO performance translates to downstream knowledge-intensive tasks such as question answering. RETRO combines a frozen Bert retriever, a differentiable encoder and a chunked cross-attention mechanism to predict tokens based on an order of magnitude more data than what is typically consumed during training. We typically train RETR…

3. Apparenté Utilisé en profondeur par cet article Une méthode qu’il utilise Pas pertinent

   MAUVE: Measuring the Gap Between Neural Text and Human Text using Divergence Frontiers

   Krishna Pillutla, S. Swayamdipta, Rowan Zellers et al. · 2021 · Neural Information Processing Systems · cité 637 fois · Résumé IA disponible

   Résumé scientifique

   As major progress is made in open-ended text generation, measuring how close machine-generated text is to human language remains a critical open problem. We introduce MAUVE, a comparison measure for open-ended text generation, which directly compares the learnt distribution from a text generation model to the distribution of human-written text using divergence frontiers. MAUVE scales up to modern text generation models by computing information divergences in a quantized embedding space. Through an extensive empirical study on three open-ended generation tasks, we find that MAUVE identifies known properties of generated text, scales naturally with model size, and correlates with human judgmen…

4. Apparenté Une méthode qu’il utilise Pas pertinent

   Attention Is All You Need

   Ashish Vaswani, Noam Shazeer, Niki Parmar et al. · 2017 · Computation and Language · cité 201 387 fois · Résumé IA disponible

   Résumé scientifique

   The dominant sequence transduction models are based on complex recurrent or convolutional neural networks in an encoder-decoder configuration. The best performing models also connect the encoder and decoder through an attention mechanism. We propose a new simple network architecture, the Transformer, based solely on attention mechanisms, dispensing with recurrence and convolutions entirely. Experiments on two machine translation tasks show these models to be superior in quality while being more parallelizable and requiring significantly less time to train. Our model achieves 28.4 BLEU on the WMT 2014 English-to-German translation task, improving over the existing best results, including ense…

5. Apparenté Utilisé en profondeur par cet article Une méthode qu’il utilise Pas pertinent

   Neural Text Generation with Unlikelihood Training

   S. Welleck, Ilia Kulikov, Stephen Roller et al. · 2019 · International Conference on Learning Representations · cité 800 fois · Pas encore résumé

   Résumé scientifique

   Neural text generation is a key tool in natural language applications, but it is well known there are major problems at its core. In particular, standard likelihood training and decoding leads to dull and repetitive outputs. While some post-hoc fixes have been proposed, in particular top-$k$ and nucleus sampling, they do not address the fact that the token-level probabilities predicted by the model are poor. In this paper we show that the likelihood objective itself is at fault, resulting in a model that assigns too much probability to sequences containing repeats and frequent words, unlike those from the human training distribution. We propose a new objective, unlikelihood training, which f…

Les articles qui le citent

1. Proche Utilise cet article en profondeur Pas pertinent

   Retrieval is Accurate Generation

   Bowen Cao, Deng Cai, Leyang Cui et al. · 2024 · International Conference on Learning Representations · cité 15 fois · Pas encore résumé

   Résumé scientifique

   Standard language models generate text by selecting tokens from a fixed, finite, and standalone vocabulary. We introduce a novel method that selects context-aware phrases from a collection of supporting documents. One of the most significant challenges for this paradigm shift is determining the training oracles, because a string of text can be segmented in various ways and each segment can be retrieved from numerous possible documents. To address this, we propose to initialize the training oracles using linguistic heuristics and, more importantly, bootstrap the oracles through iterative self-reinforcement. Extensive experiments show that our model not only outperforms standard language model…

2. Apparenté Pas pertinent

   Retrieval-Augmented Generation for AI-Generated Content: A Survey

   Peng-Hao Zhao, Hailin Zhang, Qinhan Yu et al. · 2024 · Data Science and Engineering · cité 789 fois · Résumé IA disponible

   Résumé scientifique

   Advancements in model algorithms, the growth of foundational models, and access to high-quality datasets have propelled the evolution of Artificial Intelligence Generated Content (AIGC). Despite its notable successes, AIGC still faces hurdles such as updating knowledge, handling long-tail data, mitigating data leakage, and managing high training and inference costs. Retrieval-Augmented Generation (RAG) has recently emerged as a paradigm to address such challenges. In particular, RAG introduces the information retrieval process, which enhances the generation process by retrieving relevant objects from available data stores, leading to higher accuracy and better robustness. In this paper, we c…

3. Apparenté Pas pertinent

   CopyNE: Better Contextual ASR by Copying Named Entities

   Shilin Zhou, Zheng-Hua Li, Yu Hong et al. · 2023 · Annual Meeting of the Association for Computational Linguistics · cité 19 fois · Pas encore résumé

   Résumé scientifique

   End-to-end automatic speech recognition (ASR) systems have made significant progress in general scenarios. However, it remains challenging to transcribe contextual named entities (NEs) in the contextual ASR scenario. Previous approaches have attempted to address this by utilizing the NE dictionary. These approaches treat entities as individual tokens and generate them token-by-token, which may result in incomplete transcriptions of entities. In this paper, we treat entities as indivisible wholes and introduce the idea of copying into ASR. We design a systematic mechanism called CopyNE, which can copy entities from the NE dictionary. By copying all tokens of an entity at once, we can reduce e…

4. Apparenté Pas pertinent

   Hierarchical Skip Decoding for Efficient Autoregressive Text Generation

   Yun-Qi Zhu, Xue-Bing Yang, Yuan-Yuan Wu et al. · 2024 · arXiv.org · preprint, non relu par les pairs · cité 8 fois · Pas encore résumé

   Résumé scientifique

   Autoregressive decoding strategy is a commonly used method for text generation tasks with pre-trained language models, while early-exiting is an effective approach to speedup the inference stage. In this work, we propose a novel decoding strategy named Hierarchical Skip Decoding (HSD) for efficient autoregressive text generation. Different from existing methods that require additional trainable components, HSD is a plug-and-play method applicable to autoregressive text generation models, it adaptively skips decoding layers in a hierarchical manner based on the current sequence length, thereby reducing computational workload and allocating computation resources. Comprehensive experiments on f…

5. Apparenté Pas pertinent

   With Greater Text Comes Greater Necessity: Inference-Time Training Helps Long Text Generation

   Yan Wang, D. Ma, Deng Cai · 2024 · arXiv.org · preprint, non relu par les pairs · cité 42 fois · Pas encore résumé

   Résumé scientifique

   Long text generation, such as novel writing and discourse-level translation with extremely long contexts, presents significant challenges to current language models. Existing methods mainly focus on extending the model's context window through strategies like length extrapolation. However, these approaches demand substantial hardware resources during the training and/or inference phases. Our proposed method, Temp-Lora, introduces an alternative concept. Instead of relying on the KV cache to store all context information, we embeds this information directly into a temporary Lora module. In the process of long text generation, this module is progressively trained with text generated previously…