# Sequence to Sequence Learning with Neural Networks

**Authors:** Ilya Sutskever, Oriol Vinyals, Quoc V. Le  
**arXiv:** [1409.3215](https://arxiv.org/abs/1409.3215)  
**Date:** 2014-09-10  
**Summary page:** https://www.summarizepaper.com/en/paper/1409.3215v1/

## Key points

- Authors: Ilya Sutskever, Oriol Vinyals, Quoc V. Le
- Paper title: "Sequence to Sequence Learning with Neural Networks"
- Deep Neural Networks (DNNs) have limitations in mapping sequences to sequences
- Proposed approach: end-to-end using multilayered Long Short-Term Memory (LSTM) networks
- One LSTM network maps input sequence into a fixed-dimensional vector
- Another deep LSTM network decodes target sequence from the vector
- Evaluation on English to French translation task using WMT-14 dataset
- LSTM translations achieve BLEU score of 34.7 on test set, even after penalizing for out-of-vocabulary words
- Comparison to strong phrase-based Statistical Machine Translation (SMT) system with BLEU score of 33.3 shows superiority of LSTM model
- Reranking hypotheses generated by SMT system increases LSTM's BLEU score further to 36.5, surpassing previous state-of-the-art performance
- LSTM model handles long sentences well and learns sensible phrase and sentence representations considering word order and remaining relatively invariant to active/passive voice constructions
- Reversing word order in source sentences improves LSTM's performance significantly; introduces short-term dependencies between source and target sentences making optimization easier

## Summary

Authors Ilya Sutskever, Oriol Vinyals, and Quoc V. Le present a paper titled "Sequence to Sequence Learning with Neural Networks," which explores the limitations of Deep Neural Networks (DNNs) in mapping sequences to sequences. While DNNs have achieved impressive results in learning tasks with large labeled training sets, they are unable to handle sequence-to-sequence mapping. To address this issue, the authors propose an end-to-end approach that utilizes multilayered Long Short-Term Memory (LSTM) networks. The method involves using one LSTM network to map the input sequence into a fixed-dimensional vector and another deep LSTM network to decode the target sequence from this vector. The researchers evaluate their approach on an English to French translation task using the WMT-14 dataset. Remarkably, the LSTM translations achieve a BLEU score of 34.7 on the entire test set, even after penalizing for out-of-vocabulary words. Comparing these results to a strong phrase-based Statistical Machine Translation (SMT) system, which achieves a BLEU score of 33.3 on the same dataset, demonstrates the superiority of their LSTM model. Additionally, when used to rerank 1000 hypotheses generated by the SMT system, the LSTM's BLEU score increases further to 36.5, surpassing previous state-of-the-art performance. The authors note that their LSTM model exhibits no difficulty in handling long sentences and learns sensible phrase and sentence representations that consider word order and remain relatively invariant to active and passive voice constructions. An interesting finding is that reversing the word order in source sentences (but not target sentences) significantly improves the LSTM's performance. This reversal introduces short-term dependencies between source and target sentences making it easier for optimization problem solving. Overall, this paper presents a novel approach using LSTMs for sequence learning without imposing strict assumptions on sequence structure; its impressive results achieved on English to French translation task highlight its potential for various sequence mapping tasks.