# Modeling Order in Neural Word Embeddings at Scale

**Authors:** Andrew Trask, David Gilmore, Matthew Russell  
**arXiv:** [1506.02338](https://arxiv.org/abs/1506.02338)  
**Date:** 2015-06-08  
**Summary page:** https://www.summarizepaper.com/en/paper/1506.02338v1/

## Key points

- The paper explores the use of NLP systems and their reliance on bag-of-words co-occurrence techniques in capturing word relationships.
- The authors propose a new neural language model that incorporates both word order and character order in its embedding.
- This approach creates vector spaces with meaningful substructure.
- The model achieves an impressive accuracy rate of 85.8% on a word-analogy task, surpassing previous scores by 58% error reduction.
- The model includes several parallel training methods, allowing for efficient training of a skip-gram network with 160 billion parameters overnight using only three multi-core CPUs.
- This represents a substantial advancement as it is 14 times larger than any previous neural network trained.
- The paper presents a groundbreaking approach to NLP by incorporating both word order and character order into neural word embeddings.
- The results demonstrate improved performance on word-analogy tasks and highlight the scalability potential of the proposed model for large-scale applications.

## Summary

The paper titled "Modeling Order in Neural Word Embeddings at Scale" by Andrew Trask, David Gilmore, and Matthew Russell explores the use of natural language processing (NLP) systems and their reliance on bag-of-words co-occurrence techniques to capture semantic and syntactic word relationships. While these techniques have proven valuable, they often ignore morphological information. To address this limitation, the authors propose a new neural language model that incorporates both word order and character order in its embedding. This novel approach results in the creation of several vector spaces with meaningful substructure. The effectiveness of the model is demonstrated through its performance on a recent word-analogy task, achieving an impressive accuracy rate of 85.8%. This performance surpasses the best published syntactic word-analogy scores by a significant margin of 58% error reduction. One notable aspect of the proposed model is its inclusion of several parallel training methods. Of particular importance is its ability to train a skip-gram network with an astounding 160 billion parameters overnight using only three multi-core CPUs. This represents a substantial advancement as it is 14 times larger than any previous neural network trained. Overall, this paper presents a groundbreaking approach to NLP by incorporating both word order and character order into neural word embeddings. The results demonstrate improved performance on word-analogy tasks and highlight the scalability potential of the proposed model through efficient parallel training methods which can be used for large scale applications.