# Distributed Representations of Words and Phrases and their Compositionality

**Authors:** Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg Corrado, Jeffrey Dean  
**arXiv:** [1310.4546](https://arxiv.org/abs/1310.4546)  
**Date:** 2013-10-16  
**Summary page:** https://www.summarizepaper.com/en/paper/1310.4546v1/

## Key points

- The paper introduces the continuous Skip-gram model for learning distributed vector representations of words and phrases.
- Several extensions are presented to improve vector quality and training speed.
- Subsampling frequent words speeds up training and enhances word representations' regularity.
- Negative sampling is proposed as an alternative to hierarchical softmax, addressing limitations in representing word order and idiomatic phrases.
- A method for finding phrases in text is introduced, allowing for good vector representations of millions of phrases.

## Summary

The paper titled "Distributed Representations of Words and Phrases and their Compositionality" by Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg Corrado, and Jeffrey Dean introduces the continuous Skip-gram model as an efficient method for learning high-quality distributed vector representations that capture precise syntactic and semantic word relationships. The authors present several extensions to improve the quality of the vectors and training speed. One extension involves subsampling frequent words which not only speeds up the training process but also enhances the regularity of word representations. Additionally, the paper describes a simple alternative to hierarchical softmax called negative sampling. This technique addresses an inherent limitation of word representations – their indifference to word order and inability to represent idiomatic phrases. They propose a straightforward method for finding phrases in text demonstrating that it is possible to learn good vector representations for millions of phrases. In summary, this paper presents advancements in the continuous Skip-gram model that enhance both vector quality and training speed. It introduces techniques such as subsampling frequent words and negative sampling as alternatives to hierarchical softmax. Furthermore, it addresses the limitations of word representations regarding word order and idiomatic phrases by proposing a method for identifying phrases in text.