# A Study on Neural Network Language Modeling

**Authors:** Dengliang Shi  
**arXiv:** [1708.07252](https://arxiv.org/abs/1708.07252)  
**Date:** 2017-08-24  
**Summary page:** https://www.summarizepaper.com/en/paper/1708.07252v1/

## Key points

- Exhaustive study on neural network language modeling (NNLM)
- Different architectures of basic neural network language models
- Various improvements over these models, including importance sampling, word classes, caching, and bidirectional recurrent neural networks (BiRNN)
- Thorough evaluation of advantages and disadvantages of each technique
- Loss of statistical information when processing word sequences in a certain order
- Restrictions imposed by training mechanism through weight matrixes and vectors
- Neural network language models represent approximate probabilistic distribution but do not capture intrinsic knowledge or information conveyed by word sequences in natural language
- Directions for further improving neural network language modeling

## Summary

This paper presents an exhaustive study on neural network language modeling (NNLM). The authors describe and examine different architectures of basic neural network language models, and also investigate various improvements over these models, including importance sampling, word classes, caching, and bidirectional recurrent neural networks (BiRNN). The advantages and disadvantages of each technique are thoroughly evaluated. The study also explores the limitations of neural network language modeling from two perspectives: model architecture and knowledge representation. It is observed that when a word sequence is processed word by word in a certain order, part of the statistical information is lost. Additionally, the mechanism of training neural networks through weight matrixes and vectors imposes significant restrictions on enhancing NNLM. Regarding knowledge representation, it is found that neural network language models represent the approximate probabilistic distribution of word sequences from a specific training dataset. However, they do not capture the intrinsic knowledge of a language or the information conveyed by word sequences in natural language. Finally, the paper discusses directions for further improving neural network language modeling. With this additional context provided by this summary expansion, readers gain a deeper understanding of the research conducted in this study and can identify potential areas for further exploration to improve NNLM performance.