# Regularizing and Optimizing LSTM Language Models

**Authors:** Stephen Merity, Nitish Shirish Keskar, Richard Socher  
**arXiv:** [1708.02182](https://arxiv.org/abs/1708.02182)  
**Date:** 2017-08-07  
**Summary page:** https://www.summarizepaper.com/en/paper/1708.02182v1/

## Key points

- Authors: Stephen Merity, Nitish Shirish Keskar, Richard Socher
- Focus on LSTM-based models for word-level language modeling
- Innovation: Weight-dropped LSTM technique using DropConnect for recurrent regularization
- Introduction of NT-ASGD for automatic averaging triggering based on non-monotonic condition
- Achieved state-of-the-art performance on Penn Treebank (57.3 perplexity) and WikiText-2 (65.8 perplexity)
- Integration of neural cache further improved performance with perplexity scores of 52.8 and 52.0 respectively
- Demonstrates how innovative regularization techniques can enhance efficiency and accuracy in natural language processing tasks

## Summary

In their paper titled "Regularizing and Optimizing LSTM Language Models," authors Stephen Merity, Nitish Shirish Keskar, and Richard Socher explore the use of recurrent neural networks (RNNs), specifically long short-term memory networks (LSTMs), in various sequence learning tasks. RNNs are crucial for tasks such as machine translation, language modeling, and question answering. The focus of this study is word-level language modeling, where the researchers propose different strategies to enhance LSTM-based models. One key innovation is the weight-dropped LSTM technique which utilizes DropConnect on hidden-to-hidden weights for recurrent regularization. This approach aims to improve the robustness and generalization capabilities of LSTMs. Additionally, they introduce NT-ASGD, a novel variant of averaged stochastic gradient method that automatically determines when to trigger averaging using a non-monotonic condition. These regularization techniques contribute to achieving state-of-the-art performance on two benchmark datasets: 57.3 perplexity on Penn Treebank and 65.8 perplexity on WikiText-2. Furthermore, the authors integrate a neural cache with their proposed model to further enhance performance resulting in even lower state-of-the-art perplexity scores of 52.8 on Penn Treebank and 52.0 on WikiText-2. This research not only advances word-level language modeling but also demonstrates how innovative regularization techniques can significantly improve efficiency and accuracy in natural language processing tasks.