# Analysing Dropout and Compounding Errors in Neural Language Models

**Authors:** James O' Neill, Danushka Bollegala  
**arXiv:** [1811.00998](https://arxiv.org/abs/1811.00998)  
**Date:** 2018-11-02  
**Summary page:** https://www.summarizepaper.com/en/paper/1811.00998v1/

## Key points

- Empirical analysis of dropout techniques for language modelling
- Dropout techniques include Bernoulli dropout, Gaussian dropout, Curriculum Dropout, Variational Dropout, and Concrete Dropout
- Extension of variational dropout to concrete dropout and curriculum dropout with varying schedules proposed
- Variational curriculum dropout with a linear schedule performs well compared to standard approaches
- Largest performance improvements observed when applying dropout on the decoder layer
- Analysis of errors at test time to determine presence of compounding errors and how proposed methods mitigate this issue for each dataset
- Experiments conducted on different neural network architectures (2-hidden layer LSTM, GRU, Highway network) with various types of dropout applied
- Results reported on Penn-TreeBank and WikiText-2 word-level language modelling datasets
- Study provides insights into effectiveness of different dropout techniques in improving language modelling performance
- Recommendations for regularization settings when training neural language models offered

## Summary

, , , , 

This paper presents an empirical analysis of various dropout techniques for language modelling, including Bernoulli dropout, Gaussian dropout, Curriculum Dropout, Variational Dropout, and Concrete Dropout. The authors also propose an extension of variational dropout to concrete dropout and curriculum dropout with varying schedules. The results show that these extensions perform well compared to standard dropout approaches, particularly variational curriculum dropout with a linear schedule. The largest performance improvements are observed when applying dropout on the decoder layer. The study also analyzes where most errors occur at test time to determine if the problem of compounding errors is present and how the proposed methods mitigate this issue for each dataset. This is done by conducting experiments on different neural network architectures (2-hidden layer LSTM, GRU, and Highway network) with embedding dropout, dropout on gated hidden layers, and output projection layer. The results are reported on Penn-TreeBank and WikiText-2 word-level language modelling datasets. Overall, this study provides insights into the effectiveness of different dropout techniques in improving language modelling performance and offers recommendations for regularization settings when training neural language models.