# Evade the Trap of Mediocrity: Promoting Diversity and Novelty in Text Generation via Concentrating Attention

**Authors:** Wenhao Li, Xiaoyuan Yi, Jinyi Hu, Maosong Sun, Xing Xie  
**arXiv:** [2211.07164](https://arxiv.org/abs/2211.07164)  
**Date:** 2022-11-14  
**Summary page:** https://www.summarizepaper.com/en/paper/2211.07164v1/

## Key points

- Transformer architectures are powerful tools for generating high-quality sentences
- However, they often produce repetitive and dull phrases that limit diversity and novelty of generated text
- Researchers conducted empirical and theoretical analyses to investigate the intrinsic mechanism behind this issue
- They discovered that sparser attention values in Transformers could improve diversity by avoiding representation degeneration caused by the attentive mixture of hidden states during training
- To address this problem, they introduced a novel attention regularization loss that controls the sharpness of the attention distribution, which is transparent to model structures and can be easily implemented within 20 lines of Python code.
- Their method significantly improved the diversity and novelty of generated text while maintaining comparable quality on various conditional and unconditional generation tasks.
- In particular, their model outperformed GPT-2 in generating relevant and novel expressions related to specific topics such as volleyball games.
- The paper also discusses related work on enhancing diversity in natural language generation (NLG), including incorporating randomization into decoding algorithms or substituting or supplementing maximum likelihood estimation (MLE) loss with novel objectives such as reinforcement learning or adversarial training.
- They compare their proposed method with other baselines using ROC curves to evaluate both quality and diversity metrics on NLG tasks.
- The results show that their method achieves higher diversity scores without sacrificing quality compared to other methods.
- Their approach modifies attention mechanisms to handle NLG diversity issues by concentrating on sparse attention distributions rather than scattered ones.

## Summary

In recent years, Transformer architectures have emerged as powerful tools for generating high-quality sentences. However, these models often produce repetitive and dull phrases that severely limit the diversity and novelty of generated text. To address this problem, a team of researchers conducted empirical and theoretical analyses to investigate the intrinsic mechanism behind this issue. They discovered that sparser attention values in Transformers could improve diversity by avoiding representation degeneration caused by the attentive mixture of hidden states during training. To escape from this trap, the researchers introduced a novel attention regularization loss that controls the sharpness of the attention distribution. This method is transparent to model structures and can be easily implemented within 20 lines of Python code. The researchers proved that this approach could be mathematically regarded as learning a Bayesian approximation of posterior attention. In their experiments, they found that their method significantly improved the diversity and novelty of generated text while maintaining comparable quality on various conditional and unconditional generation tasks. In particular, their model outperformed GPT-2 in generating relevant and novel expressions related to specific topics such as volleyball games. The paper also discusses related work on enhancing diversity in natural language generation (NLG), including incorporating randomization into decoding algorithms or substituting or supplementing maximum likelihood estimation (MLE) loss with novel objectives such as reinforcement learning or adversarial training. Furthermore, the paper presents a case study where they compare their proposed method with other baselines using ROC curves to evaluate both quality and diversity metrics on NLG tasks. The results show that their method achieves higher diversity scores without sacrificing quality compared to other methods. Finally, they provide additional insights into how their approach modifies attention mechanisms to handle NLG diversity issues by concentrating on sparse attention distributions rather than scattered ones. Overall, this research offers valuable contributions towards improving NLG systems' ability to generate diverse and novel content while maintaining high-quality standards.