# Pushdown Layers: Encoding Recursive Structure in Transformer Language Models

**Authors:** Shikhar Murty, Pratyusha Sharma, Jacob Andreas, Christopher D. Manning  
**arXiv:** [2310.19089](https://arxiv.org/abs/2310.19089)  
**Date:** 2023-10-29  
**Summary page:** https://www.summarizepaper.com/en/paper/2310.19089v1/

## Key points

- Pushdown Layers introduced as a new self-attention layer for Transformer language models
- Address the challenge of capturing recursive structure in human language
- Use a stack tape to track estimated depths of tokens in an incremental parse
- Allow Transformer models to softly modulate attention and learn to "skip" over closed constituents
- Achieve significantly better syntactic generalization compared to standard Transformer models
- 3-5 times more sample-efficient than standard Transformer models
- WIKITREES dataset created consisting of over 100 million tokens from Wikipedia articles
- Pushdown Transformers exhibit drastically more sample-efficient syntactic generalization compared to base Transformers on WIKITREES dataset
- Staged finetuning of GPT2-medium with Pushdown Layers improves language understanding tasks beyond just syntactic generalization
- Replacing final 12 self-attention blocks with Pushdown Layers achieves better performance on several GLUE text classification tasks

## Summary

This work introduces Pushdown Layers, a new self-attention layer for Transformer language models that addresses the challenge of capturing recursive structure in human language. Recursion is a fundamental feature of language but is difficult to model with self-attention due to the lack of an explicit mechanism for tracking recursive states. Pushdown Layers solve this problem by using a stack tape to track estimated depths of tokens in an incremental parse of the observed prefix. With Pushdown Layers, Transformer models can autoregressively update the stack tape as they predict new tokens, allowing them to softly modulate attention over tokens and learn to "skip" over closed constituents. The authors trained Transformers equipped with Pushdown Layers on a corpus of strings annotated with silver constituency parses and found that these models achieve significantly better syntactic generalization and are 3-5 times more sample-efficient compared to standard Transformer language models. To further evaluate the effectiveness of Pushdown Layers, the authors created a dataset called WIKITREES consisting of over 100 million tokens extracted from Wikipedia articles. They trained Pushdown Transformers on different amounts of data from WIKITREES and measured their sample efficiency in syntactic generalization tasks. The results showed that Pushdown Transformers exhibit drastically more sample-efficient syntactic generalization compared to base Transformers. Additionally, the authors performed staged finetuning of GPT2-medium with Pushdown Layers and observed improvements in language understanding tasks beyond just syntactic generalization. By replacing the final 12 self-attention blocks with Pushdown Layers, they achieved better performance on several GLUE text classification tasks. Overall, this work demonstrates that Pushdown Layers offer improvements in modeling recursive structure and can enhance both syntactic generalization and language understanding tasks in large-scale language modeling scenarios.