# Pretrain on just structure: Understanding linguistic inductive biases using transfer learning

**Authors:** Isabel Papadimitriou, Dan Jurafsky  
**arXiv:** [2304.13060](https://arxiv.org/abs/2304.13060)  
**Date:** 2023-04-25  
**Summary page:** https://www.summarizepaper.com/en/paper/2304.13060v1/

## Key points

- The study explores inductive learning biases in humans and transformer language models.
- The authors pretrain language models on artificial structured data and fine-tune them on English.
- Three types of inductive biases are examined: recursive, hierarchical processing; unrestricted token-token dependencies; and Zipfian power-law vocabulary distribution.
- Complex token-token interactions provide effective inductive biases, especially when context-free grammars are insufficient.
- A Zipfian vocabulary distribution serves as a good independent inductive bias regardless of grammatical structure.
- Transformer models enable controlled language learning experiments not feasible with human subjects.
- The findings contribute to understanding language learning and processing mechanisms in both humans and machines.

## Summary

In the study titled "Pretrain on just structure: Understanding linguistic inductive biases using transfer learning," authors Isabel Papadimitriou and Dan Jurafsky explore the inductive learning biases that enable both humans and transformer language models to learn language without explicit structural supervision. They investigate the effect of different inductive biases by pretraining language models on artificial structured data and then fine-tuning them on English. This experimental setup allows them to actively control the inductive bias of the language models. The authors specifically examine three types of inductive bias: 1) an inductive bias for recursive, hierarchical processing, 2) an inductive bias for unrestricted token-token dependencies that cannot be modeled by context-free grammars, and 3) an inductive bias for a Zipfian power-law vocabulary distribution. Through their experiments, they find that complex token-token interactions provide the most effective inductive biases, particularly in cases where context-free grammars are insufficient. Additionally, they discover that a Zipfian vocabulary distribution serves as a good independent inductive bias regardless of grammatical structure. This study leverages the capabilities of transformer models to conduct controlled language learning experiments which are not feasible with human subjects. The findings shed light on the structures that facilitate language learning in both humans and machines, generating hypotheses about their shared mechanisms. Overall, this research contributes to a deeper understanding of how language is learned and processed by exploring different types of inductive biases. It provides insight into how transfer learning can be used to understand linguistic induction in both humans and machines.