# Do Llamas Work in English? On the Latent Language of Multilingual Transformers

**Authors:** Chris Wendler, Veniamin Veselovsky, Giovanni Monea, Robert West  
**arXiv:** [2402.10588](https://arxiv.org/abs/2402.10588)  
**Date:** 2024-02-16  
**Summary page:** https://www.summarizepaper.com/en/paper/2402.10588v1/

## Key points

- Study title: "Do Llamas Work in English? On the Latent Language of Multilingual Transformers"
- Authors: Chris Wendler, Veniamin Veselovsky, Giovanni Monea, Robert West
- Investigation of English as an internal pivot language by multilingual language models
- Experimentation with non-English prompts and single-token continuations to analyze transformer behavior at different layers
- Intermediate embeddings initially far from output token embeddings but moving closer to input-language-specific region over time
- Potential biases towards English in multilingual language models highlighted
- Importance of considering linguistic bias in development and application of these models

## Summary

In their study titled "Do Llamas Work in English? On the Latent Language of Multilingual Transformers," authors Chris Wendler, Veniamin Veselovsky, Giovanni Monea, and Robert West investigate the use of English as an internal pivot language by multilingual language models. This is crucial for understanding their operation and potential biases. The researchers conducted experiments using non-English prompts with unique correct single-token continuations to analyze transformer behavior at different layers. They found that intermediate embeddings initially started far from output token embeddings but eventually moved closer to an input-language-specific region within the embedding space. This suggests potential biases towards English in multilingual language models. The study sheds light on how these models process information and highlights the importance of considering linguistic bias in their development and application. , specifically the  of , are investigated by Wendler et al. to determine if they utilize . Their research involves carefully crafted non-English prompts with unique correct single-token continuations to analyze transformer behavior at different layers. The results show a movement towards an input-language-specific region within the embedding space, indicating potential biases towards English in these models. This study provides valuable insights into understanding latent language dynamics within transformer models and their implications for cross-lingual natural language processing tasks.