# Griffin: Mixing Gated Linear Recurrences with Local Attention for Efficient Language Models

**Authors:** Soham De, Samuel L. Smith, Anushan Fernando, Aleksandar Botev, George Cristian-Muraru, Albert Gu, Ruba Haroun, Leonard Berrada, Yutian Chen, Srivatsan Srinivasan, Guillaume Desjardins, Arnaud Doucet, David Budden, Yee Whye Teh, Razvan Pascanu, Nando De Freitas, Caglar Gulcehre  
**arXiv:** [2402.19427](https://arxiv.org/abs/2402.19427)  
**Date:** 2024-02-29  
**Summary page:** https://www.summarizepaper.com/en/paper/2402.19427v1/

## Key points

- Authors introduce Hawk and Griffin, two novel recurrent neural network (RNN) models
- Hawk features gated linear recurrences, while Griffin combines gated linear recurrences with local attention mechanisms
- Hawk outperforms Mamba on downstream tasks, while Griffin matches the performance of Llama-2 with fewer tokens and showcases extrapolation ability on longer sequences
- Models exhibit hardware efficiency comparable to Transformers during training and offer lower latency and higher throughput during inference
- Griffin successfully scaled up to 14 billion parameters and insights provided on sharding for efficient distributed training
- Evaluation shows superior performance of Griffin in terms of both latency and throughput compared to MQA Transformer

## Summary

In their paper titled "Griffin: Mixing Gated Linear Recurrences with Local Attention for Efficient Language Models," authors Soham De, Samuel L. Smith, Anushan Fernando, and their team introduce Hawk and Griffin - two novel recurrent neural network (RNN) models designed to address the challenges of training and scaling RNNs. <br>
<Hawk>Hawk</Hawk> features gated linear recurrences while <Griffin>Griffin</Griffin> combines gated linear recurrences with local attention mechanisms. The authors demonstrate that <Hawk>Hawk</Hawk> outperforms Mamba on downstream tasks while <Griffin>Griffin</Griffin> matches the performance of Llama-2 despite being trained on significantly fewer tokens. Notably, <Griffin>Griffin</Griffin> showcases the ability to extrapolate on sequences longer than those encountered during training. The models exhibit hardware efficiency comparable to Transformers during training and offer lower latency and higher throughput during inference.<br>
Furthermore, the authors successfully scale up <Griffin>Griffin</Griffin> to 14 billion parameters and provide insights into sharding the models for efficient distributed training. In their evaluation of models with 1 billion parameters, they compare the latency and throughput of a MQA Transformer against <Hawk>Hawk</Hawk> and <Griffin>Griffin</Griffin>. The results highlight the superior performance of <Griffin> Griffin </griffen > in terms of both latency and throughput.<br>
Overall, the study presents <Hawk>Hawk</Hawk> and <griffen > griffen </griffen > as promising solutions for fast training at scale in language modeling tasks - showcasing their effectiveness in surpassing existing benchmarks while maintaining hardware efficiency and scalability.