# Tranception: protein fitness prediction with autoregressive transformers and inference-time retrieval

**Authors:** Pascal Notin, Mafalda Dias, Jonathan Frazer, Javier Marchena-Hurtado, Aidan Gomez, Debora S. Marks, Yarin Gal  
**arXiv:** [2205.13760](https://arxiv.org/abs/2205.13760)  
**Date:** 2022-05-27  
**Summary page:** https://www.summarizepaper.com/en/paper/2205.13760v1/

## Key points

- Accurate modeling of protein sequence fitness landscapes is crucial for various applications
- Deep generative models trained on multiple sequence alignments have shown success in addressing challenges
- Large language models developed using non-aligned protein sequences show promise in bridging performance gap
- Tranception, a novel transformer architecture, leverages autoregressive predictions and retrieval of homologous sequences for state-of-the-art fitness prediction performance
- Tranception is trained on UniRef, a large-scale protein sequence database, with 700M parameters and approximately 250 million sequences
- Sequences in the multiple sequence alignment are re-weighted to address biases in protein databases
- ProteinGym is introduced as an extensive set of multiplexed assays for variant effects to facilitate more rigorous testing across a wider range of protein families

## Summary

The accurate modeling of protein sequence fitness landscapes is crucial for various applications such as assessing the impact of human variants on disease susceptibility and predicting immune-escape mutations in viruses. Deep generative models trained on multiple sequence alignments have shown significant success in addressing these challenges. However, their performance heavily relies on the availability of comprehensive and diverse alignments for effective training. To overcome this limitation, large language models have been developed using vast amounts of non-aligned protein sequences from diverse families. These models show promise in bridging the performance gap by offering a broader scope of application. One such advancement is Tranception, a novel transformer architecture that leverages autoregressive predictions and retrieval of homologous sequences during inference to achieve state-of-the-art fitness prediction performance. To enhance the training process, Tranception is trained on UniRef, a large-scale protein sequence database. Thorough ablations are conducted during its development to optimize its performance. Unlike previous observations with masked-language model architectures, it has been found that maintaining the granularity available in the dataset benefits downstream task performance. Furthermore, extensive data processing and augmentations are performed to refine the training dataset used for developing Tranception. The final model with 700M parameters is trained on UniRef100 after preprocessing, resulting in a training dataset comprising approximately 250 million sequences. To address biases in protein databases due to human and evolutionary sampling, sequences in the multiple sequence alignment (MSA) are re-weighted using a specific scheme. Additionally, an estimation method is employed to calculate log likelihoods for protein sequences based on weighted arithmetic averages derived from autoregressive and retrieval inference modes. Overall, Tranception demonstrates superior performance on multiple mutants and shows robustness to shallow alignments while also being capable of scoring indels compared to existing approaches. In order to facilitate more rigorous testing across a wider range of protein families, ProteinGym is introduced as an extensive set of multiplexed assays for variant effects that significantly increases both the number and diversity of assays compared to current benchmarks.