# UniT: Multimodal Multitask Learning with a Unified Transformer

**Authors:** Ronghang Hu, Amanpreet Singh  
**arXiv:** [2102.10772](https://arxiv.org/abs/2102.10772)  
**Date:** 2021-02-22  
**Summary page:** https://www.summarizepaper.com/en/paper/2102.10772v3/

## Key points

- The UniT model is a unified transformer architecture that can learn multiple tasks across different domains simultaneously.
- It shares the same model parameters across all tasks instead of separately fine-tuning task-specific models.
- The UniT model uses an encoder-decoder architecture to encode each input modality with an encoder and make predictions on each task with a shared decoder over the encoded input representations, followed by task-specific output heads.
- The image encoder encodes visual inputs, while the text encoder encodes language inputs.
- UniT handles all tasks in a shared model without losing general knowledge across domains due to fine-tuning on specific downstream tasks.
- In experiments conducted using UniT, seven tasks were learned jointly over eight datasets, achieving strong performance on each task with significantly fewer parameters than previous methods.
- The UniT model allows for end-to-end training directly over image pixels instead of relying on pretrained detectors.

## Summary

The UniT model is a unified transformer architecture that can simultaneously learn multiple tasks across different domains, including object detection, natural language understanding, and multimodal reasoning. Unlike previous multi-task learning efforts with transformers, the UniT model shares the same model parameters across all tasks instead of separately fine-tuning task-specific models. This approach allows for handling a much higher variety of tasks across different domains while achieving strong performance on each task with significantly fewer parameters. The UniT model uses an encoder-decoder architecture to encode each input modality with an encoder and make predictions on each task with a shared decoder over the encoded input representations, followed by task-specific output heads. The entire model is jointly trained end-to-end with losses from each task. The image encoder encodes visual inputs, while the text encoder encodes language inputs. The joint decoder uses per-task query embedding followed by task-specific heads to make the final outputs for each task. Compared to multimodal pretraining approaches such as VirTex, Voken, and VisualBERT that build specialized models through fine-tuning on each downstream task after pretraining on multimodal data such as image captions, UniT handles all tasks in a shared model without losing general knowledge across domains due to fine-tuning on specific downstream tasks. This ability to jointly solve different tasks across domains is a critical step towards general intelligence. In experiments conducted using UniT, seven tasks were learned jointly over eight datasets, achieving strong performance on each task with significantly fewer parameters than previous methods. Additionally, the UniT model allows for end-to-end training directly over image pixels instead of relying on pretrained detectors. Overall, the UniT model provides a promising approach for handling multiple modalities and inputs in multi-task learning scenarios while maintaining high performance levels and reducing computational complexity.