# Janus: Decoupling Visual Encoding for Unified Multimodal Understanding and Generation

**Authors:** Chengyue Wu, Xiaokang Chen, Zhiyu Wu, Yiyang Ma, Xingchao Liu, Zizheng Pan, Wen Liu, Zhenda Xie, Xingkai Yu, Chong Ruan, Ping Luo  
**arXiv:** [2410.13848](https://arxiv.org/abs/2410.13848)  
**Date:** 2024-10-17  
**Summary page:** https://www.summarizepaper.com/en/paper/2410.13848v1/

## Key points

- Janus is an autoregressive framework that revolutionizes multimodal understanding and generation
- Decouples visual encoding into separate pathways while using a unified transformer architecture
- Allows for independent selection of encoding methods by both multimodal understanding and generation components
- Outperforms previous unified models and even matches or exceeds task-specific models' performance
- Utilizes a semantic tokenizer to enhance model accuracy through SigLIP supervision for semantic information reconstruction
- Simple, flexible, and effective solution for multimodal understanding and generation challenges

## Summary

, , , , 

In this paper, the authors introduce Janus, a groundbreaking autoregressive framework that revolutionizes multimodal understanding and generation. Unlike previous models, Janus decouples visual encoding into separate pathways while still utilizing a unified transformer architecture for processing. This innovative approach addresses conflicting information granularity requirements between multimodal understanding and generation, resulting in enhanced performance in both tasks. The concept of Janus is symbolically represented by the Roman god with two faces looking in opposite directions – one aged and wise, the other youthful and curious. Just like Janus gazes into the past and future simultaneously, this model allows for independent selection of encoding methods by both multimodal understanding and generation components. The flexibility provided by this decoupling not only resolves conflicts within the visual encoder but also enhances overall framework adaptability. Experimental results demonstrate that Janus outperforms previous unified models and even matches or exceeds task-specific models' performance. Its simplicity, high flexibility, and effectiveness position it as a strong contender for next-generation unified multimodal models. The detailed qualitative comparisons with diffusion-based models like SDXL and autoregressive model LlamaGen further highlight Janus's superiority in visual generation tasks. Moreover, the paper discusses the architecture of a semantic tokenizer used in an ablation study to enhance model performance further. By integrating pre-trained SigLIP supervision for semantic information reconstruction with raw image supervision for RGB values reconstruction, the semantic tokenizer aids in improving overall model accuracy. In conclusion, Janus emerges as a simple yet powerful solution for multimodal understanding and generation challenges. Its ability to alleviate conflicts between different task demands on the visual encoder showcases its potential for future advancements in multimodal modeling. With easy extensibility to incorporate more input modalities, Janus serves as an inspiration for developing next-generation general-purpose multimodal models that excel in diverse applications across various domains.