# SPHINX: The Joint Mixing of Weights, Tasks, and Visual Embeddings for Multi-modal Large Language Models

**Authors:** Ziyi Lin, Chris Liu, Renrui Zhang, Peng Gao, Longtian Qiu, Han Xiao, Han Qiu, Chen Lin, Wenqi Shao, Keqin Chen, Jiaming Han, Siyuan Huang, Yichi Zhang, Xuming He, Hongsheng Li, Yu Qiao  
**arXiv:** [2311.07575](https://arxiv.org/abs/2311.07575)  
**Date:** 2023-11-13  
**Summary page:** https://www.summarizepaper.com/en/paper/2311.07575v1/

## Key points

- SPHINX is a versatile multi-modal large language model (MLLM) that enhances vision-language alignment and enables multi-purpose capabilities in language models.
- SPHINX unfreezes the large language model (LLM) during pre-training to achieve stronger vision-language alignment.
- SPHINX incorporates a weight mix strategy between LLMs trained on real-world and synthetic data to efficiently incorporate diverse semantics while maintaining robustness.
- SPHINX focuses on enabling multi-purpose capabilities through a variety of mixed tasks for joint visual instruction tuning, including region-level understanding, caption grounding, document layout detection, and human pose estimation.
- SPHINX proposes extracting comprehensive visual embeddings from various network architectures, pre-training paradigms, and information granularity to provide more robust image representations.
- SPHINX demonstrates superior multi-modal understanding capabilities across a wide range of applications based on the proposed joint mixing approach.
- An efficient strategy is introduced to improve performance on high-resolution images by mixing different scales and high-resolution sub-images. This allows SPHINX to attain exceptional visual parsing and reasoning performance on existing evaluation benchmarks.

## Summary

, , , , 

SPHINX is a versatile multi-modal large language model (MLLM) that incorporates a joint mixing of model weights, tuning tasks, and visual embeddings. The goal of SPHINX is to enhance vision-language alignment and enable multi-purpose capabilities in language models. To achieve stronger vision-language alignment, SPHINX unfreezes the large language model (LLM) during pre-training. It introduces a weight mix strategy between LLMs trained on real-world and synthetic data, efficiently incorporating diverse semantics while maintaining robustness. In addition to vision-language alignment, SPHINX also focuses on enabling multi-purpose capabilities through a variety of mixed tasks for joint visual instruction tuning. These task-specific instructions are designed to avoid inter-task conflict and include challenging tasks such as region-level understanding, caption grounding, document layout detection, and human pose estimation. This contributes to mutual enhancement across different scenarios. Furthermore, SPHINX proposes extracting comprehensive visual embeddings from various network architectures, pre-training paradigms, and information granularity. This provides language models with more robust image representations. Based on the proposed joint mixing approach, SPHINX demonstrates superior multi-modal understanding capabilities across a wide range of applications. To further improve performance on high-resolution images, an efficient strategy is introduced involving mixing different scales and high-resolution sub-images. This allows SPHINX to attain exceptional visual parsing and reasoning performance on existing evaluation benchmarks.