MM1: Methods, Analysis & Insights from Multimodal LLM Pre-training

Ranked by how close their title and abstract are to this paper’s. Select some to read them together.

Closest papers

1. Very close Cites this paper Uses this paper in depth Not relevant

   MM1.5: Methods, Analysis & Insights from Multimodal LLM Fine-tuning

   Hao-Tian Zhang, Ming-Fei Gao, Zhe Gan et al. · 2024 · arXiv.org · preprint, not peer reviewed · cited 80 times · Not summarized yet

   Abstract

   We present MM1.5, a new family of multimodal large language models (MLLMs) designed to enhance capabilities in text-rich image understanding, visual referring and grounding, and multi-image reasoning. Building upon the MM1 architecture, MM1.5 adopts a data-centric approach to model training, systematically exploring the impact of diverse data mixtures across the entire model training lifecycle. This includes high-quality OCR data and synthetic captions for continual pre-training, as well as an optimized visual instruction-tuning data mixture for supervised fine-tuning. Our models range from 1B to 30B parameters, encompassing both dense and mixture-of-experts (MoE) variants, and demonstrate t…

2. Very close Cited by this paper Cites this paper Not relevant

   Efficient Multimodal Learning from Data-centric Perspective

   Muyang He, Yexin Liu, Bo-Ya Wu et al. · 2024 · arXiv.org · preprint, not peer reviewed · cited 168 times · Not summarized yet

   Abstract

   Multimodal Large Language Models (MLLMs) have demonstrated notable capabilities in general visual understanding and reasoning tasks. However, their deployment is hindered by substantial computational costs in both training and inference, limiting accessibility to the broader research and user communities. A straightforward solution is to leverage smaller pre-trained vision and language models, which inevitably cause significant performance drops. In this paper, we demonstrate the possibility of training a smaller but better MLLM with high-quality training data. Specifically, we introduce Bunny, a family of lightweight MLLMs with flexible vision and language backbones for efficient multimodal…

3. Very close Cites this paper Not relevant

   A Survey on Multimodal Large Language Models

   Shukang Yin, Chaoyou Fu, Sirui Zhao et al. · 2023 · Computer Vision and Pattern Recognition · cited 1,744 times · AI summary available

   Abstract

   Multimodal Large Language Model (MLLM) recently has been a new rising research hotspot, which uses powerful Large Language Models (LLMs) as a brain to perform multimodal tasks. The surprising emergent capabilities of MLLM, such as writing stories based on images and OCR-free math reasoning, are rare in traditional methods, suggesting a potential path to artificial general intelligence. In this paper, we aim to trace and summarize the recent progress of MLLM. First of all, we present the formulation of MLLM and delineate its related concepts. Then, we discuss the key techniques and applications, including Multimodal Instruction Tuning (M-IT), Multimodal In-Context Learning (M-ICL), Multimodal…

4. Very close Not relevant

   MMRL: Multi-Modal Representation Learning for Vision-Language Models

   Yuncheng Guo, Xiaodong Gu · 2025 · Machine Learning · cited 70 times · Not summarized yet

   Abstract

   Large-scale pre-trained Vision-Language Models (VLMs) have become essential for transfer learning across diverse tasks. However, adapting these models with limited few-shot data often leads to overfitting, diminishing their performance on new tasks. To tackle this issue, we propose a novel Multi-Modal Representation Learning (MMRL) framework that introduces a shared, learnable, and modality-agnostic representation space. MMRL projects the space tokens to text and image representation tokens, facilitating more effective multi-modal interactions. Unlike previous approaches that solely optimize class token features, MMRL integrates representation tokens at higher layers of the encoders--where d…

5. Very close Cites this paper Not relevant

   Mono-InternVL: Pushing the Boundaries of Monolithic Multimodal Large Language Models with Endogenous Visual Pre-training

   G. Luo, Xue Yang, Wen-Han Dou et al. · 2024 · Computer Vision and Pattern Recognition · cited 103 times · Not summarized yet

   Abstract

   In this paper, we focus on monolithic Multimodal Large Language Models (MLLMs) that integrate visual encoding and language decoding into a single LLM. In particular, we identify that existing pre-training strategies for monolithic MLLMs often suffer from unstable optimization or catastrophic forgetting. To address this issue, our core idea is to embed a new visual parameter space into a pre-trained LLM, thereby stably learning visual knowledge from noisy data while freezing the LLM. Based on this principle, we present Mono-InternVL, a novel monolithic MLLM that seamlessly integrates a set of visual experts via a multimodal mixture-of-experts structure. Moreover, we propose an innovative pre-…

6. Very close Cites this paper Uses this paper in depth Not relevant

   Accelerating Pre-training of Multimodal LLMs via Chain-of-Sight

   Zi-Yuan Huang, Kaixiang Ji, Biao Gong et al. · 2024 · Neural Information Processing Systems · cited 9 times · Not summarized yet

   Abstract

   This paper introduces Chain-of-Sight, a vision-language bridge module that accelerates the pre-training of Multimodal Large Language Models (MLLMs). Our approach employs a sequence of visual resamplers that capture visual details at various spacial scales. This architecture not only leverages global and local visual contexts effectively, but also facilitates the flexible extension of visual tokens through a compound token scaling strategy, allowing up to a 16x increase in the token count post pre-training. Consequently, Chain-of-Sight requires significantly fewer visual tokens in the pre-training phase compared to the fine-tuning phase. This intentional reduction of visual tokens during pre-…

7. Very close Cites this paper Not relevant

   A Survey of Multimodal Large Language Model from A Data-centric Perspective

   Tianyi Bai, Hao Liang, Binwang Wan et al. · 2024 · arXiv.org · preprint, not peer reviewed · cited 100 times · Not summarized yet

   Abstract

   Multimodal large language models (MLLMs) enhance the capabilities of standard large language models by integrating and processing data from multiple modalities, including text, vision, audio, video, and 3D environments. Data plays a pivotal role in the development and refinement of these models. In this survey, we comprehensively review the literature on MLLMs from a data-centric perspective. Specifically, we explore methods for preparing multimodal data during the pretraining and adaptation phases of MLLMs. Additionally, we analyze the evaluation methods for the datasets and review the benchmarks for evaluating MLLMs. Our survey also outlines potential future research directions. This work …

8. Very close Not relevant

   Breaking the Modality Barrier: Universal Embedding Learning with Multimodal LLMs

   Tian-Cheng Gu, Kai-Cheng Yang, Zi-Yong Feng et al. · 2025 · ACM Multimedia · cited 89 times · Not summarized yet

   Abstract

   The Contrastive Language-Image Pre-training (CLIP) framework has become a widely used approach for multimodal representation learning, particularly in image-text retrieval and clustering. However, its efficacy is constrained by three key limitations: (1) text token truncation, (2) isolated image-text encoding, and (3) deficient compositionality due to bag-of-words behavior. While recent Multimodal Large Language Models (MLLMs) have demonstrated significant advances in generalized vision-language understanding, their potential for learning transferable multimodal representations remains underexplored. In this work, we present UniME (Universal Multimodal Embedding), a novel two-stage framework…

Prior work it builds on

1. Close Used in depth by this paper A method it uses Not relevant

   Improved Baselines with Visual Instruction Tuning

   Haotian Liu, Chunyuan Li, Yuheng Li et al. · 2023 · Computer Vision and Pattern Recognition · cited 6,154 times · AI summary available

   Abstract

   Large multimodal models (LMM) have recently shown encouraging progress with visual instruction tuning. In this note, we show that the fully-connected vision-language cross-modal connector in LLaVA is surprisingly powerful and data-efficient. With simple modifications to LLaVA, namely, using CLIP-ViT-L-336px with an MLP projection and adding academic-task-oriented VQA data with simple response formatting prompts, we establish stronger baselines that achieve state-of-the-art across 11 benchmarks. Our final 13B checkpoint uses merely 1.2M publicly available data, and finishes full training in ~1 day on a single 8-A100 node. We hope this can make state-of-the-art LMM research more accessible. Co…

2. Very close A method it uses Not relevant

   mPLUG-Owl: Modularization Empowers Large Language Models with Multimodality

   Qinghao Ye, Haiyang Xu, Guohai Xu et al. · 2023 · Computation and Language · preprint, not peer reviewed · cited 1,318 times · AI summary available

   Abstract

   Large language models (LLMs) have demonstrated impressive zero-shot abilities on a variety of open-ended tasks, while recent research has also explored the use of LLMs for multi-modal generation. In this study, we introduce mPLUG-Owl, a novel training paradigm that equips LLMs with multi-modal abilities through modularized learning of foundation LLM, a visual knowledge module, and a visual abstractor module. This approach can support multiple modalities and facilitate diverse unimodal and multimodal abilities through modality collaboration. The training paradigm of mPLUG-Owl involves a two-stage method for aligning image and text, which learns visual knowledge with the assistance of LLM whil…

3. Very close A method it uses Not relevant

   Eyes Wide Shut? Exploring the Visual Shortcomings of Multimodal LLMs

   Shengbang Tong, Zhuang Liu, Yuexiang Zhai et al. · 2024 · Computer Vision and Pattern Recognition · cited 988 times · Not summarized yet

   Abstract

   Is vision good enough for language? Recent advancements in multimodal models primarily stem from the powerful reasoning abilities of large language models (LLMs). However, the visual component typically depends only on the instance-level contrastive language-image pre-training (CLIP). Our research reveals that the visual capabilities in recent MultiModal LLMs (MLLMs) still exhibit systematic shortcomings. To understand the roots of these errors, we explore the gap between the visual embedding space of CLIP and vision-only self-supervised learning. We identify “CLIP-blind pairs”- images that CLIP perceives as similar despite their clear visual differences. With these pairs, we construct the M…

4. Close A method it uses Not relevant

   SPHINX: The Joint Mixing of Weights, Tasks, and Visual Embeddings for Multi-modal Large Language Models

   Ziyi Lin, Chris Liu, Renrui Zhang et al. · 2023 · Computer Vision and Pattern Recognition · preprint, not peer reviewed · cited 316 times · AI summary available

   Abstract

   We present SPHINX, a versatile multi-modal large language model (MLLM) with a joint mixing of model weights, tuning tasks, and visual embeddings. First, for stronger vision-language alignment, we unfreeze the large language model (LLM) during pre-training, and introduce a weight mix strategy between LLMs trained by real-world and synthetic data. By directly integrating the weights from two domains, the mixed LLM can efficiently incorporate diverse semantics with favorable robustness. Then, to enable multi-purpose capabilities, we mix a variety of tasks for joint visual instruction tuning, and design task-specific instructions to avoid inter-task conflict. In addition to the basic visual ques…

5. Close Used in depth by this paper A method it uses Not relevant

   Generative Multimodal Models are In-Context Learners

   Quan Sun, Yufeng Cui, Xiaosong Zhang et al. · 2023 · Computer Vision and Pattern Recognition · cited 568 times · Not summarized yet

   Abstract

   The human ability to easily solve multimodal tasks in context (i.e., with only a few demonstrations or simple instructions), is what current multimodal systems have largely struggled to imitate. In this work, we demonstrate that the task-agnostic in-context learning capabilities of large multimodal models can be significantly enhanced by effective scaling-up. We introduce Emu2, a generative multimodal model with 37 billion parameters, trained on large-scale multimodal sequences with a unified autoregressive objective. Emu2 exhibits strong multimodal in-context learning abilities, even emerging to solve tasks that require on-the-fly reasoning, such as visual prompting and object-grounded gene…

Papers that cite it

1. Very close Uses this paper in depth Not relevant

   Cambrian-1: A Fully Open, Vision-Centric Exploration of Multimodal LLMs

   Shengbang Tong, Ellis Brown, Peng-Hao Wu et al. · 2024 · Neural Information Processing Systems · cited 1,062 times · Not summarized yet

   Abstract

   We introduce Cambrian-1, a family of multimodal LLMs (MLLMs) designed with a vision-centric approach. While stronger language models can enhance multimodal capabilities, the design choices for vision components are often insufficiently explored and disconnected from visual representation learning research. This gap hinders accurate sensory grounding in real-world scenarios. Our study uses LLMs and visual instruction tuning as an interface to evaluate various visual representations, offering new insights into different models and architectures -- self-supervised, strongly supervised, or combinations thereof -- based on experiments with over 20 vision encoders. We critically examine existing M…

2. Very close Uses this paper in depth Not relevant

   Efficient multimodal large language models: a survey

   Yizhang Jin, Jian Li, Yexin Liu et al. · 2024 · Visual Intelligence · cited 165 times · Not summarized yet

   Abstract

   In the past years, multimodal large language models (MLLMs) have demonstrated remarkable performance in tasks such as visual question answering and visual understanding and reasoning. However, the extensive model size and high training and inference costs have hindered the widespread application of MLLMs in academia and industry. Thus, studying efficient and lightweight MLLMs has enormous potential, especially in edge computing scenarios. In this survey, we provide a comprehensive and systematic review of the current state of efficient MLLMs. Specifically, this survey summarizes the timeline of representative efficient MLLMs, the current state of research in structures and strategies, and th…

3. Very close Uses this paper in depth Not relevant

   BLIP-3: A Family of Open Large Multimodal Models

   Le Xue, Manli Shu, Anas Awadalla et al. · 2024 · 2025 IEEE/CVF International Conference on Computer Vision Workshops (ICCVW) · cited 129 times · Not summarized yet

   Abstract

   This paper introduces BLIP-3, an open framework for developing Large Multimodal Models (LMMs). The framework comprises meticulously curated datasets, a training recipe, model architectures, and a resulting suite of LMMs. We release 4B and 14B models, including both the pre-trained base model and the instruction fine-tuned ones. Our models undergo rigorous evaluation across a range of tasks, including both single and multi-image benchmarks. Our models demonstrate competitive performance among open-source LMMs with similar model sizes. Our resulting LMMs demonstrate competitive performance among open-source LMMs with similar model sizes, with the ability to comprehend interleaved image-text in…

4. Very close Uses this paper in depth Not relevant

   Multimodal Autoregressive Pre-training of Large Vision Encoders

   Enrico Fini, Mustafa Shukor, Xiujun Li et al. · 2024 · Computer Vision and Pattern Recognition · cited 140 times · Not summarized yet

   Abstract

   We introduce a novel method for pre-training of large-scale vision encoders. Building on recent advancements in autoregressive pre-training of vision models, we extend this framework to a multimodal setting, i.e., images and text. In this paper, we present AIMV2, a family of generalist vision encoders characterized by a straightforward pre-training process, scalability, and remarkable performance across a range of downstream tasks. This is achieved by pairing the vision encoder with a multimodal decoder that autoregressively generates raw image patches and text tokens. Our encoders excel not only in multimodal evaluations but also in vision benchmarks such as localization, grounding, and cla…

5. Close Uses this paper in depth Not relevant

   How far are we to GPT-4V? Closing the gap to commercial multimodal models with open-source suites

   Zhe Chen, Wei-Yun Wang, Hao Tian et al. · 2024 · Science China Information Sciences · cited 1,330 times · Not summarized yet

   Abstract

   In this paper, we introduce InternVL 1.5, an open-source multimodal large language model (MLLM) to bridge the capability gap between open-source and proprietary commercial models in multimodal understanding. We introduce three simple improvements. (1) Strong vision encoder: we explored a continuous learning strategy for the large-scale vision foundation model — InternViT-6B, boosting its visual understanding capabilities, and making it can be transferred and reused in different LLMs. (2) Dynamic high-resolution: we divide images into tiles ranging from 1 to 40 of 448×448 pixels according to the aspect ratio and resolution of the input images, which supports up to 4K resolution input. (3) Hig…