Chameleon: Mixed-Modal Early-Fusion Foundation Models

Ranked by how close their title and abstract are to this paper’s. Select some to read them together.

Closest papers

1. Very close Cites this paper Uses this paper in depth Not relevant

   CheXmix: Unified Generative Pretraining for Vision Language Models in Medical Imaging

   Ashwin Kumar, Robbie Holland, Corey D. Barrett et al. · 2026 · 2026 IEEE/CVF Conference on Computer Vision and Pattern Recognition- FINDINGS Track (CVPRF) · cited 2 times · Not summarized yet

   Abstract

   Recent medical multimodal foundation models are built as multimodal LLMs (MLLMs) by connecting a CLIPpretrained vision encoder to an LLM using LLaVA-style finetuning. This two-stage, decoupled approach introduces a projection layer that can distort visual features. This is especially concerning in medical imaging where subtle cues are essential for accurate diagnoses. In contrast, earlyfusion generative approaches such as Chameleon eliminate the projection bottleneck by processing image and text tokens within a single unified sequence, enabling joint representation learning that leverages the inductive priors of language models. We present CheXmix, a unified early-fusion generative model tra…

2. Very close Not relevant

   Generative Pretraining in Multimodality

   Quan Sun, Qiying Yu, Yufeng Cui et al. · 2023 · Computer Vision and Pattern Recognition · cited 170 times · AI summary available

   Abstract

   We present Emu, a Transformer-based multimodal foundation model, which can seamlessly generate images and texts in multimodal context. This omnivore model can take in any single-modality or multimodal data input indiscriminately (e.g., interleaved image, text and video) through a one-model-for-all autoregressive training process. First, visual signals are encoded into embeddings, and together with text tokens form an interleaved input sequence. Emu is then end-to-end trained with a unified objective of classifying the next text token or regressing the next visual embedding in the multimodal sequence. This versatile multimodality empowers the exploration of diverse pretraining data sources at…

3. Close Cited by this paper Not relevant

   Scaling Autoregressive Multi-Modal Models: Pretraining and Instruction Tuning

   L. Yu, Bo-Wen Shi, Ramakanth Pasunuru et al. · 2023 · arXiv.org · preprint, not peer reviewed · cited 184 times · Not summarized yet

   Abstract

   We present CM3Leon (pronounced"Chameleon"), a retrieval-augmented, token-based, decoder-only multi-modal language model capable of generating and infilling both text and images. CM3Leon uses the CM3 multi-modal architecture but additionally shows the extreme benefits of scaling up and tuning on more diverse instruction-style data. It is the first multi-modal model trained with a recipe adapted from text-only language models, including a large-scale retrieval-augmented pre-training stage and a second multi-task supervised fine-tuning (SFT) stage. It is also a general-purpose model that can do both text-to-image and image-to-text generation, allowing us to introduce self-contained contrastive …

4. Close Not relevant

   mPLUG-2: A Modularized Multi-modal Foundation Model Across Text, Image and Video

   Haiyang Xu, Qinghao Ye, Ming Yan et al. · 2023 · ICML2023 · cited 257 times · AI summary available

   Abstract

   Recent years have witnessed a big convergence of language, vision, and multi-modal pretraining. In this work, we present mPLUG-2, a new unified paradigm with modularized design for multi-modal pretraining, which can benefit from modality collaboration while addressing the problem of modality entanglement. In contrast to predominant paradigms of solely relying on sequence-to-sequence generation or encoder-based instance discrimination, mPLUG-2 introduces a multi-module composition network by sharing common universal modules for modality collaboration and disentangling different modality modules to deal with modality entanglement. It is flexible to select different modules for different unders…

5. Close Not relevant

   OmChat: A Recipe to Train Multimodal Language Models with Strong Long Context and Video Understanding

   Tiancheng Zhao, Qianqian Zhang, Kyusong Lee et al. · 2024 · Computer Vision and Pattern Recognition · preprint, not peer reviewed · cited 11 times · AI summary available

   Abstract

   We introduce OmChat, a model designed to excel in handling long contexts and video understanding tasks. OmChat's new architecture standardizes how different visual inputs are processed, making it more efficient and adaptable. It uses a dynamic vision encoding process to effectively handle images of various resolutions, capturing fine details across a range of image qualities. OmChat utilizes an active progressive multimodal pretraining strategy, which gradually increases the model's capacity for long contexts and enhances its overall abilities. By selecting high-quality data during training, OmChat learns from the most relevant and informative data points. With support for a context length o…

6. Close Not relevant

   X-LXMERT: Paint, Caption and Answer Questions with Multi-Modal Transformers

   Jaemin Cho, Jiasen Lu, Dustin Schwenk et al. · 2020 · Conference on Empirical Methods in Natural Language Processing · cited 109 times · Not summarized yet

   Abstract

   Mirroring the success of masked language models, vision-and-language counterparts like ViLBERT, LXMERT and UNITER have achieved state of the art performance on a variety of multimodal discriminative tasks like visual question answering and visual grounding. Recent work has also successfully adapted such models towards the generative task of image captioning. This begs the question: Can these models go the other way and generate images from pieces of text? Our analysis of a popular representative from this model family - LXMERT - finds that it is unable to generate rich and semantically meaningful imagery with its current training setup. We introduce X-LXMERT, an extension to LXMERT with trai…

7. Close Not relevant

   MM1: Methods, Analysis & Insights from Multimodal LLM Pre-training

   Brandon McKinzie, Zhe Gan, Jean-Philippe Fauconnier et al. · 2024 · Computer Vision and Pattern Recognition · preprint, not peer reviewed · cited 277 times · AI summary available

   Abstract

   In this work, we discuss building performant Multimodal Large Language Models (MLLMs). In particular, we study the importance of various architecture components and data choices. Through careful and comprehensive ablations of the image encoder, the vision language connector, and various pre-training data choices, we identified several crucial design lessons. For example, we demonstrate that for large-scale multimodal pre-training using a careful mix of image-caption, interleaved image-text, and text-only data is crucial for achieving state-of-the-art (SOTA) few-shot results across multiple benchmarks, compared to other published pre-training results. Further, we show that the image encoder t…

8. Close Not relevant

   OmniMamba: Efficient and Unified Multimodal Understanding and Generation via State Space Models

   Jialv Zou, Ben-Cheng Liao, Qian Zhang et al. · 2025 · arXiv.org · preprint, not peer reviewed · cited 13 times · Not summarized yet

   Abstract

   Recent advancements in unified multimodal understanding and visual generation (or multimodal generation) models have been hindered by their quadratic computational complexity and dependence on large-scale training data. We present OmniMamba, the first linear-architecture-based multimodal generation model that generates both text and images through a unified next-token prediction paradigm. The model fully leverages Mamba-2's high computational and memory efficiency, extending its capabilities from text generation to multimodal generation. To address the data inefficiency of existing unified models, we propose two key innovations: (1) decoupled vocabularies to guide modality-specific generatio…

Prior work it builds on

1. Close Used in depth by this paper A method it uses Not relevant

   Flamingo: a Visual Language Model for Few-Shot Learning

   Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc et al. · 2022 · Computer Vision and Pattern Recognition · cited 6,929 times · AI summary available

   Abstract

   Building models that can be rapidly adapted to novel tasks using only a handful of annotated examples is an open challenge for multimodal machine learning research. We introduce Flamingo, a family of Visual Language Models (VLM) with this ability. We propose key architectural innovations to: (i) bridge powerful pretrained vision-only and language-only models, (ii) handle sequences of arbitrarily interleaved visual and textual data, and (iii) seamlessly ingest images or videos as inputs. Thanks to their flexibility, Flamingo models can be trained on large-scale multimodal web corpora containing arbitrarily interleaved text and images, which is key to endow them with in-context few-shot learni…

2. Related Used in depth by this paper A method it uses Not relevant

   Llama 2: Open Foundation and Fine-Tuned Chat Models

   Hugo Touvron, Louis Martin, Kevin R. Stone et al. · 2023 · arXiv.org · preprint, not peer reviewed · cited 18,905 times · Not summarized yet

   Abstract

   In this work, we develop and release Llama 2, a collection of pretrained and fine-tuned large language models (LLMs) ranging in scale from 7 billion to 70 billion parameters. Our fine-tuned LLMs, called Llama 2-Chat, are optimized for dialogue use cases. Our models outperform open-source chat models on most benchmarks we tested, and based on our human evaluations for helpfulness and safety, may be a suitable substitute for closed-source models. We provide a detailed description of our approach to fine-tuning and safety improvements of Llama 2-Chat in order to enable the community to build on our work and contribute to the responsible development of LLMs.

3. Related A method it uses Not relevant

   Hierarchical Text-Conditional Image Generation with CLIP Latents

   Aditya Ramesh, Prafulla Dhariwal, Alex Nichol et al. · 2022 · Computer Vision and Pattern Recognition · preprint, not peer reviewed · cited 9,661 times · AI summary available

   Abstract

   Contrastive models like CLIP have been shown to learn robust representations of images that capture both semantics and style. To leverage these representations for image generation, we propose a two-stage model: a prior that generates a CLIP image embedding given a text caption, and a decoder that generates an image conditioned on the image embedding. We show that explicitly generating image representations improves image diversity with minimal loss in photorealism and caption similarity. Our decoders conditioned on image representations can also produce variations of an image that preserve both its semantics and style, while varying the non-essential details absent from the image representa…

4. Close Not relevant

   CM3: A Causal Masked Multimodal Model of the Internet

   Armen Aghajanyan, Po-Yao (Bernie) Huang, Candace Ross et al. · 2022 · arXiv.org · preprint, not peer reviewed · cited 183 times · Not summarized yet

   Abstract

   We introduce CM3, a family of causally masked generative models trained over a large corpus of structured multi-modal documents that can contain both text and image tokens. Our new causally masked approach generates tokens left to right while also masking out a small number of long token spans that are generated at the end of the string, instead of their original positions. The casual masking object provides a type of hybrid of the more common causal and masked language models, by enabling full generative modeling while also providing bidirectional context when generating the masked spans. We train causally masked language-image models on large-scale web and Wikipedia articles, where each do…

5. Close Not relevant

   Unified-IO 2: Scaling Autoregressive Multimodal Models with Vision, Language, Audio, and Action

   Jia-Sen Lu, Christopher Clark, Sangho Lee et al. · 2023 · Computer Vision and Pattern Recognition · cited 404 times · Not summarized yet

   Abstract

   We present Unified-IO 2,the. first autoregressive multi-modal model that is capable of understanding and generating image, text, audio, and action. To unify different modalities, we tokenize inputs and outputs - images, text, audio, action, bounding boxes etc., into a shared semantic space and then process them with a single encoder-decoder transformer model. Since training with such diverse modalities is challenging, we propose various architectural improvements to stabilize model training. We train our model from scratch on a large multimodal pre-training corpus from diverse sources with a multimodal mixture of denoisers objective. To learn an expansive set of skills, such as following mul…

Papers that cite it

1. Close Uses this paper in depth Not relevant

   Argus-Unified: Towards A Compact and Economical Unified Model for Image Understanding and Generation

   Wei-Ming Zhuang, Jia-Bo Huang, Jingtao Li et al. · 2026 · arXiv.org · preprint, not peer reviewed · Not summarized yet

   Abstract

   Unifying visual understanding and generation in one model holds immense promise, but remains challenging and expensive due to heavy compute and data demands and conflicts between the visual features needed for these two capabilities. To address these challenges, we present Argus-Unified, a compact, effective and unified multimodal model built with low demand on computation and data. Instead of aligning modalities from scratch, Argus-Unified effectively leverages pretrained vision-language models (VLMs) that provide strong multimodal priors. Specifically, we introduce hybrid visual tokens that preserve continuous tokens for understanding while learning discrete tokens for generation from a fr…

2. Related Uses this paper in depth Not relevant

   LatentUM: Unleashing the Potential of Interleaved Cross-Modal Reasoning via a Latent-Space Unified Model

   Jiachun Jin, Ze-Tong Zhou, Xiao Yang et al. · 2026 · arXiv.org · preprint, not peer reviewed · cited 10 times · Not summarized yet

   Abstract

   Unified models (UMs) hold promise for their ability to understand and generate content across heterogeneous modalities. Compared to merely generating visual content, the use of UMs for interleaved cross-modal reasoning is more promising and valuable, e.g., for solving understanding problems that require dense visual thinking, improving visual generation through self-reflection, or modeling visual dynamics of the physical world guided by stepwise action interventions. However, existing UMs necessitate pixel decoding as a bridge due to their disjoint visual representations for understanding and generation, which is both ineffective and inefficient. In this paper, we introduce LatentUM, a novel…

3. Close Uses this paper in depth Not relevant

   TextBind: Your Vision-Language Models are Naturally Unified Multimodal Models

   · 2026 · 2026 IEEE/CVF Conference on Computer Vision and Pattern Recognition- FINDINGS Track (CVPRF) · Not summarized yet

   Abstract

   Unified multimodal models seek to both understand and generate across modalities such as images and text. While recent unified models have made impressive achievements, they often come with prohibitive training costs and nontrivial performance regressions. In this work, we introduce TextBind, a simple implementation that endows pre-trained vision-language models (VLMs) with diverse visual generation abilities. By following a customized system prompt, the VLM generates a special tag that triggers our imagegeneration or editing module, which is a diffusion model conditioned on the VLM's full-context hidden states. TextBind seamlessly supports diverse generative and understanding tasks, includi…

4. Close Not relevant

   MMCORE: MultiModal COnnection with Representation Aligned Latent Embeddings

   Zijie Li, Yichun Shi, Jingxiang Sun et al. · 2026 · arXiv.org · preprint, not peer reviewed · cited 1 time · Not summarized yet

   Abstract

   We present MMCORE, a unified framework designed for multimodal image generation and editing. MMCORE leverages a pre-trained Vision-Language Model (VLM) to predict semantic visual embeddings via learnable query tokens, which subsequently serve as conditioning signals for a diffusion model. This streamlined design effectively transfers the rich understanding and reasoning capabilities of VLMs into the visual generation process. By obviating the need for deep fusion between autoregressive and diffusion models or training from scratch, MMCORE significantly reduces computational overhead while maintaining high-fidelity synthesis. MMCORE seamlessly integrates text-to-image synthesis with interleav…

5. Related Uses this paper in depth Not relevant

   Lance: Unified Multimodal Modeling by Multi-Task Synergy

   Fengyi Fu, Mengqi Huang, Shao-Jin Wu et al. · 2026 · arXiv.org · preprint, not peer reviewed · cited 12 times · Not summarized yet

   Abstract

   We present Lance, a lightweight native unified model supporting multimodal understanding, generation, and editing for both images and videos. Rather than relying on model capacity scaling or text-image-dominant designs, Lance explores a practical paradigm for unified multimodal modeling via collaborative multi-task training. It is grounded in two core principles: unified context modeling and decoupled capability pathways. Specifically, Lance is trained from scratch and employs a dual-stream mixture-of-experts architecture on shared interleaved multimodal sequences, enabling joint context learning while decoupling the pathways for understanding and generation. We further introduce modality-aw…