# VLIS: Unimodal Language Models Guide Multimodal Language Generation

**Authors:** Jiwan Chung, Youngjae Yu  
**arXiv:** [2310.09767](https://arxiv.org/abs/2310.09767)  
**Date:** 2023-10-15  
**Summary page:** https://www.summarizepaper.com/en/paper/2310.09767v1/

## Key points

- Multimodal language generation is a rapidly growing field
- Authors propose a framework called VLIS to overcome challenges in vision-language models
- VLIS uses pointwise mutual information as importance sampling weights to adjust token likelihood
- VLIS improves performance on tasks like commonsense understanding and complex text generation
- VLIS outperforms Socratic Model implementation based on GPT-3 175B in contextualized captioning using the Concadia dataset
- Captions generated by VLIS are better aligned with caption-style and reflect article context accurately compared to baselines
- VLIS shows promising results in paragraph captioning with three in-context examples (3-shot)
- VLIS represents a promising direction for multimodal language generation and enhances performance on tasks requiring complex linguistic understanding.

## Summary

Multimodal language generation, which combines language and vision, is a rapidly growing field. To overcome the challenges existing vision-language models face in tasks requiring complex linguistic understanding, the authors propose a novel framework called Visual-Language models as Importance Sampling weights (VLIS). This framework extracts pointwise mutual information from a visual-language model and uses it as an importance sampling weight to adjust token likelihood from a text-only model. The results show that VLIS improves performance on various tasks such as commonsense understanding and complex text generation. In the task of contextualized captioning using the Concadia dataset, VLIS outperforms the Socratic Model implementation based on GPT-3 175B. The captions generated by VLIS are better aligned with caption-style and reflect the Wikipedia article context more accurately compared to baselines. In paragraph captioning with three in-context examples (3-shot), VLIS shows promising results even in this challenging setting. These findings suggest that VLIS represents a promising direction for multimodal language generation and can enhance performance on diverse tasks requiring complex linguistic understanding.