# LLaVA-o1: Let Vision Language Models Reason Step-by-Step

**Authors:** Guowei Xu, Peng Jin, Li Hao, Yibing Song, Lichao Sun, Li Yuan  
**arXiv:** [2411.10440](https://arxiv.org/abs/2411.10440)  
**Date:** 2024-11-15  
**Summary page:** https://www.summarizepaper.com/en/paper/2411.10440v1/

## Key points

- Authors introduce a novel Vision-Language Model (VLM) called LLaVA-o1 to address limitations in systematic and structured reasoning in current VLMs
- LLaVA-o1 autonomously conducts multistage reasoning through sequential stages of summarization, visual interpretation, logical reasoning, and conclusion generation
- Researchers compile the LLaVA-o1-100k dataset for development and evaluation, integrating samples from various sources with structured reasoning annotations
- An inference-time stage-level beam search method is proposed to enable effective scaling during inference
- LLaVA-o1 outperforms its base model by 8.9% on multimodal reasoning benchmarks with only 100k training samples and a simple yet effective inference time scaling method

## Summary

, , , , 

In their paper titled "LLaVA-o1: Let Vision Language Models Reason Step-by-Step," authors Guowei Xu, Peng Jin, Li Hao, Yibing Song, Lichao Sun, and Li Yuan introduce a novel Vision-Language Model (VLM) called LLaVA-o1. The researchers highlight the limitations of current VLMs in performing systematic and structured reasoning, particularly in complex visual question-answering tasks. To address this challenge, they propose LLaVA-o1 as a solution. LLaVA-o1 is designed to autonomously conduct multistage reasoning by independently engaging in sequential stages of summarization, visual interpretation, logical reasoning, and conclusion generation. This approach differs from traditional chain-of-thought prompting methods and enables significant improvements in precision on reasoning-intensive tasks. To support the development and evaluation of LLaVA-o1, the researchers compile the LLaVA-o1-100k dataset by integrating samples from various visual question answering sources and providing structured reasoning annotations. They also propose an inference-time stage-level beam search method to enable effective scaling during inference. Remarkably, with only 100k training samples and a simple yet effective inference time scaling method, LLaVA-o1 outperforms its base model by 8.9% on a wide range of multimodal reasoning benchmarks. This performance surpasses that of larger and even closed-source models such as Gemini-1.5-pro, GPT-4o-mini, and Llama-3.2-90B-Vision-Instruct. Overall,<Organization> LLAVA-o1 represents a significant advancement in the field of Vision-Language Models by demonstrating improved capabilities in autonomous multistage reasoning for complex visual question-answering tasks. The methodology presented in this work opens up new possibilities for enhancing reasoning abilities in AI systems through structured approaches and effective inference-time scaling techniques.