# Tuning Large Multimodal Models for Videos using Reinforcement Learning from AI Feedback

**Authors:** Daechul Ahn, Yura Choi, Youngjae Yu, Dongyeop Kang, Jonghyun Choi  
**arXiv:** [2402.03746](https://arxiv.org/abs/2402.03746)  
**Date:** 2024-02-06  
**Summary page:** https://www.summarizepaper.com/en/paper/2402.03746v2/

## Key points

- Recent advancements in large language models have enabled the development of Video-Language Multimodal Models (VLMMs).
- Previous approaches for VLMMs involved instruction-tuned datasets, integrating LLM with visual encoders, and incorporating additional learnable modules.
- Aligning video and text modalities has been challenging due to limited volume and quality of multimodal instruction-tune data compared to text-only data.
- Introducing a novel alignment strategy called Context-Aware Reward Modeling (RM) where a multimodal AI system provides self-preference feedback to refine alignment of video and text modalities.
- Utilizing detailed video descriptions as context during the generation of preference feedback enhances understanding of video content.
- Training the VLM-SFT model using pre-trained image-text models with various video-text instruction-tune datasets, including object-centric narrative datasets.
- Employing reinforcement learning through RM outperforms existing approaches like SFT model across diverse video benchmarks.
- Training details involve sampling 50 frames from each video, extracting spatial and temporal features using CLIP visual encoder, and implementing two-stage curriculum learning based on sample difficulty.
- Generating responses from existing instruction-tune data to train the RM model and further refining it using reinforcement learning on existing datasets.
- The multimodal RLAIF approach showcases enhanced performance in aligning video and text modalities, offering a promising solution for future research.

## Summary

Recent advancements in large language models have paved the way for the development of . Previous approaches for VLMMs involved  with instruction-tuned datasets, integrating LLM with visual encoders, and incorporating additional learnable modules. However, aligning video and text modalities has remained a challenge due to the limited volume and quality of multimodal instruction-tune data compared to text-only data. To address this issue, we introduce a novel alignment strategy called , where a multimodal AI system oversees itself by providing self-preference feedback to refine its alignment of video and text modalities. Specifically, we propose context-aware reward modeling by utilizing detailed video descriptions as context during the generation of preference feedback to enhance the understanding of video content. In our experiments, we train the VLM-SFT model using pre-trained image-text models with various video-text instruction-tune datasets. We also utilize open-source datasets such as object-centric narrative video-text instruction-tune dataset for training. By employing reinforcement learning through , we outperform existing approaches like SFT model across diverse video benchmarks. Furthermore, our training details involve sampling 50 frames from each video and extracting spatial and temporal features using CLIP visual encoder. We implement two-stage curriculum learning by dividing SFT data into easy and hard groups based on sample difficulty. Additionally, we generate responses from existing instruction-tune data to train the RM model and use reinforcement learning with trained RM on existing datasets for further refinement. In conclusion, our multimodal RLAIF approach showcases enhanced performance in aligning video and text modalities, offering a promising solution for future research in this area. We are committed to open-sourcing our code, models, and datasets to facilitate continued exploration in this field.