Results of the summarizing process for the arXiv paper: 2406.06004v1

In their paper titled "FLEUR: An Explainable Reference-Free Evaluation Metric for Image Captioning Using a Large Multimodal Model," authors Yebin Lee, Imseong Park, and Myungjoo Kang address the limitations of existing image captioning evaluation metrics that focus on assigning a single numerical score to a caption by comparing it with reference captions. These methods lack explainability in the assigned score and require expensive reference captions. To overcome these challenges, the authors propose FLEUR, a novel reference-free metric that introduces explainability into image captioning evaluation metrics. By leveraging a large multimodal model, FLEUR can evaluate captions against images without the need for reference captions and provide explanations for the assigned scores. The authors also introduce score smoothing techniques to align closely with human judgment and ensure robustness to user-defined grading criteria. Through extensive experimentation, FLEUR demonstrates high correlations with human judgment across various image captioning evaluation benchmarks. Furthermore, FLEUR achieves state-of-the-art results on popular datasets such as Flickr8k-CF, COMPOSITE, and Pascal-50S within the domain of reference-free evaluation metrics. The authors have made their source code and results publicly available at https://github.com/Yebin46/FLEUR. This research was accepted at ACL (Main) 2024 and contributes significantly to advancing explainable and accurate evaluation metrics in the field of image captioning.

• - Authors address limitations of existing image captioning evaluation metrics
• - Focus on assigning a single numerical score by comparing with reference captions
• - Lack explainability in assigned score and require expensive reference captions
• - Proposal of FLEUR as a novel reference-free metric for image captioning evaluation
• - Introduces explainability into evaluation metrics
• - Leverages large multimodal model to evaluate captions against images without reference captions
• - Provides explanations for assigned scores
• - Introduction of score smoothing techniques to align with human judgment and ensure robustness to user-defined grading criteria
• - FLEUR demonstrates high correlations with human judgment across various image captioning benchmarks
• - Achieves state-of-the-art results on datasets such as Flickr8k-CF, COMPOSITE, and Pascal-50S within the domain of reference-free evaluation metrics
• - Source code and results publicly available at https://github.com/Yebin46/FLEUR

Summary- Authors talk about problems with current ways of checking how good captions are on pictures. - They suggest a new method called FLEUR that doesn't need other captions to compare against. - FLEUR explains why it gives scores and uses a big model to check captions without references. - It also smoothens scores to match what people think and works well on different tests. - The results and code for FLEUR can be found at a website. Definitions- Image captioning: Adding words to describe what's happening in a picture. - Metrics: Measurements used to see how well something is doing. - Explainability: Making clear why something happens or is done in a certain way. - Reference-free: Not needing other examples as comparisons.

FLEUR: An Explainable Reference-Free Evaluation Metric for Image Captioning Using a Large Multimodal Model

Image captioning is a challenging task in the field of computer vision and natural language processing, where the goal is to generate a descriptive sentence that accurately describes an image. With the increasing popularity of social media platforms and e-commerce websites, there has been a growing demand for automatic image captioning systems. These systems have various applications such as aiding visually impaired individuals, improving search engine results, and enhancing user experience on social media platforms. To evaluate the performance of image captioning models, researchers have developed several evaluation metrics over the years. However, these metrics often rely on comparing generated captions with reference captions provided by humans. This approach has limitations as it requires expensive human annotations and lacks explainability in the assigned scores. To address these challenges, Yebin Lee, Imseong Park, and Myungjoo Kang propose FLEUR (Fully Learnable Explainable Universal Reference-free metric), a novel reference-free evaluation metric for image captioning. The authors' primary motivation behind developing FLEUR was to introduce explainability into existing evaluation metrics while eliminating the need for reference captions. The proposed metric leverages a large multimodal model trained on multiple datasets to evaluate captions against images without any references. This not only reduces annotation costs but also enables more efficient evaluation of large datasets. One of the key features of FLEUR is its ability to provide explanations for assigned scores through attention maps generated by the multimodal model. These attention maps highlight which parts of an image were most influential in generating specific words in a caption. This allows users to understand why certain scores were assigned and provides insights into how their model can be improved. Furthermore, FLEUR incorporates score smoothing techniques that align closely with human judgment and ensure robustness to user-defined grading criteria. This addresses another limitation of existing metrics where slight variations in human annotations can significantly affect the assigned scores. To evaluate the effectiveness of FLEUR, the authors conducted extensive experiments on popular image captioning datasets such as Flickr8k-CF, COMPOSITE, and Pascal-50S. The results showed that FLEUR achieved high correlations with human judgment and outperformed existing reference-free metrics. Additionally, FLEUR also achieved state-of-the-art results on these datasets within the domain of reference-free evaluation metrics. The authors have made their source code and results publicly available at https://github.com/Yebin46/FLEUR. This not only promotes reproducibility but also allows other researchers to build upon this work and further improve image captioning evaluation metrics. This research was accepted at ACL (Main) 2024, one of the top conferences in natural language processing, which highlights its significance in advancing explainable and accurate evaluation metrics for image captioning. In conclusion, FLEUR is a novel reference-free evaluation metric that addresses limitations of existing methods by introducing explainability into image captioning evaluations while eliminating the need for expensive reference captions. Through extensive experimentation, it has shown promising results and has been accepted at a prestigious conference. With its potential to enhance automatic image captioning systems' performance and reduce annotation costs, FLEUR is a significant contribution to the field of computer vision and natural language processing.