# FLASK: Fine-grained Language Model Evaluation based on Alignment Skill Sets

**Authors:** Seonghyeon Ye, Doyoung Kim, Sungdong Kim, Hyeonbin Hwang, Seungone Kim, Yongrae Jo, James Thorne, Juho Kim, Minjoon Seo  
**arXiv:** [2307.10928](https://arxiv.org/abs/2307.10928)  
**Date:** 2023-07-20  
**Summary page:** https://www.summarizepaper.com/en/paper/2307.10928v4/

## Key points

- Complexity in evaluating Large Language Models (LLMs) lies in aligning models with human values and required skills for instructions
- Previous studies focused on coarse-grained evaluation methods, lacking interpretability for nuanced user instructions
- Introduction of fine-grained evaluation protocol FLASK (Fine-grained Language Model Evaluation based on Alignment Skill Sets)
- FLASK breaks down coarse-level scoring into skill set-level scoring for detailed analysis of model performance
- Granularity of evaluation crucial for comprehensive understanding and reliability of evaluations
- High correlation between model-based and human-based evaluations using FLASK
- FLASK evaluation data and code implementation publicly available on GitHub
- Acknowledgments to KAIST-NAVER Hypercreative AI Center, IITP grant, contributors, and evaluators at KAIST
- Paper published at ICLR 2024 showcasing significance in advancing LLM analysis.

## Summary

In the realm of evaluating Large Language Models (LLMs), the complexity lies in aligning these models with human values and skills required for various instructions. Previous studies have primarily focused on coarse-grained evaluation methods, which provide an overall preference-based assessment. However, this approach lacks interpretability as it does not account for the nuanced nature of user instructions that demand specific skill compositions at an instance level. To address this limitation, a new fine-grained evaluation protocol called FLASK (Fine-grained Language Model Evaluation based on Alignment Skill Sets) has been introduced. This protocol breaks down the coarse-level scoring into skill set-level scoring for each instruction, allowing for a more detailed analysis. Through experimental observations, it has been noted that the granularity of evaluation plays a crucial role in obtaining a comprehensive understanding of model performance and enhancing the reliability of evaluations. By utilizing FLASK, comparisons between multiple open-source and proprietary LLMs have shown a high correlation between model-based and human-based evaluations. The evaluation data and code implementation for FLASK have been made publicly available on GitHub. Acknowledgments mention support from KAIST-NAVER Hypercreative AI Center and Institute of Information & Communications Technology Planning & Evaluation (IITP) grant funded by the Korean government. The paper also expresses gratitude to individuals who contributed to helpful discussions and feedback, as well as members of KAIST who participated in human evaluations for FLASK. This work has been published as a conference paper at ICLR 2024, highlighting its significance in advancing the field through an extensive analysis of LLMs.