# Evaluating the Robustness to Instructions of Large Language Models

**Authors:** Yuansheng Ni, Sichao Jiang, Xinyu wu, Hui Shen, Yuli Zhou  
**arXiv:** [2308.14306](https://arxiv.org/abs/2308.14306)  
**Date:** 2023-08-28  
**Summary page:** https://www.summarizepaper.com/en/paper/2308.14306v2/

## Key points

- Growing interest in Instruction fine-tuning to improve zero-shot capabilities of Large Language Models (LLMs)
- Impressive results in enhancing performance of moderately sized LLMs
- Evaluation of robustness of instruction-tuned LLMs on familiar and unfamiliar tasks
- Six models explored: Alpaca, Vicuna, WizardLM, Flan-T5-XL/XXL, T0++
- Real-world relation extraction datasets used for analysis
- Performance and robustness of models when following instructions discussed
- Model's ability to handle unfamiliar instructions deteriorated, leading to poorer performance
- Robustness for relation extraction (RE) instructions worse than question-answering (QA) instructions
- FLAN-T5 model's performance improved with larger parameter sizes but still inferior for RE instruction compared to QA instruction
- Instruction fine-tuning has potential for improving zero-shot capabilities in LLMs but challenges remain in handling unfamiliar instructions and maintaining robustness across tasks.

## Summary

Recently, there has been a growing interest in the use of Instruction fine-tuning as a means to improve the zero-shot capabilities of Large Language Models (LLMs) on new tasks. This technique has demonstrated impressive results in enhancing the performance of moderately sized LLMs, often achieving comparable performance levels to larger model variants. The main focus of this study is to evaluate the robustness of instruction-tuned LLMs when faced with both familiar and unfamiliar tasks. To conduct this evaluation, six models were explored: Alpaca, Vicuna, WizardLM and Traditional Task-oriented Models (Flan-T5-XL/XXL and T0++). Real-world relation extraction datasets were used as case studies for this analysis. The evaluation aimed to comprehensively assess these instruction-following LLMs that have been fine-tuned using open-domain instructions and task-oriented instructions. The primary discussion revolves around the performance and robustness of these models when following instructions. It was observed that in most cases, the model's ability to handle unfamiliar instructions significantly deteriorated, leading to poorer performance. Additionally, it was found that the robustness of the models for relation extraction (RE) instructions was worse compared to question-answering (QA) instructions. Furthermore, an interesting discovery was made regarding the FLAN-T5 model; its performance improved as the parameter count increased up until a certain threshold size (3B). However, despite this improvement in performance with larger parameter sizes, its robustness for RE instruction remained inferior compared to its robustness towards QA instruction. In conclusion, this study highlights the potential of Instruction fine-tuning for improving zero-shot capabilities in Large Language Models. However, it also emphasizes the challenges associated with handling unfamiliar instructions and maintaining robustness across different types of tasks. These findings contribute valuable insights into optimizing instruction-tuned LLMs for various applications.