# Evaluating Instruction-Tuned Large Language Models on Code Comprehension and Generation

**Authors:** Zhiqiang Yuan, Junwei Liu, Qiancheng Zi, Mingwei Liu, Xin Peng, Yiling Lou  
**arXiv:** [2308.01240](https://arxiv.org/abs/2308.01240)  
**Date:** 2023-08-02  
**Summary page:** https://www.summarizepaper.com/en/paper/2308.01240v1/

## Key points

- Instructed LLMs are competitive and can outperform small SOTA models in code comprehension and generation tasks
- Larger instructed LLMs do not always perform better on code-related tasks
- Incorporating demonstration examples improves performance, but can sometimes lead to instability or worse performance
- BM25-based shot selection strategy performs significantly better than random or fixed selection methods for generation problems
- Fine-tuning enhances model performance compared to zero/one shot performance
- Instructed LLMs outperform small SOTA models and similar scaled LLMs without instruction tuning after fine-tuning on downstream task dataset
- Practical implications for model recommendation and usage, as well as insights into performance and cost trade-offs
- Suggestions for future research directions in this area

## Summary

In this work, the authors evaluate 10 open-source instructed Large Language Models (LLMs) on four representative code comprehension and generation tasks. They make several key findings. Firstly, in the zero-shot setting, instructed LLMs are highly competitive on code comprehension and generation tasks, and in some cases even outperform small State-of-the-Art (SOTA) models that are fine-tuned specifically for each downstream task. Interestingly, the authors also observe that larger instructed LLMs do not always perform better on code-related tasks. Secondly, in the few-shot setting, the authors find that incorporating demonstration examples significantly improves the performance of instructed LLMs on most code comprehension and generation tasks. However, they note that these examples can sometimes lead to unstable or even worse performance. Furthermore, a widely used BM25-based shot selection strategy is found to perform significantly better than basic random selection or fixed selection methods only on generation problems. Thirdly, in the fine-tuning setting, it is observed that fine-tuning further enhances model performance on downstream code comprehension and generation tasks compared to zero/one shot performance. Additionally after being fine tuned on same downstream task dataset instructed LLMs outperform both small SOTA models as well as similar scaled LLMs without instruction tuning. Based on their findings the authors provide practical implications for model recommendation and usage as well as insights into performance and cost trade offs. They also suggest future research directions in this area. Overall this study contributes to understanding how instructed LLMs perform on various code related tasks under different settings and provides valuable insights for improving their effectiveness in real world applications.