# API-Bank: A Benchmark for Tool-Augmented LLMs

**Authors:** Minghao Li, Feifan Song, Bowen Yu, Haiyang Yu, Zhoujun Li, Fei Huang, Yongbin Li  
**arXiv:** [2304.08244](https://arxiv.org/abs/2304.08244)  
**Date:** 2023-04-14  
**Summary page:** https://www.summarizepaper.com/en/paper/2304.08244v1/

## Key points

- Recent research highlights the potential of Large Language Models (LLMs) to enhance contextual processing abilities using external tools
- Lack of systematic evaluation for LLMs' effectiveness in using these tools to respond to human instructions
- Introduction of API-Bank, a benchmark designed for Tool-Augmented LLMs
- API-Bank consists of 53 commonly used API tools and provides a comprehensive workflow
- Includes 264 annotated dialogues with a total of 568 API calls
- Aims to evaluate LLMs' ability to plan step-by-step API calls, retrieve relevant APIs, and accurately execute them to meet human needs
- Experimental results show improved tool usage in GPT-3.5 compared to GPT-3 and stronger planning performance in GPT-4
- Room for improvement compared to human performance
- Error analysis and case studies demonstrate feasibility and highlight challenges for future research
- Benchmark provides insights into capabilities and limitations of Tool-Augmented LLMs
- Foundation for further advancements in utilizing external tools for LLMs' contextual processing abilities towards achieving Artificial General Intelligence

## Summary

Recent research has highlighted the potential of Large Language Models (LLMs) to enhance their contextual processing abilities by utilizing external tools, which could pave the way for Artificial General Intelligence. To address the lack of systematic evaluation to demonstrate the effectiveness of LLMs in using these tools to respond to human instructions, this paper introduces API-Bank, a benchmark specifically designed for Tool-Augmented LLMs. API-Bank consists of 53 commonly used API tools and provides a comprehensive workflow for Tool-Augmented LLMs. It also includes 264 annotated dialogues that encompass a total of 568 API calls. These resources aim to thoroughly evaluate the ability of LLMs to plan step-by-step API calls, retrieve relevant APIs, and accurately execute API calls to meet human needs. Experimental results indicate that GPT-3.5 exhibits improved tool usage compared to GPT-3, while GPT-4 demonstrates stronger planning performance. However, there is still significant room for improvement when compared to human performance. Additionally, detailed error analysis and case studies showcase the feasibility of incorporating Tool-Augmented LLMs into daily use while highlighting the primary challenges that future research should address. Overall, this benchmark provides valuable insights into the capabilities and limitations of Tool-Augmented LLMs and serves as a foundation for further advancements in this field. By systematically evaluating LLMs' ability to utilize external tools in responding to human instructions, researchers can work towards enhancing their contextual processing abilities and ultimately contribute towards achieving Artificial General Intelligence.