# Benchmarking Agentic Workflow Generation

**Authors:** Shuofei Qiao, Runnan Fang, Zhisong Qiu, Xiaobin Wang, Ningyu Zhang, Yong Jiang, Pengjun Xie, Fei Huang, Huajun Chen  
**arXiv:** [2410.07869](https://arxiv.org/abs/2410.07869)  
**Date:** 2024-10-10  
**Summary page:** https://www.summarizepaper.com/en/paper/2410.07869v3/

## Key points

- Large Language Models (LLMs) are capable of addressing reasoning and planning challenges by breaking down complex problems into actionable workflows.
- Existing workflow evaluation frameworks have limitations such as narrow scenario coverage, simplistic workflow structures, and lenient evaluation criteria.
- The authors introduce WorfBench—a unified benchmark for workflow generation that encompasses multifaceted scenarios and intricate graph-based workflow structures.
- WorfEval is a systematic evaluation protocol that leverages subsequence and subgraph matching algorithms to assess the proficiency of LLM agents in generating workflows accurately.
- Notable disparities between sequence planning capabilities and graph planning abilities of LLM agents were uncovered through extensive evaluations, with even GPT-4 exhibiting a gap of approximately 15% in this regard.
- Generated workflows have the potential to enhance downstream tasks, leading to superior performance with reduced inference time.

## Summary

, , , , 

In their paper titled "Benchmarking Agentic Workflow Generation," authors Shuofei Qiao, Runnan Fang, Zhisong Qiu, Xiaobin Wang, Ningyu Zhang, Yong Jiang, Pengjun Xie, Fei Huang, and Huajun Chen delve into the realm of Large Language Models (LLMs) and their remarkable capacity to tackle a diverse array of tasks. These LLMs have spearheaded significant progress in addressing reasoning and planning challenges by breaking down complex problems into actionable workflows. However, existing workflow evaluation frameworks have been criticized for either focusing solely on overall performance metrics or facing limitations such as narrow scenario coverage, simplistic workflow structures, and lenient evaluation criteria. To address these shortcomings, the authors introduce WorfBench—a unified benchmark for workflow generation that encompasses multifaceted scenarios and intricate graph-based workflow structures. Additionally, they present WorfEval—a systematic evaluation protocol that leverages subsequence and subgraph matching algorithms to precisely assess the LLM agent's proficiency in generating workflows. Through extensive evaluations across various types of LLMs, the researchers uncover notable disparities between the sequence planning capabilities and graph planning abilities of LLM agents. Surprisingly, even GPT-4 exhibits a discernible gap of approximately 15% in this regard. Furthermore, the authors train two open-source models and evaluate their generalization capabilities on unseen tasks. They also observe that the generated workflows have the potential to enhance downstream tasks, enabling them to achieve superior performance with reduced inference time. The code and dataset associated with this research are openly available on GitHub at https://github.com/zjunlp/WorfBench. This study was presented at ICLR 2025 and sheds light on the evolving landscape of agentic workflow generation within the realm of advanced language models.