Running summarizing tools on a new article
Check the paper below, then start its synthesis.
Empowering Large Language Model Agent through Step-Level Self-Critique and Self-Training
Yuanzhao Zhai (National University of Defense Technology), Huanxi Liu (National University of Defense Technology), Zhuo Zhang (Harbin Institute of Technology, Peng Cheng Laboratory), Tong Lin (National University of Defense Technology), Kele Xu (National University of Defense Technology), Cheng Yang (National University of Defense Technology), Dawei Feng (National University of Defense Technology), Bo Ding (National University of Defense Technology), Huaimin Wang (National University of Defense Technology)
Abstract: Large Language Model (LLM) agents frequently produce sub-optimal actions when tackling complex, multi-step decision-making tasks. Employing self-critique to identify flaws and suggest enhancements is an effective strategy for refining actions. Although trajectory-level critique is commonly employed, it often fails to identify flawed steps accurately. In this paper, we introduce SLSC-MCTS, a method that integrates Monte Carlo Tree Search with Step-Level Self-Critique to enhance LLM agents during both testing and self-training phases. During decision tree expansion with SLSC-MCTS, the LLM agent initially generates an action, receives environmental feedback, and subsequently generates further actions via self-critique and refinement. Through multiple episodes of SLSC-MCTS, LLM agents can effectively utilize step-level critiques while disregarding ineffective ones based on node values, thereby incorporating the critiques more robustly. Additionally, our method further empowers LLM agents in a self-training manner, collecting training data from the constructed decision tree to iteratively fine-tune the LLM agents. The self-training data gathered via SLSC-MCTS is diverse and high-quality, which further enhances the reasoning, critiquing, and refining abilities of LLM agents. Experimental results demonstrate that SLSC-MCTS significantly improves LLM agents during testing, surpassing state-of-the-art baselines and achieving shorter task completion trajectories across information retrieval benchmarks such as WebShop and HotPotQA. After three iterations of self-training, LLM agents established by Llama-3.1-8B-Instruct show substantial improvement, even surpassing human experts in WebShop.