# SELA: Tree-Search Enhanced LLM Agents for Automated Machine Learning

**Authors:** Yizhou Chi, Yizhang Lin, Sirui Hong, Duyi Pan, Yaying Fei, Guanghao Mei, Bangbang Liu, Tianqi Pang, Jacky Kwok, Ceyao Zhang, Bang Liu, Chenglin Wu  
**arXiv:** [2410.17238](https://arxiv.org/abs/2410.17238)  
**Date:** 2024-10-22  
**Summary page:** https://www.summarizepaper.com/en/paper/2410.17238v1/

## Key points

- Traditional methods focus on optimizing fixed pipelines
- Newer frameworks autonomously build pipelines
- Limitations of LLM-based agents in automating machine learning tasks:
- Struggle with generating diverse and optimal code even after multiple iterations
- Leverages tree search algorithms to enhance AutoML significantly
- Represents pipeline configurations as trees for intelligent experimentation and iterative refinement
- SELA achieved an impressive win rate ranging from 65% to 80% against each baseline across all datasets
- Provides a fresh perspective on tackling complex machine learning challenges
- Integration of tree search algorithms with LLMs represents a significant advancement in problem-solving within artificial intelligence

## Summary

In the realm of , various approaches have been developed to streamline the process of model selection and ensembling. Traditional methods focus on optimizing fixed pipelines, while newer frameworks based on  autonomously build pipelines. Despite the promise shown by LLM-based agents in automating machine learning tasks, they often struggle with generating diverse and optimal code even after multiple iterations. To address these limitations, a groundbreaking system called  has been introduced. SELA leverages  to enhance the AutoML process significantly. By representing pipeline configurations as trees, SELA enables agents to conduct experiments intelligently and refine their strategies iteratively. This innovative approach facilitates a more effective exploration of the machine learning solution space, allowing SELA to discover optimal pathways based on experimental feedback and ultimately improving the quality of solutions generated. In a comprehensive evaluation across 20 diverse machine learning datasets, traditional AutoML methods were compared with agent-based approaches like SELA. The results demonstrated that SELA achieved an impressive win rate ranging from 65% to 80% against each baseline across all datasets. These findings underscore the substantial potential of agent-based strategies in AutoML, providing a fresh perspective on tackling complex machine learning challenges. Moreover, SELA's integration of tree search algorithms with LLMs represents a significant advancement in problem-solving within artificial intelligence. This integration allows for more efficient exploration and refinement of machine learning pipelines through intelligent experimentation and feedback mechanisms. Overall, the introduction of SELA marks a significant step forward in the field of AutoML, showcasing how innovative agent-based systems can revolutionize the way we approach machine learning tasks and pave the way for more efficient and effective solutions in complex problem domains.