# ScribeAgent: Towards Specialized Web Agents Using Production-Scale Workflow Data

**Authors:** Junhong Shen, Atishay Jain, Zedian Xiao, Ishan Amlekar, Mouad Hadji, Aaron Podolny, Ameet Talwalkar  
**arXiv:** [2411.15004](https://arxiv.org/abs/2411.15004)  
**Date:** 2024-11-22  
**Summary page:** https://www.summarizepaper.com/en/paper/2411.15004v1/

## Key points

- Introduction of innovative approach to enhancing Large Language Model (LLM) agents by fine-tuning open-source models with real-world workflow data
- Reformatting proprietary workflow data into next-step prediction format and training LLMs using LoRA framework
- Development of specialized web agents, ScribeAgent, excelling in web understanding and planning
- Superior performance demonstrated on public benchmarks Mind2Web and WebArena compared to existing models
- State-of-the-art direct generation results achieved on Mind2Web with 32B-parameter ScribeAgent-Large
- Significant improvement in task success rates on WebArena with 7B ScribeAgent-Small
- Insights provided for future web agent research, including direct fine-tuning on structured inputs like HTML-DOM and effective HTML preprocessing strategies
- Analysis of design choices in fine-tuning process, such as LLM backbone selection and context window optimization
- Potential benefits of developing specialized web agents through fine-tuning with large-scale real-world data for improved performance and reduced serving costs

## Summary

, , , , 

In this study, we introduce an innovative approach to enhancing Large Language Model (LLM) agents by fine-tuning open-source models with a vast dataset of real-world workflow data. Our methodology involves reformatting the proprietary workflow data into a next-step prediction format and training the LLMs using the parameter-efficient LoRA framework. With over 6 billion tokens in our training dataset, we develop specialized web agents, called ScribeAgent, that excel in web understanding and planning. Through extensive evaluations on public benchmarks like Mind2Web and WebArena, ScribeAgent demonstrates superior performance compared to existing GPT-4-based models and multi-stage agents. The 32B-parameter ScribeAgent-Large achieves state-of-the-art direct generation results on Mind2Web, surpassing baseline performance by 5-10% across all test sets. On WebArena, our 7B ScribeAgent-Small significantly improves task success rates from 37.2% to 51.3%, establishing itself as the top-performing text-only LLM agent. Our study also offers valuable insights for future web agent research. We highlight the feasibility and benefits of direct fine-tuning on structured inputs like HTML-DOM to enhance target identification accuracy. Additionally, we propose effective strategies for HTML preprocessing to balance information retention with context length minimization. Our analysis delves into various design choices in fine-tuning, such as LLM backbone selection and context window optimization. Overall, our work underscores the potential of developing specialized web agents through fine-tuning with large-scale real-world data. This approach not only enhances agent capabilities relative to prompt-engineered alternatives but also enables the creation of more efficient models with reduced serving costs. By showcasing the effectiveness of specialized fine-tuning in improving agent performance across diverse benchmarks, we pave the way for advancements in web agent development and deployment strategies.