AI summary · arXiv: 2404.01744v1 · cs.CL
Octopus v2: On-device language model for super agent
The essentials
- Language models have proven effective in software applications, particularly in tasks related to automatic workflow
- Large-scale language models show high performance in cloud environments but raise concerns over privacy and cost
- On-device models for function calling face challenges such as latency and accuracy issues
- Research introduces a new method empowering on-device model with 2 billion parameters to outperform GPT-4 in accuracy and latency while reducing context length by 95%
- Efforts are underway to deploy smaller-scale Large Language Models (LLMs) on edge devices like PCs and smartphones due to memory limitations and lower inference speeds
- Open-source models of manageable sizes like Gemma-2B, Gemma-7B, StableCode-3B, and Llama-7B introduced for deployment on edge devices
- Advancements in function-calling capabilities of smaller-scale models observed through projects like NexusRaven, Toolformer, ToolAlpaca, Gorrilla, ToolLlama,and Taskmatrix
- Field of AI agents evolving rapidly with advancements in AI assistant tools like MultiOn and Adept AI along with AI consumer products like Rabbit R1 and Humane AI Pin gaining traction
Key points written by AI: they may contain errors, check them against the paper.
Ask questions about this paper to our AI assistant
How to chat with this paper? Illustrated guide
You can also chat with multiple papers at once here.
Keep this conversation — a free account saves your questions and answers, your collections of papers and your alerts.
Continue with Google Create a free accountTools for this paper
Advanced tools Premium
For researchers who follow a field closely: citation alerts, journal-club slides, version comparison and Zotero export. Discover Premium
Papers around this one
See them on a mapRanked by how close their title and abstract are to this paper’s. Select some to read them together.
Closest papers
-
Octopus: On-device language model for function calling of software APIsWei Chen, Zhiyuan Li, Mingyuan Ma · 2024 · Computation and Language · cited 31 times · AI summary available
Abstract
In the rapidly evolving domain of artificial intelligence, Large Language Models (LLMs) play a crucial role due to their advanced text processing and generation abilities. This study introduces a new strategy aimed at harnessing on-device LLMs in invoking software APIs. We meticulously compile a dataset derived from software API documentation and apply fine-tuning to LLMs with capacities of 2B, 3B and 7B parameters, specifically to enhance their proficiency in software API interactions. Our approach concentrates on refining the models' grasp of API structures and syntax, significantly enhancing the accuracy of API function calls. Additionally, we propose \textit{conditional masking} techniqu…
-
Octopus v3: Technical Report for On-device Sub-billion Multimodal AI AgentWei Chen, Zhiyuan Li · 2024 · Computation and Language · preprint, not peer reviewed · cited 10 times · AI summary available
Abstract
A multimodal AI agent is characterized by its ability to process and learn from various types of data, including natural language, visual, and audio inputs, to inform its actions. Despite advancements in large language models that incorporate visual data, such as GPT-4V, effectively translating image-based data into actionable outcomes for AI agents continues to be challenging. In this paper, we introduce a multimodal model that incorporates the concept of functional token specifically designed for AI agent applications. To ensure compatibility with edge devices, our model is optimized to a compact size of less than 1B parameters. Like GPT-4, our model can process both English and Chinese. W…
-
Octopus v4: Graph of language modelsWei Chen, Zhi-Yuan Li · 2024 · arXiv.org · preprint, not peer reviewed · cited 15 times · Not summarized yet
Abstract
Language models have been effective in a wide range of applications, yet the most sophisticated models are often proprietary. For example, GPT-4 by OpenAI and various models by Anthropic are expensive and consume substantial energy. In contrast, the open-source community has produced competitive models, like Llama3. Furthermore, niche-specific smaller language models, such as those tailored for legal, medical or financial tasks, have outperformed their proprietary counterparts. This paper introduces a novel approach that employs functional tokens to integrate multiple open-source models, each optimized for particular tasks. Our newly developed Octopus v4 model leverages functional tokens to …
-
Octo-planner: On-device Language Model for Planner-Action AgentsWei Chen, Zhi-Yuan Li, Zhen Guo et al. · 2024 · International Workshop on Engineering Multi-Agent Systems · cited 12 times · Not summarized yet
Abstract
AI agents have become increasingly significant in various domains, enabling autonomous decision-making and problem-solving. To function effectively, these agents require a planning process that determines the best course of action and then executes the planned actions. In this paper, we present an efficient on-device Planner-Action framework that separates planning and action execution into two distinct components: a planner agent based on Phi-3 Mini, a 3.8 billion parameter LLM optimized for edge devices, and an action agent using the Octopus model for function execution. The planner agent first responds to user queries by decomposing tasks into a sequence of sub-steps, which are then execu…
-
TinyAgent: Function Calling at the EdgeLutfi Eren Erdogan, Nicholas Lee, Siddharth Jha et al. · 2024 · Conference on Empirical Methods in Natural Language Processing · cited 85 times · Not summarized yet
Abstract
Recent large language models (LLMs) have enabled the development of advanced agentic systems that can integrate various tools and APIs to fulfill user queries through function calling. However, the deployment of these LLMs on the edge has not been explored since they typically require cloud-based infrastructure due to their substantial model size and computational demands. To this end, we present TinyAgent, an end-to-end framework for training and deploying task-specific small language model agents capable of function calling for driving agentic systems at the edge. We first show how to enable accurate function calling for open-source models via the LLMCompiler framework. We then systematica…
-
OpenPhone: Mobile Agentic Foundation ModelsYa Jiang, Chao Huang · 2025 · Annual Meeting of the Association for Computational Linguistics · cited 5 times · Not summarized yet
Abstract
With the advancement of multimodal large language models (MLLMs), building GUI agent systems has become an increasingly promising direction--especially for mobile platforms, given their rich app ecosystems and intuitive touch interactions. Yet mobile GUI agents face a critical dilemma: truly on-device models (4B or smaller) lack sufficient performance, while capable models (starting from 7B) are either too large for mobile deployment or prohibitively costly (e.g., cloud-only closed-source MLLMs). To resolve this, we propose OpenPhone, a mobile GUI agent system that leverages device-cloud collaboration to tap the cost-efficiency of on device models and the high capability of cloud models, whi…
-
POSTER: TinyLLM: Evaluation and Optimization of Small Language Models for Agentic Tasks on Edge DevicesMohd Ariful Haque, Fahad Rahman, K. Gupta et al. · 2026 · ACM International Conference on Computing Frontiers · cited 2 times · Not summarized yet
Abstract
As autonomous AI agents increasingly rely on Large Language Models (LLMs) for decision making; the necessity for local, privacy preserving execution on edge devices has become paramount. This paper investigates the effectiveness of Small Language Models (SLMs) with equal or fewer than 4 billion parameters for agentic tasks, specifically function and API calling. Using the Berkeley Function Calling Leaderboard (BFCL), we provide a comprehensive evaluation of models including xLAM-2, Qwen3, and TinyLlama. We introduce a Direct Preference Optimization (DPO) pipeline that leverages AgentBank (ALFRED) trajectories to align SLMs toward expert level tool use behavior. Our experimental results demon…
-
On-Device Language Models: A Comprehensive ReviewJia-Jun Xu, Zhi-Yuan Li, Wei Chen et al. · 2024 · arXiv.org · preprint, not peer reviewed · cited 206 times · Not summarized yet
Abstract
The advent of large language models (LLMs) revolutionized natural language processing applications, and running LLMs on edge devices has become increasingly attractive for reasons including reduced latency, data localization, and personalized user experiences. This comprehensive review examines the challenges of deploying computationally expensive LLMs on resource-constrained devices and explores innovative solutions across multiple domains. The paper investigates the development of on-device language models, their efficient architectures, including parameter sharing and modular designs, as well as state-of-the-art compression techniques like quantization, pruning, and knowledge distillation…
Prior work it builds on
-
MobileLLM: Optimizing Sub-billion Parameter Language Models for On-Device Use CasesZechun Liu, Changsheng Zhao, Forrest Iandola et al. · 2024 · Machine Learning · cited 356 times · Not summarized yet
Abstract
This paper addresses the growing need for efficient large language models (LLMs) on mobile devices, driven by increasing cloud costs and latency concerns. We focus on designing top-quality LLMs with fewer than a billion parameters, a practical choice for mobile deployment. Contrary to prevailing belief emphasizing the pivotal role of data and parameter quantity in determining model quality, our investigation underscores the significance of model architecture for sub-billion scale LLMs. Leveraging deep and thin architectures, coupled with embedding sharing and grouped-query attention mechanisms, we establish a strong baseline network denoted as MobileLLM, which attains a remarkable 2.7%/4.3% …
-
LLMCad: Fast and Scalable On-device Large Language Model InferenceDa-Liang Xu, Wang-Song Yin, Xin Jin et al. · 2023 · arXiv.org · preprint, not peer reviewed · cited 78 times · Not summarized yet
Abstract
Generative tasks, such as text generation and question answering, hold a crucial position in the realm of mobile applications. Due to their sensitivity to privacy concerns, there is a growing demand for their execution directly on mobile devices. Currently, the execution of these generative tasks heavily depends on Large Language Models (LLMs). Nevertheless, the limited memory capacity of these devices presents a formidable challenge to the scalability of such models. In our research, we introduce LLMCad, an innovative on-device inference engine specifically designed for efficient generative Natural Language Processing (NLP) tasks. The core idea behind LLMCad revolves around model collaborat…
-
Multi-Agent Collaboration: Harnessing the Power of Intelligent LLM AgentsYashar Talebirad, Amirhossein Nadiri · 2023 · Artificial Intelligence · preprint, not peer reviewed · cited 510 times · AI summary available
Abstract
In this paper, we present a novel framework for enhancing the capabilities of large language models (LLMs) by leveraging the power of multi-agent systems. Our framework introduces a collaborative environment where multiple intelligent agent components, each with distinctive attributes and roles, work together to handle complex tasks more efficiently and effectively. We demonstrate the practicality and versatility of our framework through case studies in artificial general intelligence (AGI), specifically focusing on the Auto-GPT and BabyAGI models. We also examine the "Gorilla" model, which integrates external APIs into the LLM. Our framework addresses limitations and challenges such as loop…
-
AppAgent: Multimodal Agents as Smartphone UsersC. Zhang, Zhao Yang, Jia-Xuan Liu et al. · 2023 · International Conference on Human Factors in Computing Systems · cited 588 times · Not summarized yet
Abstract
Recent advancements in large language models (LLMs) have led to the creation of intelligent agents capable of performing complex tasks. This paper introduces a novel LLM-based multimodal agent framework designed to operate smartphone applications. Our framework allows the agent to mimic human-like interactions such as tapping and swiping through a simplified action space, eliminating the need for system back-end access and enhancing its versatility across various apps. Central to the agent’s functionality is an innovative in-context learning method, where it either autonomously explores or learns from human demonstrations, creating a knowledge base used to execute complex tasks across divers…
-
Gorilla: Large Language Model Connected with Massive APIsShishir G. Patil, Tianjun Zhang, Xin Wang et al. · 2023 · Neural Information Processing Systems · cited 1,838 times · AI summary available
Abstract
Large Language Models (LLMs) have seen an impressive wave of advances recently, with models now excelling in a variety of tasks, such as mathematical reasoning and program synthesis. However, their potential to effectively use tools via API calls remains unfulfilled. This is a challenging task even for today's state-of-the-art LLMs such as GPT-4, largely due to their inability to generate accurate input arguments and their tendency to hallucinate the wrong usage of an API call. We release Gorilla, a finetuned LLaMA-based model that surpasses the performance of GPT-4 on writing API calls. When combined with a document retriever, Gorilla demonstrates a strong capability to adapt to test-time d…
Papers that cite it
-
ELMS: Elasticized Large Language Models On Mobile DevicesWang-Song Yin, Rongjie Yi, Da-Liang Xu et al. · 2024 · arXiv.org · preprint, not peer reviewed · cited 9 times · Not summarized yet
-
Optimizing Small Language Models for In-Vehicle Function-CallingYahya S. Khiabani, Farris Atif, Chieh-Hsin Hsu et al. · 2025 · arXiv.org · preprint, not peer reviewed · cited 7 times · Not summarized yet
Abstract
We propose a holistic approach for deploying Small Language Models (SLMs) as function-calling agents within vehicles as edge devices, offering a more flexible and robust alternative to traditional rule-based systems. By leveraging SLMs, we simplify vehicle control mechanisms and enhance the user experience. Given the in-vehicle hardware constraints, we apply state-of-the-art model compression techniques, including structured pruning, healing, and quantization, ensuring that the model fits within the resource limitations while maintaining acceptable performance. Our work focuses on optimizing a representative SLM, Microsoft's Phi-3 mini, and outlines best practices for enabling embedded model…
-
OmniVLM: A Token-Compressed, Sub-Billion-Parameter Vision-Language Model for Efficient On-Device InferenceWei Chen, Zhi-Yuan Li, Shuo Xin · 2024 · arXiv.org · preprint, not peer reviewed · cited 12 times · Not summarized yet
Abstract
We present OmniVLM, a sub-billion-parameter vision-language model for efficient on-device inference. OmniVLM introduces a token compression mechanism that reduces visual token sequence length from 729 to 81 tokens, significantly reducing computational overhead while preserving visual-semantic fidelity. Through a multi-stage training pipeline of pretraining, supervised fine-tuning, and minimal-edit Direct Preference Optimization (DPO), OmniVLM matches the performance of larger models. On multiple benchmarks including ScienceQA, POPE, and MMMU, OmniVLM outperforms existing baselines like nanoLLAVA within a 968M-parameter footprint. Empirical results on the same laptop demonstrate 9.1x faster t…
-
AndroidGen: Building an Android Language Agent under Data ScarcityHan-Yu Lai, Junjie Gao, Xiao Liu et al. · 2025 · Annual Meeting of the Association for Computational Linguistics · cited 12 times · Not summarized yet
Abstract
Large language models have opened up a world of possibilities for various NLP tasks, sparking optimism for the future. Despite their potential, LLMs have yet to be widely used as agents on real mobile devices. The main challenge is the need for high-quality data sources. Time constraints and labor intensity often hinder human annotation. On the other hand, existing LLMs exhibit inadequate completion rates and need a robust data filtration strategy. Given these challenges, we develop a framework called AndroidGen to enhance the capabilities of LLM-based agents under data scarcity. In addition, we leverage AndroidGen to collect trajectories given human tasks and train open-source LLMs on these…
-
AgentCPM-GUI: Building Mobile-Use Agents with Reinforcement Fine-TuningZhong Zhang, Ya-Xi Lu, Yikun Fu et al. · 2025 · Conference on Empirical Methods in Natural Language Processing · cited 51 times · Not summarized yet
Abstract
The recent progress of large language model agents has opened new possibilities for automating tasks through graphical user interfaces (GUIs), especially in mobile environments where intelligent interaction can greatly enhance usability. However, practical deployment of such agents remains constrained by several key challenges. Existing training data is often noisy and lack semantic diversity, which hinders the learning of precise grounding and planning. Models trained purely by imitation tend to overfit to seen interface patterns and fail to generalize in unfamiliar scenarios. Moreover, most prior work focuses on English interfaces while overlooks the growing diversity of non-English applic…
Go further
How are these papers found?
We compare the title and abstract of this paper with those of every paper summarized on this site, and with the papers Semantic Scholar recommends, those it cites and those citing it (for a much-cited paper, its most recent citations and, from OpenAlex, the most cited ones). Among the prior work and the follow-ups, the papers that use each other in depth, a method it uses and the much-cited follow-ups come a little higher. The levels (very close, close, related) come from the similarity of the texts, not from a judgment of quality; * marks a paper without an abstract, compared by its title. The sentence under a paper is written by an AI from the two abstracts: check it in the papers. A paper marked not relevant by three readers leaves the list.
Updated on 3 November 2026.
The previous summary was created more than a year ago and can be re-run (if necessary) by clicking on the Run button below.
Disclaimer: The AI-based summarization tool and virtual assistant provided on this website may not always provide accurate and complete summaries or responses. We encourage you to carefully review and evaluate the generated content to ensure its quality and relevance to your needs.