# Octopus v2: On-device language model for super agent

**Authors:** Wei Chen, Zhiyuan Li  
**arXiv:** [2404.01744](https://arxiv.org/abs/2404.01744)  
**Date:** 2024-04-02  
**Summary page:** https://www.summarizepaper.com/en/paper/2404.01744v1/

## Key points

- Language models have proven effective in software applications, particularly in tasks related to automatic workflow
- Large-scale language models show high performance in cloud environments but raise concerns over privacy and cost
- On-device models for function calling face challenges such as latency and accuracy issues
- Research introduces a new method empowering on-device model with 2 billion parameters to outperform GPT-4 in accuracy and latency while reducing context length by 95%
- Efforts are underway to deploy smaller-scale Large Language Models (LLMs) on edge devices like PCs and smartphones due to memory limitations and lower inference speeds
- Open-source models of manageable sizes like Gemma-2B, Gemma-7B, StableCode-3B, and Llama-7B introduced for deployment on edge devices
- Advancements in function-calling capabilities of smaller-scale models observed through projects like NexusRaven, Toolformer, ToolAlpaca, Gorrilla, ToolLlama,and Taskmatrix
- Field of AI agents evolving rapidly with advancements in AI assistant tools like MultiOn and Adept AI along with AI consumer products like Rabbit R1 and Humane AI Pin gaining traction

## Summary

In recent years, language models have proven to be highly effective in various software applications, particularly in tasks related to automatic workflow. These models possess the crucial ability to call functions, which is essential for creating AI agents. However, while large-scale language models have shown high performance in cloud environments, they are often associated with concerns over privacy and cost. On the other hand, on-device models for function calling face challenges such as latency and accuracy issues. To address these challenges, our research introduces a new method that empowers an on-device model with 2 billion parameters to outperform GPT-4 in both accuracy and latency while reducing the context length by 95%. By comparing our method to Llama-7B with a RAG-based function calling mechanism, we were able to enhance latency by 35-fold. This significant improvement reduces latency levels to a point where deployment across various edge devices in production environments becomes feasible, meeting the performance requirements for real-world applications. Additionally, efforts are underway to deploy smaller-scale Large Language Models (LLMs) on edge devices like PCs and smartphones due to memory limitations and lower inference speeds. Open-source models of manageable sizes such as Gemma-2B, Gemma-7B, StableCode-3B, and Llama-7B have been introduced for this purpose. Research initiatives like Llama cpp and the MLC LLM framework enable the operation of 7B language models on mobile phones and other edge devices across different hardware platforms. Furthermore, advancements in function-calling capabilities of smaller-scale models have been observed through projects like NexusRaven, Toolformer, ToolAlpaca, Gorrilla, ToolLlama,and Taskmatrix. These projects demonstrate that 7B and 13B models can effectively call external APIs comparable to GPT-4 using a RAG-based method for function calling. Overall, the field of AI agents is rapidly evolving with advancements in AI assistant tools like MultiOn and Adept AI along with AI consumer products like Rabbit R1 and Humane AI Pin gaining traction. The use of language models in developing dependable software that empowers users through API calling and reasoning abilities showcases promising trends in the industry. Despite these advancements, concerns remain regarding privacy issues associated with reliance on cloud-based models along with inference costs and the need for constant Wi-Fi connectivity.