# Voice2Action: Language Models as Agent for Efficient Real-Time Interaction in Virtual Reality

**Authors:** Yang Su  
**arXiv:** [2310.00092](https://arxiv.org/abs/2310.00092)  
**Date:** 2023-09-29  
**Summary page:** https://www.summarizepaper.com/en/paper/2310.00092v1/

## Key points

- The Voice2Action framework aims to address challenges of deploying Large Language Models (LLMs) in virtual reality (VR) environments.
- LLMs are task-driven autonomous agents trained to follow natural language instructions with few examples.
- Online interactions and complexity of manipulation categories in 3D environments have made it difficult to deploy LLMs in VR.
- Voice2Action framework hierarchically analyzes voice signals and textual commands through action and entity extraction.
- It divides execution tasks into canonical interaction subsets in real-time and prevents errors through environment feedback.
- Voice2Action enables more efficient and accurate performance compared to approaches without optimizations.
- Experiments conducted in an urban engineering VR environment using synthetic instruction data showed that Voice2Action outperformed other approaches without optimizations.
- This work highlights the potential of using LLMs as agents for efficient real-time interaction in VR.

## Summary

The Voice2Action framework, proposed by Yang Su from Cornell Tech, aims to address the challenges of deploying Large Language Models (LLMs) in virtual reality (VR) environments. LLMs are trained to follow natural language instructions with only a few examples and are designed as task-driven autonomous agents that can adapt to different execution environments. However, the lack of efficiency in online interactions and the complexity of manipulation categories in 3D environments have made it difficult to deploy LLMs in VR. The <Organization>Voice2Action</Organization> framework hierarchically analyzes customized voice signals and textual commands through action and entity extraction. It then divides the execution tasks into canonical interaction subsets in real-time while also preventing errors through feedback from the environment. The goal is to enable more efficient and accurate performance compared to approaches without optimizations. To evaluate the effectiveness of <Organization>Voice2Action</Organization>, experiments were conducted in an urban engineering VR environment using synthetic instruction data. The results demonstrated that <Organization>Voice2Action</Organization> outperformed other approaches without optimizations. This work highlights the potential of using LLMs as agents for efficient real-time interaction in VR.