# PaLM-E: An Embodied Multimodal Language Model

**Authors:** Danny Driess, Fei Xia, Mehdi S. M. Sajjadi, Corey Lynch, Aakanksha Chowdhery, Brian Ichter, Ayzaan Wahid, Jonathan Tompson, Quan Vuong, Tianhe Yu, Wenlong Huang, Yevgen Chebotar, Pierre Sermanet, Daniel Duckworth, Sergey Levine, Vincent Vanhoucke, Karol Hausman, Marc Toussaint, Klaus Greff, Andy Zeng  
**arXiv:** [2303.03378](https://arxiv.org/abs/2303.03378)  
**Date:** 2023-03-06  
**Summary page:** https://www.summarizepaper.com/en/arxiv-id/2303.03378v1/

## Key points

- The PaLM-E model is an embodied multimodal language model designed for grounding in robotics problems.
- The model incorporates real-world continuous sensor modalities into language models, establishing a direct link between words and percepts.
- The input to the model consists of multi-modal sentences that interleave visual, continuous state estimation, and textual input encodings.
- PaLM-E can address various embodied reasoning tasks from different observation modalities on multiple embodiments.
- Positive transfer is exhibited by the model through diverse joint training across internet-scale language, vision, and visual-language domains.
- The largest PaLM-E-562B with 562B parameters is not only trained on robotics tasks but also a visual-language generalist with state-of-the-art performance on OK-VQA while retaining generalist language capabilities with increasing scale.
- Embodied language models could enable general inference in the real world for robotics problems by incorporating real-world sensor data directly into natural language processing systems.
- This approach has several potential benefits over traditional approaches to grounding in robotics problems, including more efficient use of resources and better integration between perception and action planning systems.

## Summary

The PaLM-E model is an embodied multimodal language model designed to address the challenge of grounding in robotics problems. The model incorporates real-world continuous sensor modalities into language models, establishing a direct link between words and percepts. The input to the model consists of multi-modal sentences that interleave visual, continuous state estimation, and textual input encodings. These encodings are trained end-to-end with a pre-trained large language model for multiple embodied tasks such as sequential robotic manipulation planning, visual question answering, and captioning. The evaluations show that PaLM-E can address various embodied reasoning tasks from different observation modalities on multiple embodiments. Additionally, the model exhibits positive transfer by benefiting from diverse joint training across internet-scale language, vision, and visual-language domains. The largest PaLM-E-562B with 562B parameters is not only trained on robotics tasks but also a visual-language generalist with state-of-the-art performance on OK-VQA while retaining generalist language capabilities with increasing scale. The authors of this paper propose that their embodied language models could enable general inference in the real world for robotics problems by incorporating real-world sensor data directly into natural language processing systems. This approach has several potential benefits over traditional approaches to grounding in robotics problems, including more efficient use of resources and better integration between perception and action planning systems. Overall, the PaLM-E model represents an exciting development in the field of natural language processing and robotics research. By integrating real world sensor data into natural language processing systems, this approach has the potential to enable more robust and effective communication between humans and robots in a wide range of contexts.