# Grounding Large Language Models in Interactive Environments with Online Reinforcement Learning

**Authors:** Thomas Carta, Clément Romac, Thomas Wolf, Sylvain Lamprier, Olivier Sigaud, Pierre-Yves Oudeyer  
**arXiv:** [2302.02662](https://arxiv.org/abs/2302.02662)  
**Date:** 2023-02-06  
**Summary page:** https://www.summarizepaper.com/en/paper/2302.02662v1/

## Key points

- Authors explore the alignment between Large Language Models (LLMs) and their environment
- LLMs lack grounding, limiting their functional competence
- Proposed agent uses LLM as a policy and updates it through online Reinforcement Learning
- Study focuses on higher-level forms of functional grounding in interactive textual environment and spatial tasks
- Scientific questions addressed: Can LLMs enhance sample efficiency? How can LLMs boost generalization? What is the impact of online learning?
- Variants of FLAN-T5 are functionally grounded to investigate effects on learning and generalization
- Research contributes to understanding effective utilization of LLMs in decision-making processes
- Importance of aligning LLMs' knowledge with environment through functional grounding emphasized

## Summary

In the paper titled "Grounding Large Language Models in Interactive Environments with Online Reinforcement Learning," authors Thomas Carta, Clément Romac, Thomas Wolf, Sylvain Lamprier, Olivier Sigaud and Pierre-Yves Oudeyer explore the alignment between Large Language Models (LLMs) and their environment. LLMs have shown success in capturing abstract knowledge about the world's physics to solve decision-making problems but lack of grounding can limit their functional competence. To address this issue, an agent is proposed that uses an LLM as a policy and progressively updates it as it interacts with its environment by leveraging online Reinforcement Learning to improve performance in goal-solving tasks. The study focuses on higher-level forms of functional grounding using an interactive textual environment and a set of spatial and navigation tasks. The authors aim to answer several scientific questions: 1) Can LLMs enhance sample efficiency for online learning in various RL tasks? 2) How can LLMs boost different forms of generalization? 3) What is the impact of online learning? To investigate these questions, they functionally ground several variants (size and architecture) of FLAN-T5. By studying the effects of functional grounding on LLMs' abilities to learn and generalize across RL tasks, this research contributes to understanding how LLMs can be effectively utilized in decision-making processes. Overall, this paper highlights the importance of aligning LLMs' knowledge with their environment through functional grounding. The findings shed light on how LLMs can be leveraged to improve sample efficiency and generalization capabilities in RL tasks while emphasizing the impact of online learning.