# STEVE-1: A Generative Model for Text-to-Behavior in Minecraft

**Authors:** Shalev Lifshitz, Keiran Paster, Harris Chan, Jimmy Ba, Sheila McIlraith  
**arXiv:** [2306.00937](https://arxiv.org/abs/2306.00937)  
**Date:** 2023-06-01  
**Summary page:** https://www.summarizepaper.com/en/paper/2306.00937v1/

## Key points


## Summary

In the field of Artificial Intelligence, constructing models that can respond to text instructions is a challenging task, particularly for sequential decision-making tasks. However, a recent study by Shalev Lifshitz, Keiran Paster, Harris Chan, Jimmy Ba and Sheila McIlraith introduces an innovative approach to this problem. They present STEVE-1 - an instruction-tuned Video Pretraining (VPT) model for Minecraft that demonstrates the effectiveness of the unCLIP approach used in DALL-E 2 for creating instruction-following sequential decision-making agents. STEVE-1 is trained in two steps: first adapting the pretrained VPT model to follow commands in MineCLIP's latent space and then training a prior to predict latent codes from text. This allows them to finetune VPT through self-supervised behavioral cloning and hindsight relabeling without requiring costly human text annotations. By leveraging pretrained models like VPT and MineCLIP and employing best practices from text-conditioned image generation techniques, STEVE-1 costs just $60 to train and can follow a wide range of short-horizon open-ended text and visual instructions in Minecraft. The authors provide experimental evidence highlighting key factors for downstream performance such as pretraining, classifier-free guidance and data scaling. They also demonstrate that STEVE-1 sets a new bar for open ended instruction following in Minecraft with low level controls (mouse and keyboard) and raw pixel inputs far outperforming previous baselines. Overall, this study presents an innovative solution to the challenge of constructing AI models that can respond effectively to text instructions. The authors' use of pretrained models like VPT and MineCLIP along with their self supervised behavioral cloning approach provides a cost effective way to create highly effective instruction following agents like STEVE 1. All resources including model weights, training scripts and evaluation tools are made available for further research purposes.