# Robust Robotic Control from Pixels using Contrastive Recurrent State-Space Models

**Authors:** Nitish Srivastava, Walter Talbott, Martin Bertran Lopez, Shuangfei Zhai, Josh Susskind  
**arXiv:** [2112.01163](https://arxiv.org/abs/2112.01163)  
**Date:** 2021-12-02  
**Summary page:** https://www.summarizepaper.com/en/paper/2112.01163v1/

## Key points

- The paper addresses the challenge of learning world models in unconstrained environments with high-dimensional observation spaces, such as images.
- The presence of irrelevant background distractions and unimportant visual details makes modeling challenging.
- The authors propose a recurrent latent dynamics model that predicts the next observation contrastively to overcome this issue.
- Training the model to predict future observations helps shape an agent's latent state space effectively and achieve robust robotic control.
- The proposed approach is compared to alternative methods like bisimulation methods and demonstrates superior performance.
- The Distracting Control Suite is used as a benchmark, and the approach achieves state-of-the-art results on this benchmark.
- This paper presents a novel approach to learning world models by leveraging contrastive prediction with a recurrent latent dynamics model.

## Summary

The paper titled "Robust Robotic Control from Pixels using Contrastive Recurrent State-Space Models" addresses the challenge of learning world models in unconstrained environments with high-dimensional observation spaces, such as images. One of the main difficulties in this task is the presence of irrelevant background distractions and unimportant visual details that make modeling challenging. To overcome this issue, the authors propose a recurrent latent dynamics model that predicts the next observation contrastively. By training their model to predict future observations, they are able to shape an agent's latent state space effectively and achieve robust robotic control even in the presence of simultaneous camera, background and color distractions. The authors compare their approach to alternative methods like bisimulation methods that rely on state-similarity measures derived from future reward or optimal actions and demonstrate superior performance. To evaluate their proposed method, the authors use the Distracting Control Suite which serves as a challenging benchmark for pixel-based robotic control. Their approach achieves state-of-the-art results on this benchmark, highlighting its effectiveness in addressing the challenges posed by irrelevant distractions and unimportant visual details. Overall, this paper presents a novel approach to learning world models in unconstrained environments by leveraging contrastive prediction with a recurrent latent dynamics model. The results demonstrate its robustness in controlling robots based on pixel inputs and its superiority over alternative methods.