# Non-Adversarial Inverse Reinforcement Learning via Successor Feature Matching

**Authors:** Arnav Kumar Jain, Harley Wiltzer, Jesse Farebrother, Irina Rish, Glen Berseth, Sanjiban Choudhury  
**arXiv:** [2411.07007](https://arxiv.org/abs/2411.07007)  
**Date:** 2024-11-11  
**Summary page:** https://www.summarizepaper.com/en/paper/2411.07007v1/

## Key points

- Traditional approaches in inverse reinforcement learning (IRL) treat IRL as an adversarial game, with an adversary searching over reward models while a learner optimizes the reward through reinforcement learning procedures.
- A recent study titled "Non-Adversarial Inverse Reinforcement Learning via Successor Feature Matching" proposes a novel approach to IRL that deviates from the adversarial paradigm by focusing on direct policy optimization instead of adversarial optimization.
- The key innovation in this approach is exploiting a linear factorization of the return as the inner product of successor features and a reward vector, eliminating the need for explicitly learning a reward function.
- This non-adversarial method seamlessly integrates with existing actor-critic reinforcement learning algorithms and is applicable in state-only settings without requiring expert action labels.
- Empirical results show that this method can learn from as little as a single expert demonstration and achieve enhanced performance across various control tasks, contributing valuable insights into advancing IRL techniques.

## Summary

In the field of inverse reinforcement learning (IRL), where an agent aims to imitate expert demonstrations by interacting with its environment, traditional approaches have treated IRL as an adversarial game. In this setup, an adversary searches over reward models while a learner optimizes the reward through repeated reinforcement learning procedures. However, this game-solving framework is computationally intensive and challenging to stabilize. In a recent study titled "Non-Adversarial Inverse Reinforcement Learning via Successor Feature Matching," authors Arnav Kumar Jain, Harley Wiltzer, Jesse Farebrother, Irina Rish, Glen Berseth, and Sanjiban Choudhury propose a novel approach to IRL that deviates from the adversarial paradigm. Instead of focusing on learning a reward function through adversarial optimization, the researchers introduce a method based on direct policy optimization. The key innovation in their approach lies in exploiting a linear factorization of the return as the inner product of successor features and a reward vector. By designing an IRL algorithm that utilizes policy gradient descent on the discrepancy between learner and expert features, they eliminate the need for explicitly learning a reward function. This non-adversarial method seamlessly integrates with existing actor-critic reinforcement learning algorithms. One notable advantage of this new approach is its applicability in state-only settings without requiring expert action labels—a scenario where traditional behavior cloning methods fall short. The empirical results presented in the study showcase the effectiveness of their method, demonstrating that it can learn from as little as a single expert demonstration and achieve enhanced performance across various control tasks. Overall, this research contributes valuable insights into advancing IRL techniques by introducing a non-adversarial framework that simplifies computational complexity and enhances stability in training processes.