# Reinforcement Learning with Exogenous States and Rewards

**Authors:** George Trimponias, Thomas G. Dietterich  
**arXiv:** [2303.12957](https://arxiv.org/abs/2303.12957)  
**Date:** 2023-03-22  
**Summary page:** https://www.summarizepaper.com/en/paper/2303.12957v1/

## Key points

- Reinforcement learning can be hindered by exogenous state variables and rewards
- External factors introduce uncontrolled variation into the reward signal, making optimization difficult
- Authors George Trimponias and Thomas G. Dietterich formalize exogenous factors in their paper
- Reward function is broken down into endogenous and exogenous components to split the MDP
- Optimal policy for endogenous MDP also serves as optimal policy for original MDP
- Algorithms are introduced to identify intertwined exogenous and endogenous subspaces
- These algorithms help uncover exogenous space, eliminate exogenous rewards, and focus on solving for endogenous MDP
- Methods successfully accelerate reinforcement learning processes in real-time settings
- Strategic decomposition of state variables and rewards leads to more efficient solutions

## Summary

Reinforcement learning can be hindered by the presence of exogenous state variables and rewards. These external factors introduce uncontrolled variation into the reward signal, making it difficult to optimize for the desired outcome. In their paper titled "Reinforcement Learning with Exogenous States and Rewards," authors George Trimponias and Thomas G. Dietterich delve into the formalization of these exogenous factors. They demonstrate that by breaking down the reward function into endogenous and exogenous components, the Markov Decision Process (MDP) at hand can be split into an exogenous Markov Reward Process (based on the exogenous reward) and an endogenous Markov Decision Process (focused on optimizing the endogenous reward). The key insight provided is that any optimal policy for the endogenous MDP also serves as an optimal policy for the original MDP. This approach is especially beneficial as solving for the endogenous MDP becomes a more manageable task due to typically lower variance associated with its reward. The authors further explore scenarios where there is no predefined decomposition of state space into exogenous and endogenous components; instead, these subspaces must be discovered. To address this challenge, they introduce algorithms that effectively identify these subspaces even when they are intertwined through linear combinations. These algorithms can be seamlessly integrated into reinforcement learning processes to uncover the exogenous space, eliminate exogenous rewards, and shift focus towards solving for the endogenous MDP. Through experiments conducted on various complex synthetic MDPs, Trimponias and Dietterich showcase how these methods successfully unveil substantial exogenous state spaces in real-time settings. This leads to significant accelerations in reinforcement learning processes and highlights a novel approach to handling exogeneous factors in such settings. By strategically decomposing state variables and rewards into distinct components, researchers and practitioners can streamline optimization processes and achieve more efficient solutions in challenging environments.