# Traffic expertise meets residual RL: Knowledge-informed model-based residual reinforcement learning for CAV trajectory control

**Authors:** Zihao Sheng, Zilin Huang, Sikai Chen  
**arXiv:** [2408.17380](https://arxiv.org/abs/2408.17380)  
**Date:** 2024-08-30  
**Summary page:** https://www.summarizepaper.com/en/paper/2408.17380v1/

## Key points

- Model-based reinforcement learning (RL) can enhance sample efficiency compared to model-free RL by leveraging a virtual environment model
- Accurately representing environmental dynamics in complex systems poses a significant challenge due to uncertainties
- Inaccuracies in the environment model can hinder the performance and sample efficiency of model-based RL approaches
- Traditional model-based RL methods often require extensive training time from scratch, limiting their effectiveness compared to model-free approaches
- The novel knowledge-informed model-based residual reinforcement learning framework tailored for CAV trajectory control tasks integrates traffic expertise into the learning process to improve efficiency and avoid starting from scratch
- The approach combines the Intelligent Driver Model (IDM) for fundamental dynamics with neural networks for residual dynamics, ensuring adaptability to complex scenarios while enhancing learning efficiency
- The strategy combines traditional control methods with residual RL techniques, enabling efficient learning and policy optimization without complete retraining
- Experimental results demonstrate that the proposed approach outperforms baseline agents in terms of sample efficiency, traffic flow smoothness, and overall traffic mobility

## Summary

Model-based reinforcement learning (RL) has the potential to enhance sample efficiency compared to model-free RL by leveraging a virtual environment model. However, accurately representing environmental dynamics in complex systems poses a significant challenge due to uncertainties. Inaccuracies in the environment model can hinder the performance and sample efficiency of model-based RL approaches. Despite its advantages, traditional model-based RL methods often require extensive training time from scratch, limiting their effectiveness compared to model-free approaches. To address these challenges, this paper introduces a novel knowledge-informed model-based residual reinforcement learning framework tailored for CAV trajectory control tasks. By incorporating established traffic expert knowledge into the learning process, the framework aims to improve learning efficiency and circumvent the need to start from square one. The approach integrates traffic expertise into a virtual environment model, utilizing the Intelligent Driver Model (IDM) for fundamental dynamics and neural networks for residual dynamics. This hybrid approach ensures adaptability to complex scenarios while enhancing learning efficiency. The proposed strategy combines traditional control methods with residual RL techniques, enabling efficient learning and policy optimization without requiring complete retraining. The framework is specifically applied to CAV trajectory control tasks aimed at mitigating stop-and-go waves in mixed traffic flow scenarios. Experimental results demonstrate that the proposed approach outperforms baseline agents in terms of sample efficiency, traffic flow smoothness, and overall traffic mobility. The work by Zihao Sheng, Zilin Huang, and Sikai Chen showcases how infusing expert knowledge into a model-based RL framework can significantly enhance performance in CAV trajectory control tasks. The source code and supplementary materials for this study are available at https://github.com/zihaosheng/traffic-expertise-RL/.