# Multi-Agent Reinforcement Learning for Fast-Timescale Demand Response of Residential Loads

**Authors:** Vincent Mai, Philippe Maisonneuve, Tianyu Zhang, Hadi Nekoei, Liam Paull, Antoine Lesage-Landry  
**arXiv:** [2301.02593](https://arxiv.org/abs/2301.02593)  
**Date:** 2023-01-06  
**Summary page:** https://www.summarizepaper.com/en/paper/2301.02593v1/

## Key points

- The challenge of integrating high amounts of renewable energy resources into electrical power grids requires coping with high amplitude and fast timescale variations in power generation.
- Frequency regulation using demand response is proposed as a solution, which coordinates temporally flexible loads such as air conditioners to counteract these variations.
- A decentralized agent trained with multi-agent proximal policy optimization (MA-PPO) with localized communication is proposed to overcome existing approaches for discrete control struggling to provide satisfactory performance for fast timescale action selection with hundreds of agents.
- Two communication frameworks are explored: hand-engineered or learned through targeted multi-agent communication.
- The resulting policies perform well and robustly for frequency regulation and scale seamlessly to arbitrary numbers of houses for constant processing times.
- An open-source, multi-agent environment simulating the real-world problem of frequency regulation through demand response at the second timescale is presented; this simulator is compatible with the OpenAI Gym framework.
- The main contributions of this paper are threefold: development of a decentralized agent trained by MA-PPO that can handle fast timescale action selection with hundreds of agents; two local communication frameworks that outperform baselines; an open-source multi-agent environment that simulates real-world problems related to frequency regulation through demand response.
- This work demonstrates how multi-agent reinforcement learning (MARL) can be used successfully to solve complex multi-agent problems induced by renewable energy integration in electrical power grids.
- Future works could include sim2real transfer and integration of more complex flexible loads as well as addressing power grid safety issues.

## Summary

This paper addresses the challenge of integrating high amounts of renewable energy resources into electrical power grids, which requires the ability to cope with high amplitude and fast timescale variations in power generation. The authors propose a solution through frequency regulation using demand response, which coordinates temporally flexible loads such as air conditioners to counteract these variations. To overcome existing approaches for discrete control with dynamic constraints struggling to provide satisfactory performance for fast timescale action selection with hundreds of agents, the authors propose a decentralized agent trained with multi-agent proximal policy optimization (MA-PPO) with localized communication. They explore two communication frameworks: hand-engineered or learned through targeted multi-agent communication. The resulting policies perform well and robustly for frequency regulation and scale seamlessly to arbitrary numbers of houses for constant processing times. An open-source, multi-agent environment simulating the real-world problem of frequency regulation through demand response at the second timescale is also presented; this simulator is compatible with the OpenAI Gym framework. The main contributions of this paper are threefold: first, development of a decentralized agent trained by MA-PPO that can handle fast timescale action selection with hundreds of agents; second, two local communication frameworks that outperform baselines; and third, an open-source multi-agent environment that simulates real-world problems related to frequency regulation through demand response. Overall, this work demonstrates how multi-agent reinforcement learning (MARL) can be used successfully to solve complex multi-agent problems induced by renewable energy integration in electrical power grids. Future works could include sim2real transfer and integration of more complex flexible loads as well as addressing power grid safety issues.