# Autonomous Penetration Testing using Reinforcement Learning

**Authors:** Jonathon Schwartz, Hanna Kurniawati  
**arXiv:** [1905.05965](https://arxiv.org/abs/1905.05965)  
**Date:** 2019-05-15  
**Summary page:** https://www.summarizepaper.com/en/paper/1905.05965v1/

## Key points

- Penetration testing (pentesting) is a crucial practice in cybersecurity that involves controlled attacks on computer systems to evaluate security measures.
- There is a shortage of qualified cybersecurity experts, leading researchers to explore the integration of artificial intelligence techniques for automated pentesting.
- Model-free Reinforcement Learning (RL) has been investigated as an alternative to model-based planning for automated pentesting, allowing agents to learn optimal policies through interactions with the system.
- A fast and low-compute simulator was developed to train and evaluate autonomous pentesting agents, conceptualizing pentesting as a Markov Decision Process using network configurations as states, scans and exploits as actions, and machine values within the network as rewards.
- Implementations of the standard Q-learning algorithm in tabular and neural network forms showed promising results in identifying optimal attack paths across various network topologies and sizes without requiring an explicit model of action behavior.

## Summary

, , , , 

In the realm of cybersecurity, penetration testing (pentesting) is a crucial practice that involves conducting controlled attacks on computer systems to evaluate their security measures. However, this process typically demands highly skilled professionals, and the industry is facing a shortage of qualified cyber security experts. To address this challenge, researchers have been exploring the integration of artificial intelligence techniques to automate pentesting procedures. One prevalent approach to automated pentesting has been model-based planning. Nevertheless, with the rapidly evolving cyber security landscape, maintaining up-to-date models of exploits has proven to be a daunting task. In light of this, a recent study delved into the application of model-free Reinforcement Learning (RL) for automated pentesting. Unlike model-based planning, model-free RL does not rely on an explicit model of the environment but rather learns optimal policies through interactions with the system. The research team embarked on designing and constructing a fast and low-compute simulator tailored for training and evaluating autonomous pentesting agents. They conceptualized pentesting as a Markov Decision Process where network configurations served as states, available scans and exploits as actions, and rewards were determined by machine values within the network. Subsequently, they utilized this simulator to explore the feasibility of employing model-free RL in pentesting scenarios. By implementing the standard Q-learning algorithm in both tabular and neural network forms, the researchers observed promising results within their simulated environment. Both implementations successfully identified optimal attack paths across various network topologies and sizes without necessitating an explicit model of action behavior. However, it was noted that these algorithms were primarily practical for smaller networks with limited actions. Moving forward, further advancements are deemed necessary in developing scalable RL algorithms to accommodate larger and more complex environments for automated pentesting applications. This study sheds light on the potential efficacy of leveraging Reinforcement Learning in enhancing cybersecurity practices through autonomous penetration testing methodologies.