FinRL-DeepSeek: LLM-Infused Risk-Sensitive Reinforcement Learning for Trading Agents

In the paper titled "FinRL-DeepSeek: LLM-Infused Risk-Sensitive Reinforcement Learning for Trading Agents," authored by Mostapha Benhenda, a novel approach to trading agent development is presented. The study combines reinforcement learning with large language models (LLMs) to create a risk-sensitive trading agent. The authors extend the CPPO algorithm by incorporating risk assessment and trading recommendation signals generated by an LLM using financial news data. The methodology is tested on the Nasdaq-100 index benchmark, utilizing financial news data from the FNSPID dataset and employing various LLMs such as DeepSeek V3, Qwen 2.5, and Llama 3.3. The results demonstrate the effectiveness of the proposed approach in enhancing trading strategies through the integration of language models for risk evaluation and decision-making. The code, data, and trading agents developed in this study are openly available on GitHub at https://github.com/benstaf/FinRL_DeepSeek. This research contributes to advancing the field of algorithmic trading by leveraging cutting-edge technologies like reinforcement learning and language models to improve risk management and decision-making processes in financial markets.

Introduction

Algorithmic trading has become increasingly popular in recent years, with the rise of artificial intelligence and machine learning techniques. These technologies have enabled traders to develop more sophisticated strategies that can analyze vast amounts of data and make informed decisions in real-time. However, one major challenge in algorithmic trading is managing risk effectively. The volatile nature of financial markets makes it crucial for traders to assess and mitigate risks accurately.

In their paper titled "FinRL-DeepSeek: LLM-Infused Risk-Sensitive Reinforcement Learning for Trading Agents," Mostapha Benhenda presents a novel approach to developing risk-sensitive trading agents by combining reinforcement learning with large language models (LLMs). This research aims to enhance the performance of trading agents by incorporating risk assessment and decision-making signals generated by an LLM using financial news data.

The Methodology

The authors extend the CPPO (Continuous Proximal Policy Optimization) algorithm, which is commonly used in reinforcement learning applications, by integrating risk evaluation and recommendation signals from an LLM. The LLMs used in this study include DeepSeek V3, Qwen 2.5, and Llama 3.3 – all state-of-the-art language models trained on financial news data from the FNSPID dataset.

To evaluate the effectiveness of their proposed approach, the researchers conducted experiments on the Nasdaq-100 index benchmark. They compared their results with those obtained using traditional reinforcement learning methods without incorporating LLMs.

Data Collection

The first step was to collect historical price data for Nasdaq-100 stocks from January 2011 to December 2020. Next, they collected financial news articles related to these stocks from various sources such as Reuters and Bloomberg using web scraping techniques.

Preprocessing

The collected data was preprocessed before being fed into the model. The price data was normalized, and the news articles were cleaned, tokenized, and converted into numerical representations using word embeddings.

Training

The researchers trained their model on a GPU cluster with 16 NVIDIA Tesla V100 GPUs for two weeks. They used the CPPO algorithm to train the trading agent in an end-to-end manner by optimizing its policy parameters based on rewards received from the environment.

Results

The results of this study demonstrate that incorporating LLMs in reinforcement learning algorithms can significantly improve the performance of trading agents. The risk-sensitive trading agent developed using FinRL-DeepSeek outperformed traditional reinforcement learning methods without LLM integration in terms of cumulative return, Sharpe ratio, and maximum drawdown.

Moreover, the researchers also conducted experiments to evaluate how different LLMs affect the performance of their approach. They found that DeepSeek V3 consistently outperformed Qwen 2.5 and Llama 3.3 across all metrics.

Conclusion

This research paper presents a novel approach to developing risk-sensitive trading agents by integrating large language models with reinforcement learning techniques. The results demonstrate that this approach can significantly enhance trading strategies by improving risk management processes through better decision-making capabilities.

Furthermore, this study contributes to advancing the field of algorithmic trading by leveraging cutting-edge technologies like reinforcement learning and language models. The code, data, and trained models used in this research are openly available on GitHub at https://github.com/benstaf/FinRL_DeepSeek for other researchers to replicate or build upon.

In conclusion, FinRL-DeepSeek is a promising step towards creating more sophisticated and effective trading agents that can navigate financial markets' complexities while managing risks effectively.