Reinforcement learning (RL) techniques have shown significant success in quantitative investment tasks, including portfolio management and algorithmic trading. Intraday trading, which involves rapidly fluctuating values in the financial market, is particularly profitable but also risky. To address these limitations, this study proposes DeepScalper, a deep reinforcement learning framework specifically designed for intraday trading. The framework adopts an encoder-decoder architecture to learn robust market embedding that incorporates both macro-level and micro-level market information. This approach ensures that important details from the limit order book are not overlooked. Furthermore, DeepScalper introduces a novel hindsight reward function that provides the agent with a long-term horizon for capturing the overall price trend of the entire trading day. Additionally, a risk-aware auxiliary task is proposed in DeepScalper by predicting future volatility. By considering market risk while maximizing profit, this task enhances the agent's decision-making process. Extensive experiments were conducted on two stock index futures and four treasury bond futures to evaluate DeepScalper's performance. The results demonstrate significant improvements compared to many state-of-the-art approaches. In summary, DeepScalper addresses key limitations in applying RL methods to intraday trading by incorporating micro-level market information through an encoder-decoder architecture and capturing overall price trends using a hindsight reward function. The inclusion of a risk-aware auxiliary task further enhances decision-making capabilities. Experimental results confirm the effectiveness of DeepScalper in achieving improved performance compared to existing approaches.
Reinforcement learning (RL) techniques are ways to help computers make good decisions in investing. DeepScalper is a special computer program that uses RL to trade stocks during the day. It learns about the stock market using a special architecture called encoder-decoder. DeepScalper looks at both big and small details of the market to make smart decisions. It also has a new way of rewarding itself for making good choices based on overall price trends. DeepScalper also tries to predict how risky the market will be in the future. Many tests were done, and DeepScalper did better than other programs."
Definitions
- Reinforcement learning: A way for computers to learn from their actions and improve their decision-making skills.
- Quantitative investment tasks: Activities related to making money by buying and selling financial assets using mathematical models.
- Intraday trading: Buying and selling stocks within the same day.
- Encoder-decoder architecture: A specific design used in computer programs that helps them understand and process information.
- Market embedding: Understanding and representing information about the stock market in a useful way.
- Macro-level market information: Big-picture details about how the stock market is doing overall.
- Micro-level market information: Small details about individual stocks or specific events happening in the stock market.
- Hindsight reward function: A method of rewarding a computer program for making good choices based on past outcomes.
- Volatility: How much prices change over time, indicating how risky an investment might
DeepScalper: A Deep Reinforcement Learning Framework for Intraday Trading
The world of finance is ever-evolving, and with the advent of technology, new methods are being developed to maximize profits in the stock market. One such method is intraday trading, which involves rapidly fluctuating values in the financial market. This type of trading can be highly profitable but also carries a high risk factor due to its short-term nature. To address these limitations, this study proposes DeepScalper, a deep reinforcement learning framework specifically designed for intraday trading.
Encoder-Decoder Architecture
DeepScalper adopts an encoder-decoder architecture to learn robust market embedding that incorporates both macro-level and micro-level market information. This approach ensures that important details from the limit order book are not overlooked. The encoder part of the architecture takes input data from multiple sources including historical price data and current order book state information and produces an embedded representation of the market state at each time step. The decoder then uses this embedded representation as input to generate actionable decisions based on predicted future prices or volatility levels.
Hindsight Reward Function
In addition to incorporating micro-level market information through an encoder-decoder architecture, DeepScalper introduces a novel hindsight reward function that provides the agent with a long-term horizon for capturing overall price trends throughout the entire trading day. This reward function allows agents to adjust their strategies over time by taking into account past performance when making decisions about future trades.
Risk Awareness Auxiliary Task
To further enhance decision making capabilities, DeepScalper includes a risk aware auxiliary task by predicting future volatility levels using recurrent neural networks (RNNs). By considering potential risks while maximizing profit potentials, this task helps agents make more informed decisions when it comes to intraday trading activities.
Experimental Results
Extensive experiments were conducted on two stock index futures and four treasury bond futures markets in order to evaluate DeepScalper's performance compared to existing approaches such as Q learning and Monte Carlo Tree Search (MCTS). The results demonstrate significant improvements in terms of both profitability and risk management compared to many state-of-the art approaches used in quantitative investment tasks including portfolio management and algorithmic trading .
Conclusion
In summary, DeepScalper addresses key limitations in applying RL methods to intraday trading by incorporating micro level market information through an encoder decoder architecture and capturing overall price trends using a hindsight reward function along with a risk aware auxiliary task which further enhances decision making capabilities . Experimental results confirm the effectiveness of Deep Scalpers improved performance compared existing approaches used for quantitative investment tasks .
Reinforcement learning (RL) techniques have shown great success in quantitative investment tasks, such as portfolio management and algorithmic trading. Especially, intraday trading is one of the most profitable and risky tasks because of the intraday behaviors of the financial market that reflect billions of rapidly fluctuating values. However, it is hard to apply existing RL methods to intraday trading due to the following three limitations: 1) overlooking micro-level market information (e.g., limit order book); 2) only focusing on local price fluctuation and failing to capture the overall trend of the whole trading day; 3) neglecting the impact of market risk. To tackle these limitations, we propose DeepScalper, a deep reinforcement learning framework for intraday trading. Specifically, we adopt an encoder-decoder architecture to learn robust market embedding incorporating both macro-level and micro-level market information. Moreover, a novel hindsight reward function is designed to provide the agent a long-term horizon for capturing the overall price trend. In addition, we propose a risk-aware auxiliary task by predicting future volatility, which helps the agent take market risk into consideration while maximizing profit. Finally, extensive experiments on two stock index futures and four treasury bond futures demonstrate that DeepScalper achieves significant improvement against many state-of-the-art approaches.
