DeepScalper: A Risk-Aware Deep Reinforcement Learning Framework for Intraday Trading with Micro-level Market Embedding
Authors: Shuo Sun, Rundong Wang, Xu He, Junlei Zhu, Jian Li, Bo An
Abstract
Reinforcement learning (RL) techniques have shown great success in quantitative investment tasks, such as portfolio management and algorithmic trading. Especially, intraday trading is one of the most profitable and risky tasks because of the intraday behaviors of the financial market that reflect billions of rapidly fluctuating values. However, it is hard to apply existing RL methods to intraday trading due to the following three limitations: 1) overlooking micro-level market information (e.g., limit order book); 2) only focusing on local price fluctuation and failing to capture the overall trend of the whole trading day; 3) neglecting the impact of market risk. To tackle these limitations, we propose DeepScalper, a deep reinforcement learning framework for intraday trading. Specifically, we adopt an encoder-decoder architecture to learn robust market embedding incorporating both macro-level and micro-level market information. Moreover, a novel hindsight reward function is designed to provide the agent a long-term horizon for capturing the overall price trend. In addition, we propose a risk-aware auxiliary task by predicting future volatility, which helps the agent take market risk into consideration while maximizing profit. Finally, extensive experiments on two stock index futures and four treasury bond futures demonstrate that DeepScalper achieves significant improvement against many state-of-the-art approaches.
The papers around it, by date and closeness
Higher is closer to this paper; left is older. Lines join papers close to each other: groups of lines are sub-topics. Select a point to see why it is there. Back to the summary of this paper
Papers around this one
Ranked by how close their title and abstract are to this paper’s. Select some to read them together.
Closest papers
-
FinRL: Deep Reinforcement Learning Framework to Automate Trading in Quantitative FinanceXiao-Yang Liu, Hongyang Yang, Jiechao Gao et al. · 2021 · ACM International Conference on AI in Finance, 2021 · cited 189 times · AI summary available
FinRL presents an open-source pipeline for building quantitative trading agents, whereas DeepScalper proposes a specialized intraday trading framework.
Abstract
Deep reinforcement learning (DRL) has been envisioned to have a competitive edge in quantitative finance. However, there is a steep development curve for quantitative traders to obtain an agent that automatically positions to win in the market, namely \textit{to decide where to trade, at what price} and \textit{what quantity}, due to the error-prone programming and arduous debugging. In this paper, we present the first open-source framework \textit{FinRL} as a full pipeline to help quantitative traders overcome the steep learning curve. FinRL is featured with simplicity, applicability and extensibility under the key principles, \textit{full-stack framework, customization, reproducibility} an…
-
Deep reinforcement learning with positional context for intraday tradingSven Goluža, Tomislav Kovačević, Tessa Bauman et al. · 2024 · Evolutionary Systematics · cited 9 times · Not summarized yet
Abstract
Deep reinforcement learning (DRL) is a well-suited approach to financial decision-making, where an agent makes decisions based on its trading strategy developed from market observations. Existing DRL intraday trading strategies mainly use price-based features to construct the state space. They neglect the contextual information related to the position of the strategy, which is an important aspect given the sequential nature of intraday trading. In this study, we propose a novel DRL model for intraday trading that introduces positional features encapsulating the contextual information into its sparse state space. The model is evaluated over an extended period of almost a decade and across var…
-
DeepTrader: A Deep Reinforcement Learning Approach for Risk-Return Balanced Portfolio Management with Market Conditions EmbeddingZhicheng Wang, Bi-Wei Huang, Shikui Tu et al. · 2021 · AAAI Conference on Artificial Intelligence · cited 187 times · Not summarized yet
Abstract
Most existing reinforcement learning (RL)-based portfolio management models do not take into account the market conditions, which limits their performance in risk-return balancing. In this paper, we propose DeepTrader, a deep RL method to optimize the investment policy. In particular, to tackle the risk-return balancing problem, our model embeds macro market conditions as an indicator to dynamically adjust the proportion between long and short funds, to lower the risk of market fluctuations, with the negative maximum drawdown as the reward function. Additionally, the model involves a unit to evaluate individual assets, which learns dynamic patterns from historical data with the price rising …
-
Deep reinforcement learning on a multi-asset environment for tradingAli Hirsa, Joerg Osterrieder, Branka Hadji-Misheva et al. · 2021 · Trading and Market Microstructure · cited 9 times · AI summary available
This study uses a DQN with volatility-normalized daily returns to trade futures, while DeepScalper adds micro-level market embedding and risk-aware learning for intraday trading.
Abstract
Financial trading has been widely analyzed for decades with market participants and academics always looking for advanced methods to improve trading performance. Deep reinforcement learning (DRL), a recently reinvigorated method with significant success in multiple domains, still has to show its benefit in the financial markets. We use a deep Q-network (DQN) to design long-short trading strategies for futures contracts. The state space consists of volatility-normalized daily returns, with buying or selling being the reinforcement learning action and the total reward defined as the cumulative profits from our actions. Our trading strategy is trained and tested both on real and simulated price…
-
FinRL: A Deep Reinforcement Learning Library for Automated Stock Trading in Quantitative FinanceXiao-Yang Liu, Hongyang Yang, Qian Chen et al. · 2020 · Trading and Market Microstructure · cited 213 times · AI summary available
The FinRL library provides tutorials and virtual environments for developing stock-trading agents, while DeepScalper focuses on a risk-aware intraday futures strategy.
Abstract
As deep reinforcement learning (DRL) has been recognized as an effective approach in quantitative finance, getting hands-on experiences is attractive to beginners. However, to train a practical DRL trading agent that decides where to trade, at what price, and what quantity involves error-prone and arduous development and debugging. In this paper, we introduce a DRL library FinRL that facilitates beginners to expose themselves to quantitative finance and to develop their own stock trading strategies. Along with easily-reproducible tutorials, FinRL library allows users to streamline their own developments and to compare with existing schemes easily. Within FinRL, virtual environments are confi…
-
Deep Reinforcement Learning Approach for Trading Automation in The Stock MarketTaylan Kabbani, Ekrem Duman · 2022 · Trading and Market Microstructure · cited 89 times · AI summary available
This work formulates stock trading as a POMDP and combines price prediction with portfolio allocation, whereas DeepScalper targets intraday futures using micro-level data and volatility prediction.
Abstract
Deep Reinforcement Learning (DRL) algorithms can scale to previously intractable problems. The automation of profit generation in the stock market is possible using DRL, by combining the financial assets price "prediction" step and the "allocation" step of the portfolio in one unified process to produce fully autonomous systems capable of interacting with their environment to make optimal decisions through trial and error. This work represents a DRL model to generate profitable trades in the stock market, effectively overcoming the limitations of supervised learning approaches. We formulate the trading problem as a Partially Observed Markov Decision Process (POMDP) model, considering the con…
-
Deep Reinforcement Learning in Quantitative Algorithmic Trading: A ReviewTidor-Vlad Pricope · 2021 · Machine Learning · preprint, not peer reviewed · cited 59 times · AI summary available
Abstract
Algorithmic stock trading has become a staple in today's financial market, the majority of trades being now fully automated. Deep Reinforcement Learning (DRL) agents proved to be to a force to be reckon with in many complex games like Chess and Go. We can look at the stock market historical price series and movements as a complex imperfect information environment in which we try to maximize return - profit and minimize risk. This paper reviews the progress made so far with deep reinforcement learning in the subdomain of AI in finance, more precisely, automated low-frequency quantitative stock trading. Many of the reviewed studies had only proof-of-concept ideals with experiments conducted in…
-
Safe-FinRL: A Low Bias and Variance Deep Reinforcement Learning Implementation for High-Freq Stock TradingZitao Song, Xuyang Jin, Chenliang Li · 2022 · Portfolio Management · preprint, not peer reviewed · cited 3 times · AI summary available
Abstract
In recent years, many practitioners in quantitative finance have attempted to use Deep Reinforcement Learning (DRL) to build better quantitative trading (QT) strategies. Nevertheless, many existing studies fail to address several serious challenges, such as the non-stationary financial environment and the bias and variance trade-off when applying DRL in the real financial market. In this work, we proposed Safe-FinRL, a novel DRL-based high-freq stock trading strategy enhanced by the near-stationary financial environment and low bias and variance estimation. Our main contributions are twofold: firstly, we separate the long financial time series into the near-stationary short environment; seco…
Prior work it builds on
-
Deep Direct Reinforcement Learning for Financial Signal Representation and TradingYue Deng, Feng Bao, Youyong Kong et al. · 2017 · IEEE Transactions on Neural Networks and Learning Systems · cited 826 times · Not summarized yet
-
Adaptive Quantitative Trading: An Imitative Deep Reinforcement Learning ApproachYang Liu, Qi Liu, Hong-Ke Zhao et al. · 2020 · AAAI Conference on Artificial Intelligence · cited 164 times · Not summarized yet
Abstract
In recent years, considerable efforts have been devoted to developing AI techniques for finance research and applications. For instance, AI techniques (e.g., machine learning) can help traders in quantitative trading (QT) by automating two tasks: market condition recognition and trading strategies execution. However, existing methods in QT face challenges such as representing noisy high-frequent financial data and finding the balance between exploration and exploitation of the trading agent with AI techniques. To address the challenges, we propose an adaptive trading model, namely iRDPG, to automatically develop QT strategies by an intelligent trading agent. Our model is enhanced by deep rei…
-
A Deep Reinforcement Learning Framework for the Financial Portfolio Management ProblemZhengyao Jiang, Dixing Xu, Jinjun Liang · 2017 · Computational Finance · preprint, not peer reviewed · cited 522 times · AI summary available
Abstract
Financial portfolio management is the process of constant redistribution of a fund into different financial products. This paper presents a financial-model-free Reinforcement Learning framework to provide a deep machine learning solution to the portfolio management problem. The framework consists of the Ensemble of Identical Independent Evaluators (EIIE) topology, a Portfolio-Vector Memory (PVM), an Online Stochastic Batch Learning (OSBL) scheme, and a fully exploiting and explicit reward function. This framework is realized in three instants in this work with a Convolutional Neural Network (CNN), a basic Recurrent Neural Network (RNN), and a Long Short-Term Memory (LSTM). They are, along wi…
-
Model-based Deep Reinforcement Learning for Dynamic Portfolio OptimizationPengqian Yu, J. Lee, Ilya Kulyatin et al. · 2019 · arXiv.org · preprint, not peer reviewed · cited 79 times · Not summarized yet
Abstract
Dynamic portfolio optimization is the process of sequentially allocating wealth to a collection of assets in some consecutive trading periods, based on investors' return-risk profile. Automating this process with machine learning remains a challenging problem. Here, we design a deep reinforcement learning (RL) architecture with an autonomous trading agent such that, investment decisions and actions are made periodically, based on a global objective, with autonomy. In particular, without relying on a purely model-free RL agent, we train our trading agent using a novel RL architecture consisting of an infused prediction module (IPM), a generative adversarial data augmentation module (DAM) and …
-
Quantitative Stock Investment by Routing Uncertainty-Aware Trading Experts: A Multi-Task Learning ApproachShuo Sun, Rundong Wang, Bo An · 2022 · Trading and Market Microstructure · preprint, not peer reviewed · cited 5 times · AI summary available
Abstract
Quantitative investment is a fundamental financial task that highly relies on accurate stock prediction and profitable investment decision making. Despite recent advances in deep learning (DL) have shown stellar performance on capturing trading opportunities in the stochastic stock market, we observe that the performance of existing DL methods is sensitive to random seeds and network initialization. To design more profitable DL methods, we analyze this phenomenon and find two major limitations of existing works. First, there is a noticeable gap between accurate financial predictions and profitable investment strategies. Second, investment decisions are made based on only one individual predi…
Papers that cite it
-
A Methodology for Developing Deep Reinforcement Learning Trading Strategies: A Case Study in the Futures MarketLeonardo Conegundes, Adriano C. M. Pereira · 2024 · IEEE Conference on Computational Intelligence for Financial Engineering & Economics · cited 1 time · Not summarized yet
Abstract
Quantitative algorithmic trading and machine learning have transformed financial markets by introducing advanced computational techniques for market analysis, decision-making, risk management, and order execution. This paper explores Deep Reinforcement Learning (DRL) for developing day trading strategies within the Ibovespa futures market, the world's most traded stock index futures contract. We chose the Ibovespa market for its high liquidity and volatility, which present unique challenges and opportunities for trading. We propose a novel methodology for the development, evaluation, and deployment of DRL-powered strategies. This methodology involves model generation, approval, selection, an…
-
TradeMaster: A Holistic Quantitative Trading Platform Empowered by Reinforcement LearningShuo Sun, Molei Qin, Wen-Tao Zhang et al. · 2023 · Neural Information Processing Systems · cited 40 times · Not summarized yet
Abstract
The financial markets, which involve over $90 trillion market capitals, attract the attention of innumerable profit-seeking investors globally. Recent explosion of reinforcement learning in financial trading (RLFT) research has shown stellar performance on many quantitative trading tasks. However, it is still challenging to deploy reinforcement learning (RL) methods into real-world financial markets due to the highly composite nature of this domain, which entails design choices and interactions between components that collect financial data, conduct feature engineering, build market environments, make investment decisions, evaluate model behaviors and provides user interfaces. Despite the av…
-
Evolution and Challenges of Reinforcement Learning in Intraday Algorithmic Trading: A Systematic Literature ReviewT. Amorgianiotis, G. Beligiannis, E. Georgopoulos · 2026 · Applied Sciences · Not summarized yet
Abstract
Reinforcement learning has been increasingly applied to intraday financial decision-making, but the available evidence remains fragmented across heterogeneous tasks, datasets, and evaluation protocols. This systematic literature review examined peer-reviewed studies indexed in ScienceDirect and Scopus from 2018 to 2025. Across the initial search and the subsequent update, 315 bibliographic occurrences were imported; 51 redundant occurrences were removed, leaving 264 unique records for title and abstract screening. Twenty-one reports were retrieved and assessed in full text. Five were excluded during the historical StArt workflow, and 16 were historically accepted. During manuscript revision,…
-
Deep Reinforcement Learning for Financial Trading: Enhanced by Cluster Embedding and Zero-Shot PredictionHaoran Zhang, Xiaofei Li, Tian-Jiao Wan et al. · 2026 · Symmetry · cited 1 time · Not summarized yet
Abstract
Deep reinforcement learning (DRL) plays a pivotal role in decision-making within financial markets. However, DRL models are highly reliant on raw market data and often overlook the impact of future trends on model performance. To address these challenges, we propose a novel framework named Cluster Embedding-Proximal Policy Optimization (CE-PPO) for trading decision-making in financial markets. Specifically, the framework groups feature channels with intrinsic similarities and enhances the original model by leveraging clustering information instead of features from individual channels. Meanwhile, zero-shot prediction for unseen samples is achieved by assigning them to appropriate clusters. Fu…
-
EarnHFT: Efficient Hierarchical Reinforcement Learning for High Frequency TradingMolei Qin, Shuo Sun, Wen-Tao Zhang et al. · 2023 · AAAI Conference on Artificial Intelligence · cited 40 times · Not summarized yet
Abstract
High-frequency trading (HFT) is using computer algorithms to make trading decisions in short time scales (e.g., second-level), which is widely used in the Cryptocurrency (Crypto) market, (e.g., Bitcoin). Reinforcement learning (RL) in financial research has shown stellar performance on many quantitative trading tasks. However, most methods focus on low-frequency trading, e.g., day-level, which cannot be directly applied to HFT because of two challenges. First, RL for HFT involves dealing with extremely long trajectories (e.g., 2.4 million steps per month), which is hard to optimize and evaluate. Second, the dramatic price fluctuations and market trend changes of Crypto make existing algorith…
Go further
How are these papers found?
We compare the title and abstract of this paper with those of every paper summarized on this site, and with the papers Semantic Scholar recommends, those it cites and those citing it (for a much-cited paper, its most recent citations and, from OpenAlex, the most cited ones). Among the prior work and the follow-ups, the papers that use each other in depth, a method it uses and the much-cited follow-ups come a little higher. The levels (very close, close, related) come from the similarity of the texts, not from a judgment of quality; * marks a paper without an abstract, compared by its title. The sentence under a paper is written by an AI from the two abstracts: check it in the papers. A paper marked not relevant by three readers leaves the list.
Updated on 5 November 2026.