# Exploring the Advantages of Transformers for High-Frequency Trading

**Authors:** Fazl Barez, Paul Bilokon, Arthur Gervais, Nikita Lisitsyn  
**arXiv:** [2302.13850](https://arxiv.org/abs/2302.13850)  
**Date:** 2023-02-20  
**Summary page:** https://www.summarizepaper.com/en/paper/2302.13850v1/

## Key points

- The paper explores the use of deep learning Transformers architectures for high-frequency Bitcoin-USDT log-return forecasting and compares them to traditional LSTM models.
- The authors introduce a hybrid Transformer model called HFformer, which incorporates a Transformer encoder, linear decoder, spiking activations, and quantile loss function without using position encoding.
- Possible high-frequency trading strategies for use with the HFformer model are discussed, including trade sizing, trading signal aggregation, and minimal trading threshold.
- Future lines of research are suggested to improve the LOB snapshot pre-processing pipeline's noise reduction by using autoencoders with automated feature selection.
- More extensive performance assessments of the HFformer on large forecast horizons and using altcoin trading pairs such as ETH-USDT are recommended.
- Implementing the HFformer with other types of Attention modules such as auto-correlation Attention is proposed.
- A more realistic backtesting environment that accounts for the impact of placed orders and emulates other market participants' activity is suggested to assess the performance of the HFformer.
- When testing LSTM and HFformer models in log-returns forecasting from 1 to 30 ticks ahead, they achieved higher R2 scores than other deep learning architectures.
- Both LSTM and HFformer models achieved similar performance for classification tasks.
- Backtested on different trading strategies involving 1-5 trade signals during multiple signals' trades in BTC-USDT LOB data collected over two days and a month after training and validation data were found that using more than one trade signal decreases the number of trades and increases cumulative PnL of a long-short trading strategy.
- The HFformer generates long and short trade signals that result in a more balanced trading strategy than LSTM when complemented with trade sizing to improve cumulative PnL. However these methods may yield different results when trading another cryptocurrency pair or financial asset as machine learning methods are data driven and sometimes less generalizable than traditional statistical methods.

## Summary

This paper explores the use of deep learning Transformers architectures for high-frequency Bitcoin-USDT log-return forecasting and compares them to traditional Long Short-Term Memory (LSTM) models. The authors introduce a hybrid Transformer model, called HFformer, which incorporates a Transformer encoder, linear decoder, spiking activations, and quantile loss function without using position encoding. They also discuss possible high-frequency trading strategies for use with the HFformer model, including trade sizing, trading signal aggregation, and minimal trading threshold. The paper suggests several future lines of research that can be undertaken to improve the LOB snapshot pre-processing pipeline's noise reduction by using autoencoders with automated feature selection. Additionally, they suggest performing more extensive performance assessments of the HFformer on large forecast horizons and using altcoin trading pairs such as ETH-USDT. The authors also recommend implementing the HFformer with other types of Attention modules such as auto-correlation Attention. Moreover, they propose implementing a more realistic backtesting environment that accounts for the impact of placed orders and emulates other market participants' activity to assess the performance of the HFformer. When testing LSTM and HFformer models in log-returns forecasting from 1 to 30 ticks ahead, they achieved higher R2 scores than other deep learning architectures. Moreover, both LSTM and HFformer models achieved similar performance for classification tasks. Finally, when backtested on different trading strategies involving 1-5 trade signals during multiple signals' trades in BTC-USDT LOB data collected over two days and a month after training and validation data were found that using more than one trade signal decreases the number of trades and increases the cumulative PnL of a long-short trading strategy. The HFformer generates long and short trade signals that result in a more balanced trading strategy than LSTM when complemented with trade sizing to improve cumulative PnL. However these methods may yield different results when trading another cryptocurrency pair or financial asset as machine learning methods are data driven and sometimes less generalizable than traditional statistical methods.