# Are Transformers Effective for Time Series Forecasting?

**Authors:** Ailing Zeng, Muxi Chen, Lei Zhang, Qiang Xu  
**arXiv:** [2205.13504](https://arxiv.org/abs/2205.13504)  
**Date:** 2022-05-26  
**Summary page:** https://www.summarizepaper.com/en/paper/2205.13504v1/

## Key points

- Transformer-based solutions have gained popularity for time series forecasting (TSF) in recent years
- Transformers rely on self-attention mechanisms to extract semantic correlations between paired elements in a long sequence, which is permutation-invariant and anti-ordering to some extent
- However, in time series modeling, we are interested in extracting temporal relations among an ordering set of continuous points
- A study was conducted comparing Transformer-based models with an embarrassingly simple architecture named DLinear that conducts direct multi-step (DMS) forecasting
- DLinear outperformed existing complex Transformer-based models in most cases by a large margin
- The study also compared the performance of FEDformer and Autoformer against DLinear under a longer look-back window size of 336.
- The results showed that DLinear largely surpassed these state-of-the-art Transformer-based methods on datasets such as Exchange rate, Traffic, Electricity, Weather, ETTm1 and ETTh1.
- The study concludes that the relatively higher long-term forecasting accuracy of Transformer-based TSF solutions shown in existing works has little to do with the temporal relation extraction capabilities of the Transformer architecture. Instead, it is mainly due to the nonautoregressive DMS forecasting strategy used in them.
- This finding raises questions about the validity of using Transformers for other time series analysis tasks such as anomaly detection.
- Overall, this study highlights the importance of questioning established assumptions regarding model suitability for specific tasks and encourages researchers to explore simpler architectures before resorting to more complex ones.

## Summary

In recent years, there has been a surge of interest in using Transformer-based solutions for time series forecasting (TSF), particularly for the challenging long-term TSF problem. The Transformer architecture relies on self-attention mechanisms to extract semantic correlations between paired elements in a long sequence, which is permutation-invariant and anti-ordering to some extent. However, in time series modeling, we are interested in extracting temporal relations among an ordering set of continuous points. This raises the question of whether Transformer-based techniques are suitable for long-term time series forecasting despite their demonstrated performance improvements. To investigate this issue further, a study was conducted comparing Transformer-based models with an embarrassingly simple architecture named DLinear that conducts direct multi-step (DMS) forecasting. DLinear decomposes the time series into a trend and remainder series and employs two one-layer linear networks to model these two series for the forecasting task. Surprisingly, DLinear outperformed existing complex Transformer-based models in most cases by a large margin. The study also compared the performance of FEDformer and Autoformer against DLinear under a longer look-back window size of 336. The results showed that DLinear largely surpassed these state-of-the-art Transformer-based methods on datasets such as Exchange rate, Traffic, Electricity, Weather, ETTm1 and ETTh1. The study concludes that the relatively higher long-term forecasting accuracy of Transformer-based TSF solutions shown in existing works has little to do with the temporal relation extraction capabilities of the Transformer architecture. Instead, it is mainly due to the nonautoregressive DMS forecasting strategy used in them. This finding raises questions about the validity of using Transformers for other time series analysis tasks such as anomaly detection. Overall, this study highlights the importance of questioning established assumptions regarding model suitability for specific tasks and encourages researchers to explore simpler architectures before resorting to more complex ones. Section 6 concludes this paper by discussing potential future research directions.