# iTransformer: Inverted Transformers Are Effective for Time Series Forecasting

**Authors:** Yong Liu, Tengge Hu, Haoran Zhang, Haixu Wu, Shiyu Wang, Lintao Ma, Mingsheng Long  
**arXiv:** [2310.06625](https://arxiv.org/abs/2310.06625)  
**Date:** 2023-10-10  
**Summary page:** https://www.summarizepaper.com/en/paper/2310.06625v1/

## Key points

- Recent surge in linear forecasting models raises doubts about the need for architectural modifications to Transformer-based forecasters
- Transformers face challenges when forecasting series with larger lookback windows due to performance degradation and computation explosion
- Unified embedding of temporal tokens in Transformers combines multiple variates with potentially misaligned timestamps and distinct physical measurements, leading to difficulties in learning variate-centric representations and producing meaningful attention maps
- iTransformer is a novel approach that repurposes components of the Transformer architecture without any adaptation
- iTransformer inverts the duties of the attention mechanism and the feed-forward network, embedding individual time points into variate tokens for capturing multivariate correlations using the attention mechanism, while applying the feed-forward network to each variate token for learning nonlinear representations
- iTransformer achieves consistent state-of-the-art performance on several real-world datasets, improving performance, generalization ability across different variates, and utilization of arbitrary lookback windows
- iTransformer emerges as a promising alternative as the fundamental backbone for time series forecasting
- Rethinking and repurposing existing components can lead to significant improvements in forecasting models based on Transformers
- The proposed iTransformer model offers enhanced capabilities for capturing complex dependencies in time series data and demonstrates its effectiveness through empirical evaluation on various datasets.

## Summary

The recent surge in linear forecasting models has raised doubts about the need for architectural modifications to Transformer-based forecasters. These forecasters utilize Transformers to capture global dependencies among temporal tokens in time series data, where each token consists of multiple variates from the same timestamp. However, Transformers face challenges when forecasting series with larger lookback windows due to performance degradation and computation explosion. Additionally, the unified embedding of each temporal token combines multiple variates with potentially misaligned timestamps and distinct physical measurements, leading to difficulties in learning variate-centric representations and producing meaningful attention maps. In this study, the authors reflect on the roles of different components in the Transformer architecture and propose a novel approach called iTransformer that repurposes these components without any adaptation. The key idea is to invert the duties of the attention mechanism and the feed-forward network. Specifically, individual time points in a series are embedded into variate tokens, which are then used by the attention mechanism to capture multivariate correlations. On the other hand, the feed-forward network is applied to each variate token to learn nonlinear representations. The iTransformer model achieves consistent state-of-the-art performance on several real-world datasets. It enhances the Transformer family by improving its performance, generalization ability across different variates, and utilization of arbitrary lookback windows. As a result, iTransformer emerges as a promising alternative as the fundamental backbone for time series forecasting. Overall, this research highlights how rethinking and repurposing existing components can lead to significant improvements in forecasting models based on Transformers. The proposed iTransformer model offers enhanced capabilities for capturing complex dependencies in time series data and demonstrates its effectiveness through empirical evaluation on various datasets.