# A Time Series is Worth 64 Words: Long-term Forecasting with Transformers

**Authors:** Yuqi Nie, Nam H. Nguyen, Phanwadee Sinthong, Jayant Kalagnanam  
**arXiv:** [2211.14730](https://arxiv.org/abs/2211.14730)  
**Date:** 2022-11-27  
**Summary page:** https://www.summarizepaper.com/en/paper/2211.14730v1/

## Key points

- Authors propose innovative design of Transformer-based models for multivariate time series forecasting and self-supervised representation learning
- Approach includes segmentation of time series into subseries-level patches for input tokens and channel-independence
- Retains local semantic information in embedding
- Reduces computation and memory usage of attention maps quadratically within same look-back window
- Enables model to attend to longer historical data
- Channel-independent patch time series Transformer (PatchTST) shows significant improvements in long-term forecasting accuracy compared to state-of-the-art Transformer-based models
- Model applied to self-supervised pre-training tasks with excellent fine-tuning performance surpassing supervised training on large datasets
- Transferring masked pre-trained representations from one dataset to others leads to state-of-the-art forecasting accuracy
- Code available at https://github.com/yuqinie98/PatchTST

## Summary

In their paper titled "A Time Series is Worth 64 Words: Long-term Forecasting with Transformers," authors Yuqi Nie, Nam H. Nguyen, Phanwadee Sinthong, and Jayant Kalagnanam propose an innovative design of Transformer-based models for multivariate time series forecasting and self-supervised representation learning. The key components of their approach include the segmentation of time series into subseries-level patches that serve as input tokens to the Transformer model and channel-independence. This patching design offers several advantages: it retains local semantic information in the embedding, reduces computation and memory usage of attention maps quadratically within the same look-back window, and enables the model to attend to longer historical data. Their channel-independent patch time series Transformer (PatchTST) demonstrates significant improvements in long-term forecasting accuracy compared to state-of-the-art Transformer-based models. Furthermore, the authors apply their model to self-supervised pre-training tasks and achieve excellent fine-tuning performance surpassing supervised training on large datasets. They also show that transferring masked pre-trained representations from one dataset to others leads to state-of-the-art forecasting accuracy. The code for their model is available at https://github.com/yuqinie98/PatchTST. Overall, this research presents a novel approach that enhances the efficiency and effectiveness of Transformer-based models for time series forecasting and representation learning, showcasing promising results in both tasks.