# Hybrid Transformer and CNN Attention Network for Stereo Image Super-resolution

**Authors:** Ming Cheng, Haoyu Ma, Qiufang Ma, Xiaopeng Sun, Weiqi Li, Zhenyu Zhang, Xuhan Sheng, Shijie Zhao, Junlin Li, Li Zhang  
**arXiv:** [2305.05177](https://arxiv.org/abs/2305.05177)  
**Date:** 2023-05-09  
**Summary page:** https://www.summarizepaper.com/en/paper/2305.05177v1/

## Key points

- Multi-stage strategies commonly used in image restoration tasks
- Transformer-based methods successful in single-image super-resolution tasks
- No significant advantages of transformers over CNN-based methods in stereo super-resolution tasks due to two main factors:
- Single-image super-resolution transformers cannot effectively utilize complementary stereo information
- Transformers rely on large amounts of training data lacking in common stereo-image super-resolution algorithms
- Authors propose a Hybrid Transformer and CNN Attention Network (HTCAN) for stereo image super-resolution
- HTCAN combines transformer-based network for single-image enhancement with CNN-based network for stereo information fusion
- Multi-patch training strategy and larger window sizes used to activate more input pixels for super resolution
- Other advanced techniques such as data augmentation, data ensemble, and model ensemble employed to reduce overfitting and data bias
- Proposed approach achieved a score of 23.90dB and emerged as the winner in Track 1 of the NTIRE 2023 Stereo Image Super Resolution Challenge
- Importance emphasized of utilizing information from both views in stereo image super resolution
- Feature extraction capability of each view and exchange of stereo information play crucial roles in determining final performance
- Transformers suitable for stereo image super resolution due to larger receptive fields and self attention mechanisms that effectively model long range dependencies
- Transformers have higher memory and computational costs compared to CNNs, which becomes challenging with high resolution images and large number of query tokens
- CNN-based models can afford more parallel exchange modules allowing for more thorough information exchange

## Summary

Multi-stage strategies are commonly used in image restoration tasks and transformer-based methods have been successful in single-image super-resolution tasks. However, they have not shown significant advantages over CNN-based methods in stereo super-resolution tasks due to two main factors: firstly, current single-image super-resolution transformers cannot effectively utilize the complementary stereo information; secondly, transformers rely on large amounts of training data which is lacking in common stereo-image super-resolution algorithms. To address these issues, the authors propose a Hybrid Transformer and CNN Attention Network (HTCAN) for stereo image super-resolution. The HTCAN combines a transformer-based network for single-image enhancement with a CNN based network for stereo information fusion. Additionally, the authors employ a multi-patch training strategy and larger window sizes to activate more input pixels for super resolution. They also revisit other advanced techniques such as data augmentation, data ensemble and model ensemble to reduce overfitting and data bias. The effectiveness of the proposed approach is demonstrated by achieving a score of 23.90dB and emerging as the winner in Track 1 of the NTIRE 2023 Stereo Image Super Resolution Challenge. The authors emphasize the importance of utilizing information from both views in stereo image super resolution as lost information in one view may still exist in the other view and leveraging this extra information can greatly benefit reconstruction process. The feature extraction capability of each view and exchange of stereo information play crucial roles in determining final performance of a stereo image super resolution algorithm. While convolutional neural networks (CNNs) work well on locality priors but suffer from long range dependencies, transformers have larger receptive fields and self attention mechanisms that effectively model long range dependencies making them suitable for stereo image super resolution where careful utilization of information from both views is essential to avoid loss of useful information during process. However transformers come with higher memory and computational costs compared to CNNs which becomes more challenging when dealing with high resolution images and large number of query tokens while CNN based models can afford more parallel exchange modules allowing for more thorough information exchange as demonstrated by NAFSSR - previous state of art method on relatively small datasets.