The Transformer is a self-attention-based neural network architecture that was originally developed for natural language processing (NLP) tasks. However, recent advancements have shown that pure transformer-based models can also be applied to solve computer vision problems. These visual transformers typically treat an image as a sequence of patches, but they often overlook the intrinsic structural information within each patch. To address this limitation, the authors of this paper propose a novel model called Transformer-iN-Transformer (TNT), which aims to capture both patch-level and pixel-level representations in images. The TNT model consists of TNT blocks, each containing an outer transformer block for processing patch embeddings and an inner transformer block for extracting local features from pixel embeddings. In more detail, the pixel-level features are projected onto the space of patch embeddings using a linear transformation layer and then added into the corresponding patches. By stacking multiple TNT blocks together, the authors construct the TNT model for image recognition tasks. The proposed TNT architecture is evaluated on the ImageNet benchmark and various downstream tasks, demonstrating its superiority and efficiency compared to other existing models. For instance, the TNT model achieves an impressive top-1 accuracy of 81.3% on ImageNet, which is 1.5% higher than that of DeiT (a similar model) with comparable computational cost. Overall, this paper introduces a new approach to incorporate both patch-level and pixel-level representations in image analysis through the Transformer-iN-Transformer (TNT) model. The experimental results highlight its effectiveness in achieving state-of-the-art performance on image recognition tasks such as ImageNet classification. The code for implementing the proposed TNT architecture will be made available at https://github.com/huawei-noah/noah-research/tree/master/TNT.
Summary- The Transformer is a special computer program that helps with understanding and working with words.
- People have found out that the Transformer can also be used to understand and work with pictures.
- However, when using the Transformer for pictures, it sometimes forgets important details about each part of the picture.
- A new model called Transformer-iN-Transformer (TNT) has been created to fix this problem. It can understand both big parts and small details of a picture.
- The TNT model is very good at recognizing things in pictures and has better accuracy than other similar models.
Definitions
- Neural network: A computer program that tries to imitate how our brains work by learning from examples.
- Architecture: The way a computer program is built or organized.
- NLP tasks: Computer programs that help with understanding and working with words.
- Computer vision problems: Computer programs that help with understanding and working with pictures.
- Patch: A small part or piece of something bigger. In this case, it refers to small parts of a picture.
Introducing the Transformer-iN-Transformer (TNT) Model for Image Recognition
The Transformer is a self-attention based neural network architecture originally developed for natural language processing (NLP) tasks. However, recent advancements have shown that pure transformer-based models can also be applied to solve computer vision problems. To address the limitation of existing visual transformers which overlook intrinsic structural information within each patch, researchers from Huawei Noah’s Ark Lab propose a novel model called Transformer-iN-Transformer (TNT). This paper introduces TNT and evaluates its effectiveness in achieving state-of-the-art performance on image recognition tasks such as ImageNet classification.
Overview of the TNT Model
The TNT model consists of multiple TNT blocks, each containing an outer transformer block for processing patch embeddings and an inner transformer block for extracting local features from pixel embeddings. In more detail, the pixel level features are projected onto the space of patch embeddings using a linear transformation layer and then added into corresponding patches. By stacking multiple TNT blocks together, the authors construct their proposed TNT model for image recognition tasks.
Experimental Results
The proposed TNT architecture was evaluated on the ImageNet benchmark and various downstream tasks with impressive results compared to other existing models. For instance, it achieved top 1 accuracy of 81.3% on ImageNet which is 1.5% higher than DeiT (a similar model) with comparable computational cost. The code for implementing this architecture will be made available at https://github.com/huawei-noah/noah-research/tree/master/TNT .
Conclusion
This paper introduces a new approach to incorporate both patch level and pixel level representations in image analysis through the Transformer iN Transformer (TNT) model which has been proven effective in achieving state of art performance on image recognition tasks such as ImageNet classification with improved accuracy over existing models while maintaining comparable computational cost efficiency.
Transformer is a type of self-attention-based neural networks originally applied for NLP tasks. Recently, pure transformer-based models are proposed to solve computer vision problems. These visual transformers usually view an image as a sequence of patches while they ignore the intrinsic structure information inside each patch. In this paper, we propose a novel Transformer-iN-Transformer (TNT) model for modeling both patch-level and pixel-level representation. In each TNT block, an outer transformer block is utilized to process patch embeddings, and an inner transformer block extracts local features from pixel embeddings. The pixel-level feature is projected to the space of patch embedding by a linear transformation layer and then added into the patch. By stacking the TNT blocks, we build the TNT model for image recognition. Experiments on ImageNet benchmark and downstream tasks demonstrate the superiority and efficiency of the proposed TNT architecture. For example, our TNT achieves $81.3\%$ top-1 accuracy on ImageNet which is $1.5\%$ higher than that of DeiT with similar computational cost. The code will be available at https://github.com/huawei-noah/noah-research/tree/master/TNT.
