# Transformer-Based UNet with Multi-Headed Cross-Attention Skip Connections to Eliminate Artifacts in Scanned Documents

**Authors:** David Kreuzer, Michael Munz  
**arXiv:** [2306.02815](https://arxiv.org/abs/2306.02815)  
**Date:** 2023-06-05  
**Summary page:** https://www.summarizepaper.com/en/paper/2306.02815v1/

## Key points

- Authors David Kreuzer and Michael Munz address the need for high-quality text extraction in document analysis tasks such as Document Classification and Named Entity Recognition.
- Challenges posed by poor scan quality and resulting artifacts can introduce errors in the Optical Character Recognition (OCR) process.
- The proposed approach combines a modified UNet structure with a Swin Transformer backbone to eliminate typical artifacts found in scanned documents.
- Multi-headed cross-attention skip connections enhance the model's ability to address compression errors, pixelation, and random noise.
- Experimental results show a significant improvement in text extraction quality, achieving a reduced error rate of up to 53.9% on synthetic data.
- The pretrained base-model is adaptable to new artifacts, demonstrating flexibility and scalability.
- Cross-attention skip connections allow integration of textual information from the encoder or input commands to more effectively control the model's output.

## Summary

, , , , 

In their paper titled "Transformer-Based UNet with Multi-Headed Cross-Attention Skip Connections to Eliminate Artifacts in Scanned Documents," authors David Kreuzer and Michael Munz address the crucial need for high-quality text extraction in document analysis tasks such as Document Classification and Named Entity Recognition. They highlight the challenges posed by poor scan quality and resulting artifacts that can introduce errors in the Optical Character Recognition (OCR) process. Convolutional Neural Networks have shown promise in background removal tasks, but they often struggle to correct artifacts like pixelation or compression errors. To tackle these issues, the authors propose a novel approach that combines a modified UNet structure with a Swin Transformer backbone specifically designed to eliminate typical artifacts found in scanned documents. By incorporating multi-headed cross-attention skip connections, the model can selectively learn features at different levels of abstraction, enhancing its ability to address compression errors, pixelation, and random noise. Through experimentation, the authors demonstrate a significant improvement in text extraction quality, achieving a reduced error rate of up to 53.9% on synthetic data. They also highlight the adaptability of their pretrained base-model to new artifacts, showcasing the model's flexibility and scalability. Additionally, by leveraging cross-attention skip connections, textual information extracted from the encoder or input commands can be integrated to more effectively control the model's output. Overall, this innovative approach offers a promising solution for enhancing text extraction accuracy in scanned documents by effectively addressing common artifacts through a combination of UNet architecture and Transformer backbone with multi-headed cross-attention skip connections. The authors provide compelling evidence of its efficacy through experimental results and demonstrate its potential for practical applications through an illustrative example.