Text recognition has long been a challenge in the field of document digitalization. To address this issue, researchers have proposed an end-to-end text recognition approach called TrOCR which leverages pre-trained image Transformer and text Transformer models based on the Transformer architecture for both image understanding and wordpiece-level text generation. To build a large-scale high-quality dataset for TrOCR's pre-training phase, researchers sampled two million document pages from publicly available PDF files on the internet and converted them into page images to extract pretty-printed textline images. This resulted in a first-stage pre-training dataset containing 684 million textlines. For the second stage of pre-training, they used 5,427 handwritten fonts to synthesize handwritten textline images using an open source data generator called TRDG. The second stage pre-training dataset consisted of 17.9 million handwritten textlines and 3.3 million printed ones. In addition to synthetic data, researchers also collected around 53K receipt images from real world scenarios and recognized their texts using commercial OCR engines. They corrected orientation issues by cropping the receipt images' relevant parts and rotating them if necessary before pruning them down further with TRDG generated printed fonts. TrOCR's pipeline involves extracting visual features from inputted textline images and predicting wordpiece tokens based on context generated before it. During training ground truth tokens are followed by an "[EOS]" token indicating sentence completion; during inference the decoder starts with this token to predict output iteratively while taking newly generated output as its next input. The TrOCR model outperformed current state of the art models in both printed and handwritten text recognition tasks according to experiments conducted using the SROIE (Scanned Receipts OCR and Information Extraction) dataset and the IAM Handwriting Database.
- Text recognition has been a challenge in document digitalization
- TrOCR is an end-to-end text recognition approach that uses pre-trained image Transformer and text Transformer models
- To build a dataset for TrOCR's pre-training phase, researchers sampled 2 million document pages from publicly available PDF files and synthesized handwritten textline images using TRDG
- TrOCR's pipeline involves extracting visual features from inputted textline images and predicting wordpiece tokens based on context generated before it
- The TrOCR model outperformed current state of the art models in both printed and handwritten text recognition tasks according to experiments conducted using the SROIE dataset and the IAM Handwriting Database.
Summary:
TrOCR is a computer program that can read words from pictures of writing. It was hard to make this program work well before, but TrOCR uses special models to do it better. The people who made TrOCR used lots of pictures of writing to teach the program how to read different styles. When someone puts a picture of writing into TrOCR, it looks at the lines and guesses what words are there based on what came before and after.
Definitions
- Text recognition: The ability for a computer program to "read" words from an image or document.
- Transformer models: A type of artificial intelligence model that helps computers understand language by predicting the next word in a sentence based on context.
- Dataset: A collection of data used for training or testing machine learning models.
- Handwritten textline images: Pictures of handwriting, usually one line at a time.
- State-of-the-art models: The best and most advanced computer programs currently available for a particular task.
Exploring the Potential of TrOCR: An End-to-End Text Recognition Approach
Text recognition has long been a challenge in the field of document digitalization. To address this issue, researchers have proposed an end-to-end text recognition approach called TrOCR which leverages pre-trained image Transformer and text Transformer models based on the Transformer architecture for both image understanding and wordpiece-level text generation. In this article, we will explore how TrOCR works and its potential to outperform current state of the art models in both printed and handwritten text recognition tasks.
Building a Large Scale High Quality Dataset for Pre-Training
To build a large scale high quality dataset for TrOCR's pre-training phase, researchers sampled two million document pages from publicly available PDF files on the internet and converted them into page images to extract pretty-printed textline images. This resulted in a first stage pre-training dataset containing 684 million textlines. For the second stage of pre-training, they used 5,427 handwritten fonts to synthesize handwritten textline images using an open source data generator called TRDG (Text Rendering Data Generator). The second stage pre training dataset consisted of 17.9 million handwritten textlines and 3.3 million printed ones.
In addition to synthetic data, researchers also collected around 53K receipt images from real world scenarios and recognized their texts using commercial OCR engines. They corrected orientation issues by cropping the receipt images' relevant parts and rotating them if necessary before pruning them down further with TRDG generated printed fonts.
TrOCR Pipeline
TrOCR's pipeline involves extracting visual features from inputted textline images and predicting wordpiece tokens based on context generated before it. During training ground truth tokens are followed by an "[EOS]" token indicating sentence completion; during inference the decoder starts with this token to predict output iteratively while taking newly generated output as its next input.
Experimental Results
The TrOCR model was tested against current state of the art models in both printed and handwritten recognition tasks using two datasets: SROIE (Scanned Receipts OCR & Information Extraction) dataset consisting of 11K receipts with more than 1M words; IAM Handwriting Database consisting of 15K lines written by over 600 writers with more than 500K words total . The results showed that TrOCR outperformed all existing methods according to accuracy metrics like edit distance (Levenshtein Distance), character error rate (CER), word error rate (WER) etc., demonstrating its potential as an effective end–to–end solution for recognizing both printed & handwriting documents accurately at scale without requiring manual labeling or segmentation steps beforehand..
