# TrOCR: Transformer-based Optical Character Recognition with Pre-trained Models

**Authors:** Minghao Li, Tengchao Lv, Lei Cui, Yijuan Lu, Dinei Florencio, Cha Zhang, Zhoujun Li, Furu Wei  
**arXiv:** [2109.10282](https://arxiv.org/abs/2109.10282)  
**Date:** 2021-09-21  
**Summary page:** https://www.summarizepaper.com/en/paper/2109.10282v1/

## Key points

- Text recognition has been a challenge in document digitalization
- TrOCR is an end-to-end text recognition approach that uses pre-trained image Transformer and text Transformer models
- To build a dataset for TrOCR's pre-training phase, researchers sampled 2 million document pages from publicly available PDF files and synthesized handwritten textline images using TRDG
- TrOCR's pipeline involves extracting visual features from inputted textline images and predicting wordpiece tokens based on context generated before it
- The TrOCR model outperformed current state of the art models in both printed and handwritten text recognition tasks according to experiments conducted using the SROIE dataset and the IAM Handwriting Database.

## Summary

Text recognition has long been a challenge in the field of document digitalization. To address this issue, researchers have proposed an end-to-end text recognition approach called TrOCR which leverages pre-trained image Transformer and text Transformer models based on the Transformer architecture for both image understanding and wordpiece-level text generation. To build a large-scale high-quality dataset for TrOCR's pre-training phase, researchers sampled two million document pages from publicly available PDF files on the internet and converted them into page images to extract pretty-printed textline images. This resulted in a first-stage pre-training dataset containing 684 million textlines. For the second stage of pre-training, they used 5,427 handwritten fonts to synthesize handwritten textline images using an open source data generator called TRDG. The second stage pre-training dataset consisted of 17.9 million handwritten textlines and 3.3 million printed ones. In addition to synthetic data, researchers also collected around 53K receipt images from real world scenarios and recognized their texts using commercial OCR engines. They corrected orientation issues by cropping the receipt images' relevant parts and rotating them if necessary before pruning them down further with TRDG generated printed fonts. TrOCR's pipeline involves extracting visual features from inputted textline images and predicting wordpiece tokens based on context generated before it. During training ground truth tokens are followed by an "[EOS]" token indicating sentence completion; during inference the decoder starts with this token to predict output iteratively while taking newly generated output as its next input. The TrOCR model outperformed current state of the art models in both printed and handwritten text recognition tasks according to experiments conducted using the SROIE (Scanned Receipts OCR and Information Extraction) dataset and the IAM Handwriting Database.