# Sign Language Transformers: Joint End-to-end Sign Language Recognition and Translation

**Authors:** Necati Cihan Camgoz, Oscar Koller, Simon Hadfield, Richard Bowden  
**arXiv:** [2003.13830](https://arxiv.org/abs/2003.13830)  
**Date:** 2020-03-30  
**Summary page:** https://www.summarizepaper.com/en/paper/2003.13830v1/

## Key points

- The paper addresses the problem of sign language translation
- Previous research has shown that using a mid-level sign gloss representation improves translation performance
- The authors propose a novel transformer-based architecture for simultaneous sign language recognition and translation
- They utilize a Connectionist Temporal Classification (CTC) loss to bind the recognition and translation problems into a unified architecture
- The joint approach does not rely on ground-truth timing information
- The approach is evaluated on the RWTH-PHOENIX-Weather-2014T dataset, achieving state-of-the-art results for both sign language recognition and translation tasks
- Sign Language Transformers outperform existing models for translating sign video to spoken language and gloss to spoken language translations
- Translation networks achieve more than double the performance with a BLEU-4 Score of 21.80 compared to 9.58 in some cases
- New baseline translation results using transformer networks for various text-to-text sign language translation tasks are presented
- The proposed approach demonstrates significant improvements in both sign language recognition and translation tasks, with potential applications in bridging communication gaps between deaf individuals who use sign language and those who do not understand it.

## Summary

The paper titled "Sign Language Transformers: Joint End-to-end Sign Language Recognition and Translation" by Necati Cihan Camgoz, Oscar Koller, Simon Hadfield, and Richard Bowden addresses the problem of sign language translation. Previous research has shown that using a mid-level sign gloss representation significantly improves translation performance. To overcome this limitation, the authors propose a novel transformer-based architecture that simultaneously learns Continuous Sign Language Recognition and Translation in an end-to-end manner. This is achieved by utilizing a Connectionist Temporal Classification (CTC) loss to bind the recognition and translation problems into a unified architecture. Importantly, this joint approach does not rely on ground-truth timing information. The authors evaluate their approach on the challenging RWTH-PHOENIX-Weather-2014T (PHOENIX14T) dataset and report state-of-the-art results for both sign language recognition and translation tasks. Their Sign Language Transformers outperform existing models for translating sign video to spoken language as well as gloss to spoken language translations. In some cases, their translation networks achieve more than double the performance with a BLEU-4 Score of 21.80 compared to 9.58. Additionally, they present new baseline translation results using transformer networks for various text-to-text sign language translation tasks. Overall, their proposed approach demonstrates significant improvements in both sign language recognition and translation tasks, showcasing its potential for real world applications in bridging communication gaps between deaf individuals who use sign language and those who do not understand it.