# Gloss-free Sign Language Translation: Improving from Visual-Language Pretraining

**Authors:** Benjia Zhou, Zhigang Chen, Albert Clapés, Jun Wan, Yanyan Liang, Sergio Escalera, Zhen Lei, Du Zhang  
**arXiv:** [2307.14768](https://arxiv.org/abs/2307.14768)  
**Date:** 2023-07-27  
**Summary page:** https://www.summarizepaper.com/en/paper/2307.14768v1/

## Key points

- Sign Language Translation (SLT) faces challenges in translating visual-gestural language into text due to its complex nature
- Existing methods use gloss sequences as an intermediate representation, dividing the process into sign language recognition (SLR) and sign language translation (SLT)
- Limited availability of gloss-annotated data and constraints of mid-level gloss representation hinder further advancements in SLT development
- Gloss-Free SLT based on Visual-Language Pretraining (GFSLT-VLP) is a groundbreaking approach that enhances SLT without relying on gloss annotations
- GFSLT-VLP framework includes integrating Contrastive Language-Image Pre-training (CLIP) with masked self-supervised learning and constructing an end-to-end architecture with an encoder-decoder-like structure
- GFSLT-VLP has achieved significant improvements in BLEU-4 score on PHOENIX14T dataset (>+5) and CSL-Daily dataset (>+3) compared to state-of-the-art gloss-free SLT methods
- Competitive results were demonstrated on the PHOENIX14T dataset when compared against most gloss-based methods
- Developed by Benjia Zhou, Zhigang Chen, Albert Clapés, Jun Wan, Yanyan Liang, Sergio Escalera, Zhen Lei, and Du Zhang; accepted for presentation at ICCV'23
- Code for implementing GFSLT-VLP is accessible at https://github.com/zhoubenjia/GFSLT-VLP

## Summary

, , , , 

The translation of visual-gestural language into text presents a significant challenge in Sign Language Translation (SLT) due to its complex nature. Existing methods often rely on gloss sequences as an intermediate representation, breaking down the process into two stages: sign language recognition (SLR) and sign language translation (SLT). However, limited availability of gloss-annotated data and constraints of mid-level gloss representation hinder further advancements in SLT development. To overcome these obstacles, a groundbreaking approach known as Gloss-Free SLT based on Visual-Language Pretraining (GFSLT-VLP) has been introduced. This innovative method leverages pre-trained models to enhance SLT by incorporating language-oriented prior knowledge without relying on gloss annotations. The GFSLT-VLP framework consists of two key stages: first, integrating Contrastive Language-Image Pre-training (CLIP) with masked self-supervised learning to bridge the semantic gap between visual and textual representations and restore masked sentences; second, constructing an end-to-end architecture with an encoder-decoder-like structure that inherits parameters from the pre-trained Visual Encoder and Text Decoder established in the initial stage. The seamless integration of these novel designs results in a robust sign language representation and significantly enhances gloss-free sign language translation performance. Notably, GFSLT-VLP has achieved remarkable improvements in terms of BLEU-4 score on both the PHOENIX14T dataset (>+5) and the CSL-Daily dataset (>+3) compared to state-of-the-art gloss-free SLT methods. Furthermore, this approach demonstrates competitive results on the PHOENIX14T dataset when compared against most gloss-based methods. This cutting-edge methodology for enhancing sign language translation capabilities was developed by Benjia Zhou, Zhigang Chen, Albert Clapés, Jun Wan, Yanyan Liang, Sergio Escalera, Zhen Lei, and Du Zhang. Their work has been accepted for presentation at ICCV'23. For those interested in exploring this advancement further or implementing it themselves, the code is readily accessible at https://github.com/zhoubenjia/GFSLT-VLP.