# Two-Stream Network for Sign Language Recognition and Translation

**Authors:** Yutong Chen, Ronglai Zuo, Fangyun Wei, Yu Wu, Shujie Liu, Brian Mak  
**arXiv:** [2211.01367](https://arxiv.org/abs/2211.01367)  
**Date:** 2022-11-02  
**Summary page:** https://www.summarizepaper.com/en/paper/2211.01367v2/

## Key points

- Authors Yutong Chen, Ronglai Zuo, Fangyun Wei, Yu Wu, Shujie Liu, and Brian Mak introduce a novel approach to sign language recognition and translation
- Proposed model named TwoStream-SLR utilizes two separate streams to encode raw videos and keypoint sequences for enhanced sign language understanding
- Techniques such as bidirectional lateral connections, sign pyramid network with auxiliary supervision, and frame-level self-distillation are explored to facilitate interaction between the streams
- Model demonstrates competence in sign language recognition tasks
- Extended model called TwoStream-SLT adds an extra translation network component for accurate translation between sign languages
- Experimental results showcase state-of-the-art performance on SLR and SLT tasks across multiple datasets including Phoenix-2014, Phoenix-2014T, and CSL-Daily
- Research accepted by a top conference in machine learning and artificial intelligence; code and models publicly available at https://github.com/FangyunWei/SLRT

## Summary

In their paper titled "Two-Stream Network for Sign Language Recognition and Translation," authors Yutong Chen, Ronglai Zuo, Fangyun Wei, Yu Wu, Shujie Liu, and Brian Mak introduce a novel approach to sign language recognition and translation. is the process of interpreting visual languages that utilize manual articulations and non-manual elements to convey information. Existing methods often encode RGB videos directly into hidden representations for this purpose. However, RGB videos contain visual redundancy that can cause the encoder to overlook crucial information essential for sign language understanding. To address this issue and enhance the incorporation of domain knowledge such as handshape and body movement, the authors propose a  with two separate streams. These streams model both the raw videos and keypoint sequences generated by an off-the-shelf keypoint estimator. To facilitate interaction between the two streams, various techniques are explored including bidirectional lateral connections, a sign pyramid network with auxiliary supervision, and frame-level self-distillation. The resulting model, named TwoStream-SLR, demonstrates competence in . Building upon this success, the authors extend TwoStream-SLR to create a  model called TwoStream-SLT by adding an extra translation network component. Experimental results showcase state-of-the-art performance on SLR and SLT tasks across multiple datasets including Phoenix-2014, Phoenix-2014T, and CSL-Daily. This innovative approach not only improves sign language recognition but also enables accurate translation between sign languages. The research conducted by these authors has been accepted by , one of the top conferences in machine learning and artificial intelligence. Their code and models are publicly available at https://github.com/FangyunWei/SLRT.