# Multi-View Spatial-Temporal Network for Continuous Sign Language Recognition

**Authors:** Ronghui Li, Lu Meng  
**arXiv:** [2204.08747](https://arxiv.org/abs/2204.08747)  
**Date:** 2022-04-19  
**Summary page:** https://www.summarizepaper.com/en/paper/2204.08747v1/

## Key points

- Sign language is a visual language used by speaking and hearing-impaired individuals.
- Understanding and mastering sign language can be challenging due to its complexity.
- Sign language recognition algorithms help bridge the communication gap.
- Traditional methods struggle to capture spatial-temporal features and long-term dependencies of sign language.
- The Multi-View Spatial-Temporal Network (MSTN) is introduced as a novel approach for continuous sign language recognition.
- MSTN comprises three components: MSTN, Sign Language Encoder Network based on Transformer, and CTC Decoder Network.
- MSTN extracts spatial-temporal features from RGB and skeleton data for comprehensive understanding of sign language expressions.
- The Sign Language Encoder Network based on Transformer learns long-term dependencies in sign language sequences.
- The CTC Decoder Network predicts the complete meaning of continuous sign language by decoding the output from previous components.
- The proposed algorithm achieves excellent performance on SLR-100 and RWTH-PHOENIX Weather datasets.

## Summary

Sign language is a beautiful visual language that serves as the primary means of communication for speaking and hearing-impaired individuals. However, the complexity of sign language expressions poses challenges for the general public in understanding and mastering it. To bridge this communication gap, sign language recognition algorithms play a crucial role. Traditional continuous sign language recognition methods rely on Convolutional Neural Network (CNN) and Long Short-Term Memory Network (LSTM) to learn spatial and temporal features separately. However, these methods struggle to capture the intricate spatial-temporal features of sign language and fail to effectively learn long-term dependencies. To address these limitations, this paper introduces a novel approach called the Multi-View Spatial-Temporal Network (MSTN) for continuous sign language recognition. The network comprises three components: 

1. Multi-View Spatial-Temporal Feature Extractor Network (MSTN): This component directly extracts spatial-temporal features from RGB and skeleton data, enabling a comprehensive understanding of sign language expressions. 2. Sign Language Encoder Network based on Transformer: By leveraging Transformer architecture, this network effectively learns long-term dependencies in sign language sequences, enhancing the accuracy of recognition. 3. Connectionist Temporal Classification (CTC) Decoder Network: This network predicts the complete meaning of continuous sign language by decoding the output from the previous components. The proposed algorithm is evaluated using two publicly available sign language datasets: SLR-100 and PHOENIX-Weather 2014T (RWTH). The results demonstrate excellent performance on both datasets, with a word error rate of 1.9% on SLR-100 and 22.8% on RWTH-PHOENIX Weather dataset.