Manual and Non-Manual Sign Language Recognition Framework using Hybrid Deep Learning Techniques

What is it about?

This paper presents towards recognizing affective facial expressions along with hand movement analysis in Spatio-temporal representation, where faces and hands are tracked in multiple frames to extract faces and hands as the regions of interest (ROIs) and further their features are concatenated and classified into seven classes representing disgust, neutral, happy, sad, scared, angry, and surprise expressions.

Featured Image

Photo by Dayne Topkin on Unsplash

Photo by Dayne Topkin on Unsplash

Why is it important?

Combining manual and non-manual features of Sign Language is a complex domain. the current study is a contribution to understanding sign language in a better way. In this current research, three modified architectures are combined together to provide a novel hybrid architecture MM-SLR to recognize non-manual features based on facial expressions along with manual gestures in the spatial-temporal domain representing hand movements in automatic sign language recognition. Experiments are conducted on three public SLT datasets and the results from multiple aspects and multiple levels. The average loss of modified architecture is 0.34 and stable after 2000 iterations. Further qualitative analysis is performed for all three datasets with the milieus of precision, recall, and F1 score, and the model performs promising for all. Overall our model classifies the gesture based on manual and non-manual features using LSTM architecture and for PkSLMNM datasets the training and validation accuracy is 83% and 79% respectively.

Perspectives