What is it about?
This paper presents towards recognizing affective facial expressions along with hand movement analysis in Spatio-temporal representation, where faces and hands are tracked in multiple frames to extract faces and hands as the regions of interest (ROIs) and further their features are concatenated and classified into seven classes representing disgust, neutral, happy, sad, scared, angry, and surprise expressions.
Photo by Dayne Topkin on Unsplash
Why is it important?
Combining manual and non-manual features of Sign Language is a complex domain. the current study is a contribution to understanding sign language in a better way. In this current research, three modified architectures are combined together to provide a novel hybrid architecture MM-SLR to recognize non-manual features based on facial expressions along with manual gestures in the spatial-temporal domain representing hand movements in automatic sign language recognition. Experiments are conducted on three public SLT datasets and the results from multiple aspects and multiple levels. The average loss of modified architecture is 0.34 and stable after 2000 iterations. Further qualitative analysis is performed for all three datasets with the milieus of precision, recall, and F1 score, and the model performs promising for all. Overall our model classifies the gesture based on manual and non-manual features using LSTM architecture and for PkSLMNM datasets the training and validation accuracy is 83% and 79% respectively.
Read the Original
This page is a summary of: Manual and non-manual sign language recognition framework using hybrid deep learning techniques, Journal of Intelligent & Fuzzy Systems, August 2023, IOS Press,
You can read the full text:
Sign language is a non-verbal form of communication used by people with impaired hearing and speech. They also use facial actions to provide sign language prosody, similar to intonation in spoken languages. Sign Language Recognition (SLR) using hand signs is a typical way, however, face expression and body language play an important role in communication, which has not been analyzed to its fullest potential. In this paper, we present a dataset that comprises manual (hand signs) and non-manual (facial expressions and body movements) gestures of Pakistan Sign Language (PSL). It contains videos of 7 basic affective expressions performed by 100 healthy individuals, presented in an easily accessible format of .MP4 that can be used to train and test systems to make robust models for real-time applications using videos. Current data can also help with facial feature detection, classification of subjects by gender and age, or provide insights into any individual’s interest and emotional state.
The following have contributed to this page