What is it about?
we answer key questions about how different modalities like audio and video complement each other. This paper proposes a modality-referenced system. The system works based on state-of-the-art deep learning models, i.e convolutional neural networks. A pre-processing system selects the input data based on one of the modalities called reference or master. The other modality which is called slave simply adjusts or attunes itself with the master in the temporal domain.
Featured Image
Why is it important?
Psychological evidence reports that an average person shows varying degrees of tendency or desire by concurrently using words (verbal), facial and vocal expressions, postures, and gestures (nonverbal). Additionally, that person might transmit different degrees of dominance with these behaviors. Consequently, one interesting question becomes how systematically and coherently we can resolve the general interpretation or impression of an inconsistent message.
Perspectives
Many models have been developed by researchers to address the ER problem by employing different modalities and algorithms. As respects, none studied a modality-referenced system which attunes or adapts the data in an early stage like the pre-processing phase. It is important to highlight that temporal adaptation of the data between two modalities improves the recognition performance. In general, most of the research works apply the experiment on the corporal expression and the emotional information embedded in the speech analysis without considering the time block. However, research evidence from the psychology area suggests studying this assumption further. Here, we use the temporal behavior of the single modalities to investigate whether incongruity between two modalities affects the recognition rate semantically.
Noushin Hajarolasvadi
Dogu Akdeniz Universitesi
Read the Original
This page is a summary of: Deep Emotion Recognition based on Audio-Visual Correlation, IET Computer Vision, June 2020, the Institution of Engineering and Technology (the IET),
DOI: 10.1049/iet-cvi.2020.0013.
You can read the full text:
Contributors
The following have contributed to this page
