Boosting Emotion Recognition in Context using Non-target Subject Information

Recognizing emotions in context from an image has become an emerging topic in the recent past due to its high practical demand in various domains. Performance of the existing works is limited as they predominantly utilized the entire image and the primary human subject as major cues to recognize the emotions from images. However, in addition to the primary human subject, other human subjects in a scene also play a vital role in determining the image's overall emotional state. In this work, a novel visual feature type is introduced based on the features extracted from other human subjects in order to enhance emotion recognition performance. A novel deep learning based hybrid framework is also proposed to effectively integrate the proposed feature with the visual cues from the entire image and primary human subject. The extensive experiments carried out on a subset of the Emotic dataset reveal that the newly introduced visual feature type contributes to the overall emotional state of the scene. Furthermore, the proposed framework achieved 45.11% average accuracy for discrete emotion categories, which is a significant improvement in the emotion recognition performance compared to existing techniques that use emotion information from the entire image and primary human subject.