Multimodal deep learning framework for sentiment analysis from text-image web data

Selvarajah Thuseethan, Sivasubramaniam Janarthan, Sutharshan Rajasegarar, Priya Kumari, John Yearwood

Research output: Chapter in Book/Report/Conference proceedingConference Paper published in Proceedingspeer-review

Abstract

Understanding people's sentiments from data published on the web presents a significant research problem and has a variety of applications, such as learning the context, prediction of election results and opinion about an incident. So far, sentiment analysis from web data has focused primarily on a single modality, such as text or image. However, the readily available multiple modal information, such as image and different forms of texts, as a combination can help to estimate the sentiments more accurately. Further, blindly combining the visual and textual features increases the complexity of the model, which ultimately reduces the sentiment analysis performance as it often fails to capture the correct interrelationships between different modalities. Hence, in this study, a sentiment analysis framework that carefully fuses the salient visual cues and high attention textual cues is proposed, exploiting the interrelationships between multimodal web data. A multimodal deep association learner is stacked to learn the relationships between learned salient visual features and textual features. Further, to automatically learn the discriminative features from the image and text, two streams of unimodal deep feature extractors are proposed to extract the visual and textual features that are most relevant to the sentiments. Finally, the sentiment is estimated using the features that are combined using a late fusion mechanism. The extensive evaluations show that our proposed framework achieved promising results for sentiment analysis using web data, in comparison to existing unimodal approaches and multimodal approaches that blindly combine the visual and textual features.

Original languageEnglish
Title of host publicationProceedings - 2020 IEEE/WIC/ACM International Joint Conference on Web Intelligence and Intelligent Agent Technology, WI-IAT 2020
EditorsJing He, Hemant Purohit, Guangyan Huang, Xiaoying Gao, Ke Deng
PublisherIEEE, Institute of Electrical and Electronics Engineers
Pages267-274
Number of pages8
ISBN (Electronic)9781665419246
DOIs
Publication statusPublished - Dec 2020
Externally publishedYes
Event2020 IEEE/WIC/ACM International Joint Conference on Web Intelligence and Intelligent Agent Technology, WI-IAT 2020 - Virtual, Online
Duration: 14 Dec 202017 Dec 2020

Publication series

NameProceedings - 2020 IEEE/WIC/ACM International Joint Conference on Web Intelligence and Intelligent Agent Technology, WI-IAT 2020

Conference

Conference2020 IEEE/WIC/ACM International Joint Conference on Web Intelligence and Intelligent Agent Technology, WI-IAT 2020
CityVirtual, Online
Period14/12/2017/12/20

Fingerprint

Dive into the research topics of 'Multimodal deep learning framework for sentiment analysis from text-image web data'. Together they form a unique fingerprint.

Cite this