Integrating Multi-Hub Driven Attention Mechanism and Big Data Analytics for Virtual Representation of Visual Scenes

Yang Yao, Bo Gu, Mamoun Alazab, Neeraj Kumar, Yu Han

Research output: Contribution to journalArticlepeer-review

Abstract

Digital twin is the innovation backbone of the smart manufacturing by delivering virtual representation of the real world. Aiming at constructing virtual representations of visual scenes, scene graph generation is a digital twin task that not only models objects but also infers their relationships. Existing works usually learn coarse global context when predicting relationships leading to excessive redundant information being considered. In this paper, we first classify objects into different subgroups according to the degree of correlations with several hub objects. Then, we propose a multi-hub driven attention network (MHDANet) based on deep learning that drives the information to pass within the subgroups and forces objects to attend more to related objects. Consequently, MHDANet learns compact relation-aware features of visual scenes, and predicts accurate and diverse relationships. Experimental results show that MHDANet achieves superb performance on scene graph generation on real-world datasets, especially alleviates the imbalance of predicted relationship categories.

Original languageEnglish
Pages (from-to)1435-1444
Number of pages10
JournalIEEE Transactions on Industrial Informatics
Volume18
Issue number2
DOIs
Publication statusPublished - Feb 2022

Fingerprint

Dive into the research topics of 'Integrating Multi-Hub Driven Attention Mechanism and Big Data Analytics for Virtual Representation of Visual Scenes'. Together they form a unique fingerprint.

Cite this