Content-based visual landmark search via multimodal hypergraph learning

Lei Zhu, Jialie Shen, Hai Jin, Ran Zheng, Liang Xie

Research output: Contribution to journalArticlepeer-review


While content-based landmark image search has recently received a lot of attention and became a very active domain, it still remains a challenging problem. Among the various reasons, high diverse visual content is the most significant one. It is common that for the same landmark, images with a wide range of visual appearances can be found from different sources and different landmarks may share very similar sets of images. As a consequence, it is very hard to accurately estimate the similarities between the landmarks purely based on single type of visual feature. Moreover, the relationships between landmark images can be very complex and how to develop an effective modeling scheme to characterize the associations still remains an open question. Motivated by these concerns, we propose multimodal hypergraph (MMHG) to characterize the complex associations between landmark images. In MMHG, images are modeled as independent vertices and hyperedges contain several vertices corresponding to particular views. Multiple hypergraphs are firstly constructed independently based on different visual modalities to describe the hidden high-order relations from different aspects. Then, they are integrated together to involve discriminative information from heterogeneous sources. We also propose a novel content-based visual landmark search system based on MMHG to facilitate effective search. Distinguished from the existing approaches, we design a unified computational module to support query-specific combination weight learning. An extensive experiment study on a large-scale test collection demonstrates the effectiveness of our scheme over state-of-the-art approaches.

Original languageEnglish
Article number15645926
Pages (from-to)2756-2769
Number of pages14
JournalIEEE Transactions on Cybernetics
Issue number12
Early online date6 Jan 2015
Publication statusPublished - 1 Dec 2015
Externally publishedYes


Dive into the research topics of 'Content-based visual landmark search via multimodal hypergraph learning'. Together they form a unique fingerprint.

Cite this