Visual-textual joint relevance learning for tag-based social image search

Yue Gao, Meng Wang, Zheng Jun Zha, Jialie Shen, Xuelong Li, Xindong Wu

Research output: Contribution to journalArticleResearchpeer-review

Abstract

Due to the popularity of social media websites, extensive research efforts have been dedicated to tag-based social image search. Both visual information and tags have been investigated in the research field. However, most existing methods use tags and visual characteristics either separately or sequentially in order to estimate the relevance of images. In this paper, we propose an approach that simultaneously utilizes both visual and textual information to estimate the relevance of user tagged images. The relevance estimation is determined with a hypergraph learning approach. In this method, a social image hypergraph is constructed, where vertices represent images and hyperedges represent visual or textual terms. Learning is achieved with use of a set of pseudo-positive images, where the weights of hyperedges are updated throughout the learning process. In this way, the impact of different tags and visual words can be automatically modulated. Comparative results of the experiments conducted on a dataset including 370+images are presented, which demonstrate the effectiveness of the proposed approach.

Original languageEnglish
Article number6212356
Pages (from-to)363-376
Number of pages14
JournalIEEE Transactions on Image Processing
Volume22
Issue number1
DOIs
Publication statusPublished - 1 Jan 2013
Externally publishedYes

Fingerprint

Websites
Experiments

Cite this

Gao, Yue ; Wang, Meng ; Zha, Zheng Jun ; Shen, Jialie ; Li, Xuelong ; Wu, Xindong. / Visual-textual joint relevance learning for tag-based social image search. In: IEEE Transactions on Image Processing. 2013 ; Vol. 22, No. 1. pp. 363-376.
@article{a09c986c981c4889af51ee20a90c188d,
title = "Visual-textual joint relevance learning for tag-based social image search",
abstract = "Due to the popularity of social media websites, extensive research efforts have been dedicated to tag-based social image search. Both visual information and tags have been investigated in the research field. However, most existing methods use tags and visual characteristics either separately or sequentially in order to estimate the relevance of images. In this paper, we propose an approach that simultaneously utilizes both visual and textual information to estimate the relevance of user tagged images. The relevance estimation is determined with a hypergraph learning approach. In this method, a social image hypergraph is constructed, where vertices represent images and hyperedges represent visual or textual terms. Learning is achieved with use of a set of pseudo-positive images, where the weights of hyperedges are updated throughout the learning process. In this way, the impact of different tags and visual words can be automatically modulated. Comparative results of the experiments conducted on a dataset including 370+images are presented, which demonstrate the effectiveness of the proposed approach.",
keywords = "Hypergraph learning, social image search, tag, visual-textual",
author = "Yue Gao and Meng Wang and Zha, {Zheng Jun} and Jialie Shen and Xuelong Li and Xindong Wu",
year = "2013",
month = "1",
day = "1",
doi = "10.1109/TIP.2012.2202676",
language = "English",
volume = "22",
pages = "363--376",
journal = "IEEE Transactions on Image Processing",
issn = "1057-7149",
publisher = "IEEE, Institute of Electrical and Electronics Engineers",
number = "1",

}

Visual-textual joint relevance learning for tag-based social image search. / Gao, Yue; Wang, Meng; Zha, Zheng Jun; Shen, Jialie; Li, Xuelong; Wu, Xindong.

In: IEEE Transactions on Image Processing, Vol. 22, No. 1, 6212356, 01.01.2013, p. 363-376.

Research output: Contribution to journalArticleResearchpeer-review

TY - JOUR

T1 - Visual-textual joint relevance learning for tag-based social image search

AU - Gao, Yue

AU - Wang, Meng

AU - Zha, Zheng Jun

AU - Shen, Jialie

AU - Li, Xuelong

AU - Wu, Xindong

PY - 2013/1/1

Y1 - 2013/1/1

N2 - Due to the popularity of social media websites, extensive research efforts have been dedicated to tag-based social image search. Both visual information and tags have been investigated in the research field. However, most existing methods use tags and visual characteristics either separately or sequentially in order to estimate the relevance of images. In this paper, we propose an approach that simultaneously utilizes both visual and textual information to estimate the relevance of user tagged images. The relevance estimation is determined with a hypergraph learning approach. In this method, a social image hypergraph is constructed, where vertices represent images and hyperedges represent visual or textual terms. Learning is achieved with use of a set of pseudo-positive images, where the weights of hyperedges are updated throughout the learning process. In this way, the impact of different tags and visual words can be automatically modulated. Comparative results of the experiments conducted on a dataset including 370+images are presented, which demonstrate the effectiveness of the proposed approach.

AB - Due to the popularity of social media websites, extensive research efforts have been dedicated to tag-based social image search. Both visual information and tags have been investigated in the research field. However, most existing methods use tags and visual characteristics either separately or sequentially in order to estimate the relevance of images. In this paper, we propose an approach that simultaneously utilizes both visual and textual information to estimate the relevance of user tagged images. The relevance estimation is determined with a hypergraph learning approach. In this method, a social image hypergraph is constructed, where vertices represent images and hyperedges represent visual or textual terms. Learning is achieved with use of a set of pseudo-positive images, where the weights of hyperedges are updated throughout the learning process. In this way, the impact of different tags and visual words can be automatically modulated. Comparative results of the experiments conducted on a dataset including 370+images are presented, which demonstrate the effectiveness of the proposed approach.

KW - Hypergraph learning

KW - social image search

KW - tag

KW - visual-textual

UR - http://www.scopus.com/inward/record.url?scp=84871666061&partnerID=8YFLogxK

U2 - 10.1109/TIP.2012.2202676

DO - 10.1109/TIP.2012.2202676

M3 - Article

VL - 22

SP - 363

EP - 376

JO - IEEE Transactions on Image Processing

JF - IEEE Transactions on Image Processing

SN - 1057-7149

IS - 1

M1 - 6212356

ER -