Bundled Object Context for Referring Expressions

doi:10.1109/TMM.2018.2811621

CORC > 计算技术研究所 > 中国科学院计算技术研究所 > 中国科学院计算技术研究所期刊论文 > 英文

	Bundled Object Context for Referring Expressions
	Li, Xiangyang 1,2; Jiang, Shuqiang 1,2
刊名	IEEE TRANSACTIONS ON MULTIMEDIA
	2018-10-01
卷号	20 期号:10 页码:2749-2760
关键词	Bundled object context referring expression LSTM vision-language
ISSN号	1520-9210
DOI	10.1109/TMM.2018.2811621
英文摘要	Referring expressions are natural language descriptions of objects within a given scene. Context is of crucial importance for a referring expression, as the description not only depicts the properties of the object but also involves the relationships of the referred object with other ones. Most of previous work uses either the whole image or one particular contextual object as the context. However, the context of these approaches is holistic and insufficient, as a referring expression often describes relationships of multiple objects in an image. To leverage rich context information from all objects in an image, in this paper, we propose a novel scheme that is composed of a visual context long short-term memory (LSTM) module and a sentence LSTM module to model bundled object context for referring expressions. All contextual objects are arranged with their spatial locations and progressively fed into the visual context LSTM module to acquire and aggregate the context features. Then the concatenation of the learned context features and the features of the referred object are put into the sentence LSTM module to learn the probability of a referring expression. The feedback connections and internal gating mechanism of the LSTM cells enable our model to selectively propagate relevant contextual information through the whole network. Experiments on three benchmark datasets show that our methods can achieve promising results compared to state-of-the-art methods. Moreover, visualization of the internal states of the visual context LSTM cells also shows that our method can automatically select the pertinent context objects.
资助项目	National Natural Science Foundation of China[61532018] ; Beijing Municipal Commission of Science and Technology[D161100001816001] ; Lenovo Outstanding Young Scientists Program ; National Program for Special Support of Eminent Professionals ; National Program for Support of Top-notch Young Professionals
WOS研究方向	Computer Science ; Telecommunications
语种	英语
出版者	IEEE-INST ELECTRICAL ELECTRONICS ENGINEERS INC
WOS记录号	WOS:000444903000017
内容类型	期刊论文
源URL	[http://119.78.100.204/handle/2XEOYT63/4919]
专题	中国科学院计算技术研究所期刊论文_英文
通讯作者	Jiang, Shuqiang
作者单位	1.Univ Chinese Acad Sci, Beijing 100049, Peoples R China 2.Chinese Acad Sci, Inst Comp Technol, Key Lab Intelligent Informat Proc, Beijing 100190, Peoples R China
推荐引用方式 GB/T 7714	Li, Xiangyang,Jiang, Shuqiang. Bundled Object Context for Referring Expressions[J]. IEEE TRANSACTIONS ON MULTIMEDIA,2018,20(10):2749-2760.
APA	Li, Xiangyang,&Jiang, Shuqiang.(2018).Bundled Object Context for Referring Expressions.IEEE TRANSACTIONS ON MULTIMEDIA,20(10),2749-2760.
MLA	Li, Xiangyang,et al."Bundled Object Context for Referring Expressions".IEEE TRANSACTIONS ON MULTIMEDIA 20.10(2018):2749-2760.