Reducing Vision-Answer Biases for Multiple-Choice VQA

doi:10.1109/TIP.2023.3302162

CORC > 自动化研究所 > 中国科学院自动化研究所 > 多模态人工智能系统全国重点实验室

	Reducing Vision-Answer Biases for Multiple-Choice VQA
	Zhang, Xi 1,5; Zhang, Feifei 3,4; Xu, Changsheng 1,2,5
刊名	IEEE TRANSACTIONS ON IMAGE PROCESSING
	2023
卷号	32 页码:4621-4634
关键词	Multiple-choice VQA vision-answer bias causal intervention counterfactual interaction learning
ISSN号	1057-7149
DOI	10.1109/TIP.2023.3302162
通讯作者	Xu, Changsheng(csxu@nlpr.ia.ac.cn)
英文摘要	Multiple-choice visual question answering (VQA) is a challenging task due to the requirement of thorough multimodal understanding and complicated inter-modality relationship reasoning. To solve the challenge, previous approaches usually resort to different multimodal interaction modules. Despite their effectiveness, we find that existing methods may exploit a new discovered bias (vision-answer bias) to make answer prediction, leading to suboptimal VQA performances and poor generalization. To solve the issues, we propose a Causality-based Multimodal Interaction Enhancement (CMIE) method, which is model-agnostic and can be seamlessly incorporated into a wide range of VQA approaches in a plug-and-play manner. Specifically, our CMIE contains two key components: a causal intervention module and a counterfactual interaction learning module. The former devotes to removing the spurious correlation between the visual content and the answer caused by the vision-answer bias, and the latter helps capture discriminative inter-modality relationships by directly supervising multimodal interaction training via an interactive loss. Extensive experimental results on three public benchmarks and one reorganized dataset show that the proposed method can significantly improve seven representative VQA models, demonstrating the effectiveness and generalizability of the CMIE.
资助项目	National Key Research and Development Plan of China[2021ZD0112200] ; National Natural Science Foundation of China[62036012] ; National Natural Science Foundation of China[62002355] ; National Natural Science Foundation of China[61720106006] ; National Natural Science Foundation of China[62102415] ; National Natural Science Foundation of China[62072455] ; National Natural Science Foundation of China[62202331] ; National Natural Science Foundation of China[62206200] ; National Natural Science Foundation of China[62106262] ; Tianjin Natural Science Foundation[22JCYBJC00030] ; Beijing Natural Science Foundation[L201001] ; Beijing Natural Science Foundation[4222039]
WOS关键词	QUESTION ; INFERENCE
WOS研究方向	Computer Science ; Engineering
语种	英语
出版者	IEEE-INST ELECTRICAL ELECTRONICS ENGINEERS INC
WOS记录号	WOS:001049970200005
资助机构	National Key Research and Development Plan of China ; National Natural Science Foundation of China ; Tianjin Natural Science Foundation ; Beijing Natural Science Foundation
内容类型	期刊论文
源URL	[http://ir.ia.ac.cn/handle/173211/54038]
专题	多模态人工智能系统全国重点实验室
通讯作者	Xu, Changsheng
作者单位	1.Univ Chinese Acad Sci, Sch Artificial Intelligence, Beijing 100049, Peoples R China 2.Peng Cheng Lab, Shenzhen 518066, Peoples R China 3.Tianjin Univ Technol, Sch Comp Sci & Engn, Tianjin 300384, Peoples R China 4.Minist Educ, Key Lab Comp Vis & Syst, Tianjin 300384, Peoples R China 5.Chinese Acad Sci, Inst Automat, State Key Lab Multimodal Artificial Intelligence S, Beijing 100190, Peoples R China
推荐引用方式 GB/T 7714	Zhang, Xi,Zhang, Feifei,Xu, Changsheng. Reducing Vision-Answer Biases for Multiple-Choice VQA[J]. IEEE TRANSACTIONS ON IMAGE PROCESSING,2023,32:4621-4634.
APA	Zhang, Xi,Zhang, Feifei,&Xu, Changsheng.(2023).Reducing Vision-Answer Biases for Multiple-Choice VQA.IEEE TRANSACTIONS ON IMAGE PROCESSING,32,4621-4634.
MLA	Zhang, Xi,et al."Reducing Vision-Answer Biases for Multiple-Choice VQA".IEEE TRANSACTIONS ON IMAGE PROCESSING 32(2023):4621-4634.