SgVA-CLIP: Semantic-Guided Visual Adapting of Vision-Language Models for Few-Shot Image Classification

doi:10.1109/TMM.2023.3311646

CORC > 自动化研究所 > 中国科学院自动化研究所 > 模式识别国家重点实验室 > 多媒体计算与图形学团队

	SgVA-CLIP: Semantic-Guided Visual Adapting of Vision-Language Models for Few-Shot Image Classification
	Peng, Fang 1,2,4; Yang, Xiaoshan 2,3,4; Xiao, Linhui 1,2,4; Wang, Yaowei 4; Xu, Changsheng 2,3,4
刊名	IEEE TRANSACTIONS ON MULTIMEDIA
	2024
卷号	26 页码:3469-3480
关键词	Few-shot image classification vision-language models
ISSN号	1520-9210
DOI	10.1109/TMM.2023.3311646
通讯作者	Xu, Changsheng(csxu@nlpr.ia.ac.cn)
英文摘要	Although significant progress has been made in few-shot learning, most of existing few-shot image classification methods require supervised pre-training on a large amount of samples of base classes, which limits their generalization ability in real world application. Recently, large-scale Vision-Language Pre-trained models (VLPs) have been gaining increasing attention in few-shot learning because they can provide a new paradigm for transferable visual representation learning with easily available text on the Web. However, the VLPs may neglect detailed visual information that is difficult to describe by language sentences, but important for learning an effective classifier to distinguish different images. To address the above problem, we propose a new framework, named Semantic-guided Visual Adapting (SgVA), which can effectively extend vision-language pre-trained models to produce discriminative adapted visual features by comprehensively using an implicit knowledge distillation, a vision-specific contrastive loss, and a cross-modal contrastive loss. The implicit knowledge distillation is designed to transfer the fine-grained cross-modal knowledge to guide the updating of the vision adapter. State-of-the-art results on 13 datasets demonstrate that the adapted visual features can well complement the cross-modal features to improve few-shot image classification.
资助项目	National Natural Science Foundation of China
WOS研究方向	Computer Science ; Telecommunications
语种	英语
出版者	IEEE-INST ELECTRICAL ELECTRONICS ENGINEERS INC
WOS记录号	WOS:001165348200021
资助机构	National Natural Science Foundation of China
内容类型	期刊论文
源URL	[http://ir.ia.ac.cn/handle/173211/58025]
专题	自动化研究所_模式识别国家重点实验室_多媒体计算与图形学团队
通讯作者	Xu, Changsheng
作者单位	1.Chinese Acad Sci, Inst Automat, State Key Lab Multimodal Artificial Intelligence S, Beijing 100190, Peoples R China 2.Univ Chinese Acad Sci, Sch Artificial Intelligence, Beijing 100049, Peoples R China 3.Chinese Acad Sci, Inst Automat, State Key Lab Multimodal Artificial Intelligence S, Beijing 100190, Peoples R China 4.Peng Cheng Lab, Shenzhen 518066, Peoples R China
推荐引用方式 GB/T 7714	Peng, Fang,Yang, Xiaoshan,Xiao, Linhui,et al. SgVA-CLIP: Semantic-Guided Visual Adapting of Vision-Language Models for Few-Shot Image Classification[J]. IEEE TRANSACTIONS ON MULTIMEDIA,2024,26:3469-3480.
APA	Peng, Fang,Yang, Xiaoshan,Xiao, Linhui,Wang, Yaowei,&Xu, Changsheng.(2024).SgVA-CLIP: Semantic-Guided Visual Adapting of Vision-Language Models for Few-Shot Image Classification.IEEE TRANSACTIONS ON MULTIMEDIA,26,3469-3480.
MLA	Peng, Fang,et al."SgVA-CLIP: Semantic-Guided Visual Adapting of Vision-Language Models for Few-Shot Image Classification".IEEE TRANSACTIONS ON MULTIMEDIA 26(2024):3469-3480.