On Learning Semantic Representations for Large-Scale Abstract Sketches

doi:10.1109/TCSVT.2020.3041586

CORC > 自动化研究所 > 中国科学院自动化研究所 > 智能感知与计算研究中心

	On Learning Semantic Representations for Large-Scale Abstract Sketches
	Xu, Peng 1; Huang, Yongye 2; Yuan, Tongtong 3; Xiang, Tao 4; Hospedales, Timothy M.5; Song, Yi-Zhe 4; Wang, Liang 6
刊名	IEEE TRANSACTIONS ON CIRCUITS AND SYSTEMS FOR VIDEO TECHNOLOGY
	2021-09-01
卷号	31 期号:9 页码:3366-3379
关键词	Semantics Visualization Task analysis Games Feature extraction Quantization (signal) Speech recognition Practical sketch-based application semantic representation hashing retrieval zero-shot recognition edge-map dataset
ISSN号	1051-8215
DOI	10.1109/TCSVT.2020.3041586
通讯作者	Xu, Peng(peng.xu@ntu.edu.sg)
英文摘要	In this paper, we focus on learning semantic representations for large-scale highly abstract sketches that were produced by the practical sketch-based application rather than the excessively well dawn sketches obtained by crowd-sourcing. We propose a dual-branch CNN-RNN network architecture to represent sketches, which simultaneously encodes both the static and temporal patterns of sketch strokes. Based on this architecture, we further explore learning the sketch-oriented semantic representations in two practical settings, i.e., hashing retrieval and zero-shot recognition on million-scale highly abstract sketches produced by practical online interactions. Specifically, we use our dual-branch architecture as a universal representation framework to design two sketch-specific deep models: (i) We propose a deep hashing model for sketch retrieval, where a novel hashing loss is specifically designed to further accommodate both the abstract and messy traits of sketches. (ii) We propose a deep embedding model for sketch zero-shot recognition, via collecting a large-scale edge-map dataset and proposing to extract a set of semantic vectors from edge-maps as the semantic knowledge for sketch zero-shot domain alignment. Both deep models are evaluated by comprehensive experiments on million-scale abstract sketches produced by a global online game QuickDraw and outperform state-of-the-art competitors.
WOS关键词	ALGORITHMS
WOS研究方向	Engineering
语种	英语
出版者	IEEE-INST ELECTRICAL ELECTRONICS ENGINEERS INC
WOS记录号	WOS:000693647500007
内容类型	期刊论文
源URL	[http://ir.ia.ac.cn/handle/173211/45968]
专题	自动化研究所_智能感知与计算研究中心
通讯作者	Xu, Peng
作者单位	1.Nanyang Technol Univ, Sch Comp Sci & Engn, Singapore 639798, Singapore 2.ByteDance, Shenzhen 518000, Peoples R China 3.Beijing Univ Technol, Informat Technol Sch, Beijing 100124, Peoples R China 4.Univ Surrey, Ctr Vis Speech & Signal Proc CVSSP, Guildford GU2 7XH, Surrey, England 5.Univ Edinburgh, Sch Informat, Edinburgh EH8 9YL, Midlothian, Scotland 6.Chinese Acad Sci, Inst Automat, Beijing 100190, Peoples R China
推荐引用方式 GB/T 7714	Xu, Peng,Huang, Yongye,Yuan, Tongtong,et al. On Learning Semantic Representations for Large-Scale Abstract Sketches[J]. IEEE TRANSACTIONS ON CIRCUITS AND SYSTEMS FOR VIDEO TECHNOLOGY,2021,31(9):3366-3379.
APA	Xu, Peng.,Huang, Yongye.,Yuan, Tongtong.,Xiang, Tao.,Hospedales, Timothy M..,...&Wang, Liang.(2021).On Learning Semantic Representations for Large-Scale Abstract Sketches.IEEE TRANSACTIONS ON CIRCUITS AND SYSTEMS FOR VIDEO TECHNOLOGY,31(9),3366-3379.
MLA	Xu, Peng,et al."On Learning Semantic Representations for Large-Scale Abstract Sketches".IEEE TRANSACTIONS ON CIRCUITS AND SYSTEMS FOR VIDEO TECHNOLOGY 31.9(2021):3366-3379.