Structured Neural Decoding With Multitask Transfer Learning of Deep Neural Network Representations | |
Du, Changde1,2,3; Du, Changying5; Huang, Lijie3; Wang, Haibao2,3; He, Huiguang2,3,4 | |
刊名 | IEEE TRANSACTIONS ON NEURAL NETWORKS AND LEARNING SYSTEMS |
2022-02-01 | |
卷号 | 33期号:2页码:600-614 |
关键词 | Decoding Image reconstruction Functional magnetic resonance imaging Visualization Task analysis Brain Correlation Deep neural network (DNN) functional magnetic resonance imaging (fMRI) image reconstruction multioutput regression neural decoding |
ISSN号 | 2162-237X |
DOI | 10.1109/TNNLS.2020.3028167 |
通讯作者 | He, Huiguang(huiguang.he@ia.ac.cn) |
英文摘要 | The reconstruction of visual information from human brain activity is a very important research topic in brain decoding. Existing methods ignore the structural information underlying the brain activities and the visual features, which severely limits their performance and interpretability. Here, we propose a hierarchically structured neural decoding framework by using multitask transfer learning of deep neural network (DNN) representations and a matrix-variate Gaussian prior. Our framework consists of two stages, Voxel2Unit and Unit2Pixel. In Voxel2Unit, we decode the functional magnetic resonance imaging (fMRI) data to the intermediate features of a pretrained convolutional neural network (CNN). In Unit2Pixel, we further invert the predicted CNN features back to the visual images. Matrix-variate Gaussian prior allows us to take into account the structures between feature dimensions and between regression tasks, which are useful for improving decoding effectiveness and interpretability. This is in contrast with the existing single-output regression models that usually ignore these structures. We conduct extensive experiments on two real-world fMRI data sets, and the results show that our method can predict CNN features more accurately and reconstruct the perceived natural images and faces with higher quality. |
资助项目 | National Natural Science Foundation of China[61976209] ; National Natural Science Foundation of China[62020106015] ; National Natural Science Foundation of China[61906188] ; National Natural Science Foundation of China[61602449] ; Chinese Academy of Sciences (CAS) International Collaboration Key Project[173211KYSB20190024] ; CAS[XDB32040000] |
WOS关键词 | NATURAL IMAGES ; BRAIN ; RECONSTRUCTION ; FACES |
WOS研究方向 | Computer Science ; Engineering |
语种 | 英语 |
出版者 | IEEE-INST ELECTRICAL ELECTRONICS ENGINEERS INC |
WOS记录号 | WOS:000752016400015 |
资助机构 | National Natural Science Foundation of China ; Chinese Academy of Sciences (CAS) International Collaboration Key Project ; CAS |
内容类型 | 期刊论文 |
源URL | [http://ir.ia.ac.cn/handle/173211/47361] |
专题 | 类脑智能研究中心_神经计算及脑机交互 |
通讯作者 | He, Huiguang |
作者单位 | 1.Huawei Cloud BU EI Innovat Lab, Beijing 100085, Peoples R China 2.Univ Chinese Acad Sci, Sch Artificial Intelligence, Beijing 100049, Peoples R China 3.Chinese Acad Sci, Res Ctr Brain Inspired Intelligence, Natl Lab Pattern Recognit, Inst Automat, Beijing 100190, Peoples R China 4.Chinese Acad Sci, Ctr Excellence Brain Sci & Intelligence Technol, Beijing 100190, Peoples R China 5.Huawei Noahs Ark Lab, Beijing 100085, Peoples R China |
推荐引用方式 GB/T 7714 | Du, Changde,Du, Changying,Huang, Lijie,et al. Structured Neural Decoding With Multitask Transfer Learning of Deep Neural Network Representations[J]. IEEE TRANSACTIONS ON NEURAL NETWORKS AND LEARNING SYSTEMS,2022,33(2):600-614. |
APA | Du, Changde,Du, Changying,Huang, Lijie,Wang, Haibao,&He, Huiguang.(2022).Structured Neural Decoding With Multitask Transfer Learning of Deep Neural Network Representations.IEEE TRANSACTIONS ON NEURAL NETWORKS AND LEARNING SYSTEMS,33(2),600-614. |
MLA | Du, Changde,et al."Structured Neural Decoding With Multitask Transfer Learning of Deep Neural Network Representations".IEEE TRANSACTIONS ON NEURAL NETWORKS AND LEARNING SYSTEMS 33.2(2022):600-614. |
个性服务 |
查看访问统计 |
相关权益政策 |
暂无数据 |
收藏/分享 |
除非特别说明,本系统中所有内容都受版权保护,并保留所有权利。
修改评论