MapGuide: A Simple yet Effective Method to Reconstruct Continuous Language from Brain Activities | |
Xinpei, Zhao1,2; Jingyuan, Sun3; Shaonan, Wang1,2; Jing, Ye1,2; Xiaohan, Zhang1,2; Chengqing, Zong1,2 | |
2024-06 | |
会议日期 | 2024-6 |
会议地点 | Mexico City, Mexico |
关键词 | neural decoding |
卷号 | - |
期号 | - |
DOI | - |
页码 | 3822–3832 |
英文摘要 | Decoding continuous language from brain activity is a formidable yet promising field of research. It is particularly significant for aiding people with speech disabilities to communicate through brain signals. This field addresses the complex task of mapping brain signals to text. The previous best attempt reverse-engineered this process in an indirect way: it began by learning to encode brain activity from text and then guided text generation by aligning with predicted brain responses. In contrast, we propose a simple yet effective method that guides text reconstruction by directly comparing them with the predicted text embeddings mapped from brain activities. Comprehensive experiments reveal that our method significantly outperforms the current state-of-the-art model, showing average improvements of 77% and 54% on BLEU and METEOR scores. We further validate the proposed modules through detailed ablation studies and case analyses and highlight a critical correlation: the more precisely we map brain activities to text embeddings, the better the text reconstruction results. Such insight can simplify the task of reconstructing language from brain activities for future work, emphasizing the importance of improving brain-to-text-embedding mapping techniques. |
源文献作者 | Association for Computational Linguistics |
会议录 | Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers)
![]() |
会议录出版者 | Association for Computational Linguistics |
会议录出版地 | - |
语种 | 英语 |
URL标识 | 查看原文 |
内容类型 | 会议论文 |
源URL | [http://ir.ia.ac.cn/handle/173211/57635] ![]() |
专题 | 模式识别国家重点实验室_自然语言处理 |
通讯作者 | Jingyuan, Sun; Shaonan, Wang |
作者单位 | 1.State Key Laboratory of Multimodal Artificial Intelligence Systems, Institute of Automation, CAS, Beijing, China 2.School of Artificial Intelligence, University of Chinese Academy of Sciences, Beijing, China 3.KU Leuven, Leuven, Belgium |
推荐引用方式 GB/T 7714 | Xinpei, Zhao,Jingyuan, Sun,Shaonan, Wang,et al. MapGuide: A Simple yet Effective Method to Reconstruct Continuous Language from Brain Activities[C]. 见:. Mexico City, Mexico. 2024-6. |
个性服务 |
查看访问统计 |
相关权益政策 |
暂无数据 |
收藏/分享 |
除非特别说明,本系统中所有内容都受版权保护,并保留所有权利。
修改评论