Consistent4D: Consistent 360° Dynamic Object Generation from Monocular Video | |
Jiang, Yanqin4,5; Zhang, Li6; Gao, Jin4,5; Hu, Weiming3,4,5; Yao, Yao1,2 | |
2024 | |
会议日期 | May 7th, 2024 to May 11th, 2024 |
会议地点 | Vienna Austria |
英文摘要 | In this paper, we present Consistent4D, a novel approach for generating 4D dynamic objects from uncalibrated monocular videos. Uniquely, we cast the 360-degree dynamic object reconstruction as a 4D generation problem, eliminating the need for tedious multi-view data collection and camera calibration. This is achieved by leveraging the object-level 3D-aware image diffusion model as the primary supervision signal for training dynamic Neural Radiance Fields (DyNeRF). Specifically, we propose a cascade DyNeRF to facilitate stable convergence and temporal continuity under the time-discrete supervision signal. To achieve spatial and temporal consistency of the 4D generation, an interpolation-driven consistency loss is further introduced, which aligns the rendered frames with the interpolated frames from a pre-trained video interpolation model. Extensive experiments show that the proposed Consistent4D significantly outperforms previous 4D reconstruction approaches as well as per-frame 3D generation approaches, opening up new possibilities for 4D dynamic object generation from a single-view uncalibrated video. Project page: https://consistent4d.github.io |
内容类型 | 会议论文 |
源URL | [http://ir.ia.ac.cn/handle/173211/57501] ![]() |
专题 | 自动化研究所_模式识别国家重点实验室_视频内容安全团队 |
作者单位 | 1.School of Intelligence Science and Technology, Nanjing University 2.State Key Laboratory for Novel Software Technology, Nanjing University 3.School of Information Science and Technology, ShanghaiTech University 4.State Key Laboratory of Multimodal Artificial Intelligence Systems (MAIS), CASIA 5.School of Artificial Intelligence, University of Chinese Academy of Sciences 6.School of Data Science, Fudan University |
推荐引用方式 GB/T 7714 | Jiang, Yanqin,Zhang, Li,Gao, Jin,et al. Consistent4D: Consistent 360° Dynamic Object Generation from Monocular Video[C]. 见:. Vienna Austria. May 7th, 2024 to May 11th, 2024. |
个性服务 |
查看访问统计 |
相关权益政策 |
暂无数据 |
收藏/分享 |
除非特别说明,本系统中所有内容都受版权保护,并保留所有权利。
修改评论