A Parallel File System with Application-Aware Data Layout Policies for Massive Remote Sensing Image Processing in Digital Earth

	A Parallel File System with Application-Aware Data Layout Policies for Massive Remote Sensing Image Processing in Digital Earth
	Wang, Lizhe 1; Ma, Yan 1; Zomaya, Albert Y.1; Ranjan, Rajiv 1; Chen, Dan 1
刊名	IEEE TRANSACTIONS ON PARALLEL AND DISTRIBUTED SYSTEMS
	2015
卷号	26 期号:6 页码:6489-6509
关键词	Parallel file system parallel I/O data-intensive computing digital earth big data remote sensing image processing
通讯作者	Wang, LZ (reprint author), Chinese Acad Sci, Inst Remote Sensing & Digital Earth, Beijing 100094, Peoples R China.
英文摘要	Remote sensing applications in Digital Earth are overwhelmed with vast quantities of remote sensing (RS) image data. The intolerable I/O burden introduced by the massive amounts of RS data and the irregular RS data access patterns has made the traditional cluster based parallel I/O systems no longer applicable. We propose a RS data object-based parallel file system for remote sensing applications and implement it with the OrangeFS file system. It provides application-aware data layout policies, together with RS data object based data I/O interfaces, for efficient support of various data access patterns of RS applications from the server side. With the prior knowledge of the desired RS data access patterns, HPGFS could offer relevant space-filling curves to organize the sliced 3-D data bricks and distribute them over I/O servers. In this way, data layouts consistent with expected data access patterns could be created to explore data locality and achieve performance improvement. Moreover, the multi-band RS data with complex structured geographical metadata could be accessed and managed as a single data object. Through experiments on remote sensing applications with different access patterns, we have achieved performance improvement of about 30 percent for I/O and 20 percent overall.
研究领域[WOS]	Computer Science, Theory & Methods ; Engineering, Electrical & Electronic
收录类别	SCI ; EI
语种	英语
WOS记录号	WOS:000354526000001
内容类型	期刊论文
源URL	[http://ir.ceode.ac.cn/handle/183411/38200]
专题	遥感与数字地球研究所_SCI/EI期刊论文_期刊论文
作者单位	1.[Wang, Lizhe 2.Ma, Yan] Chinese Acad Sci, Inst Remote Sensing & Digital Earth, Beijing 100094, Peoples R China 3.[Zomaya, Albert Y.] Univ Sydney, Sch Informat Technol, Sydney, NSW 2006, Australia 4.[Ranjan, Rajiv] ICT CSIRO, Marsfield, NSW, Australia 5.[Chen, Dan] Wuhan Univ, Sch Comp, Wuhan 430072, Hubei, Peoples R China
推荐引用方式 GB/T 7714	Wang, Lizhe,Ma, Yan,Zomaya, Albert Y.,et al. A Parallel File System with Application-Aware Data Layout Policies for Massive Remote Sensing Image Processing in Digital Earth[J]. IEEE TRANSACTIONS ON PARALLEL AND DISTRIBUTED SYSTEMS,2015,26(6):6489-6509.
APA	Wang, Lizhe,Ma, Yan,Zomaya, Albert Y.,Ranjan, Rajiv,&Chen, Dan.(2015).A Parallel File System with Application-Aware Data Layout Policies for Massive Remote Sensing Image Processing in Digital Earth.IEEE TRANSACTIONS ON PARALLEL AND DISTRIBUTED SYSTEMS,26(6),6489-6509.
MLA	Wang, Lizhe,et al."A Parallel File System with Application-Aware Data Layout Policies for Massive Remote Sensing Image Processing in Digital Earth".IEEE TRANSACTIONS ON PARALLEL AND DISTRIBUTED SYSTEMS 26.6(2015):6489-6509.