A New Data Access Mechanism for HDFS

doi:10.1088/1742-6596/898/6/062018

	A New Data Access Mechanism for HDFS
	Li Q(李强); Sun ZY(孙震宇); Sun GX(孙功星); Li, Qiang; Sun, Zhenyu; Wei, Zhanchen ; Sun, Gongxing; Wei ZC(魏占辰)
刊名	Journal of Physics: Conference Series
	2017
卷号	898 期号:6 页码:062018
ISSN号	1742-6588
DOI	10.1088/1742-6596/898/6/062018
文献子类	Proceedings Paper
英文摘要	With the era of big data emerging, Hadoop has become the de facto standard of big data processing platform. However, it is still difficult to get legacy applications, such as High Energy Physics (HEP) applications, to run efficiently on Hadoop platform. There are two reasons which lead to the difficulties mentioned above: firstly, random access is not supported on Hadoop File System (HDFS), secondly, it is difficult to make legacy applications adopt to HDFS streaming data processing mode. In order to address the two issues, a new read and write mechanism of HDFS is proposed. With this mechanism, data access is done on the local file system instead of through HDFS streaming interfaces. To enable files modified by users, three attributes including permissions, owner and group are imposed on Block objects. Blocks stored on Datanodes have the same attributes as the file they are owned by. Users can modify blocks when the Map task running locally, and HDFS is responsible to update the rest replicas later after the block modification finished. To further improve the performance of Hadoop system, a complete localization task execution mechanism is implemented for I/O intensive jobs. Test results show that average CPU utilization is improved by 10% with the new task selection strategy, data read and write performances are improved by about 10% and 30% separately. © Published under licence by IOP Publishing Ltd.
电子版国际标准刊号	1742-6596
会议地点	San Francisco, CA, United states
会议日期	October 10, 2016 - October 14, 2016
语种	英语
内容类型	期刊论文
源URL	[http://ir.ihep.ac.cn/handle/311005/284221]
专题	中国科学院高能物理研究所
作者单位	1.University of Chinese, Academy of Sciences, Beijing, China 2.Institute of High Energy Physics, Beijing, China;
推荐引用方式 GB/T 7714	Li Q,Sun ZY,Sun GX,et al. A New Data Access Mechanism for HDFS[J]. Journal of Physics: Conference Series,2017,898(6):062018.
APA	李强.,孙震宇.,孙功星.,Li, Qiang.,Sun, Zhenyu.,...&魏占辰.(2017).A New Data Access Mechanism for HDFS.Journal of Physics: Conference Series,898(6),062018.
MLA	李强,et al."A New Data Access Mechanism for HDFS".Journal of Physics: Conference Series 898.6(2017):062018.