Metadata Distribution and Consistency Techniques for Large-Scale Cluster File Systems

doi:10.1109/TPDS.2010.154

CORC > 计算技术研究所 > 中国科学院计算技术研究所 > 中国科学院计算技术研究所期刊论文 > 英文

	Metadata Distribution and Consistency Techniques for Large-Scale Cluster File Systems
	Xiong, Jin 1; Hu, Yiming 2; Li, Guojie 1; Tang, Rongfeng 1; Fan, Zhihua 3
刊名	IEEE TRANSACTIONS ON PARALLEL AND DISTRIBUTED SYSTEMS
	2011-05-01
卷号	22 期号:5 页码:803-816
关键词	Distributed file systems metadata management
ISSN号	1045-9219
DOI	10.1109/TPDS.2010.154
英文摘要	Most supercomputers nowadays are based on large clusters, which call for sophisticated, scalable, and decentralized metadata processing techniques. From the perspective of maximizing metadata throughput, an ideal metadata distribution policy should automatically balance the namespace locality and even distribution without manual intervention. None of existing metadata distribution schemes is designed to make such a balance. We propose a novel metadata distribution policy, Dynamic Dir-Grain (DDG), which seeks to balance the requirements of keeping namespace locality and even distribution of the load by dynamic partitioning of the namespace into size-adjustable hierarchical units. Extensive simulation and measurement results show that DDG policies with a proper granularity significantly outperform traditional techniques such as the Random policy and the Subtree policy by 40 percent to 62 times. In addition, from the perspective of file system reliability, metadata consistency is an equally important issue. However, it is complicated by dynamic metadata distribution. Metadata consistency of cross-metadata server operations cannot be solved by traditional metadata journaling on each server. While traditional two-phase commit (2PC) algorithm can be used, it is too costly for distributed file systems. We proposed a consistent metadata processing protocol, S2PC-MP, which combines the two-phase commit algorithm with metadata processing to reduce overheads. Our measurement results show that S2PC-MP not only ensures fast recovery, but also greatly reduces fail-free execution overheads.
资助项目	Natural Science Foundation of China[60970025] ; National High-Technology Research and Development Program of China[2006AA01A102] ; National High-Technology Research and Development Program of China[2009AA01Z139] ; National High-Technology Research and Development Program of China[2009AA01A129] ; National Science Foundation[CCF-0541103]
WOS研究方向	Computer Science ; Engineering
语种	英语
出版者	IEEE COMPUTER SOC
WOS记录号	WOS:000288755400009
内容类型	期刊论文
源URL	[http://119.78.100.204/handle/2XEOYT63/12605]
专题	中国科学院计算技术研究所期刊论文_英文
通讯作者	Xiong, Jin
作者单位	1.Chinese Acad Sci, Inst Comp Technol, Beijing 100190, Peoples R China 2.Univ Cincinnati, Dept Elect & Comp Engn & Comp Sci, Engn Res Ctr 542, Cincinnati, OH 45221 USA 3.NetEase Com Inc, Beijing 100084, Peoples R China
推荐引用方式 GB/T 7714	Xiong, Jin,Hu, Yiming,Li, Guojie,et al. Metadata Distribution and Consistency Techniques for Large-Scale Cluster File Systems[J]. IEEE TRANSACTIONS ON PARALLEL AND DISTRIBUTED SYSTEMS,2011,22(5):803-816.
APA	Xiong, Jin,Hu, Yiming,Li, Guojie,Tang, Rongfeng,&Fan, Zhihua.(2011).Metadata Distribution and Consistency Techniques for Large-Scale Cluster File Systems.IEEE TRANSACTIONS ON PARALLEL AND DISTRIBUTED SYSTEMS,22(5),803-816.
MLA	Xiong, Jin,et al."Metadata Distribution and Consistency Techniques for Large-Scale Cluster File Systems".IEEE TRANSACTIONS ON PARALLEL AND DISTRIBUTED SYSTEMS 22.5(2011):803-816.