Metadata Distribution and Consistency Techniques for Large-Scale Cluster File Systems | |
Xiong, Jin1; Hu, Yiming2; Li, Guojie1; Tang, Rongfeng1; Fan, Zhihua3 | |
刊名 | IEEE TRANSACTIONS ON PARALLEL AND DISTRIBUTED SYSTEMS
![]() |
2011-05-01 | |
卷号 | 22期号:5页码:803-816 |
关键词 | Distributed file systems metadata management |
ISSN号 | 1045-9219 |
DOI | 10.1109/TPDS.2010.154 |
英文摘要 | Most supercomputers nowadays are based on large clusters, which call for sophisticated, scalable, and decentralized metadata processing techniques. From the perspective of maximizing metadata throughput, an ideal metadata distribution policy should automatically balance the namespace locality and even distribution without manual intervention. None of existing metadata distribution schemes is designed to make such a balance. We propose a novel metadata distribution policy, Dynamic Dir-Grain (DDG), which seeks to balance the requirements of keeping namespace locality and even distribution of the load by dynamic partitioning of the namespace into size-adjustable hierarchical units. Extensive simulation and measurement results show that DDG policies with a proper granularity significantly outperform traditional techniques such as the Random policy and the Subtree policy by 40 percent to 62 times. In addition, from the perspective of file system reliability, metadata consistency is an equally important issue. However, it is complicated by dynamic metadata distribution. Metadata consistency of cross-metadata server operations cannot be solved by traditional metadata journaling on each server. While traditional two-phase commit (2PC) algorithm can be used, it is too costly for distributed file systems. We proposed a consistent metadata processing protocol, S2PC-MP, which combines the two-phase commit algorithm with metadata processing to reduce overheads. Our measurement results show that S2PC-MP not only ensures fast recovery, but also greatly reduces fail-free execution overheads. |
资助项目 | Natural Science Foundation of China[60970025] ; National High-Technology Research and Development Program of China[2006AA01A102] ; National High-Technology Research and Development Program of China[2009AA01Z139] ; National High-Technology Research and Development Program of China[2009AA01A129] ; National Science Foundation[CCF-0541103] |
WOS研究方向 | Computer Science ; Engineering |
语种 | 英语 |
出版者 | IEEE COMPUTER SOC |
WOS记录号 | WOS:000288755400009 |
内容类型 | 期刊论文 |
源URL | [http://119.78.100.204/handle/2XEOYT63/12605] ![]() |
专题 | 中国科学院计算技术研究所期刊论文_英文 |
通讯作者 | Xiong, Jin |
作者单位 | 1.Chinese Acad Sci, Inst Comp Technol, Beijing 100190, Peoples R China 2.Univ Cincinnati, Dept Elect & Comp Engn & Comp Sci, Engn Res Ctr 542, Cincinnati, OH 45221 USA 3.NetEase Com Inc, Beijing 100084, Peoples R China |
推荐引用方式 GB/T 7714 | Xiong, Jin,Hu, Yiming,Li, Guojie,et al. Metadata Distribution and Consistency Techniques for Large-Scale Cluster File Systems[J]. IEEE TRANSACTIONS ON PARALLEL AND DISTRIBUTED SYSTEMS,2011,22(5):803-816. |
APA | Xiong, Jin,Hu, Yiming,Li, Guojie,Tang, Rongfeng,&Fan, Zhihua.(2011).Metadata Distribution and Consistency Techniques for Large-Scale Cluster File Systems.IEEE TRANSACTIONS ON PARALLEL AND DISTRIBUTED SYSTEMS,22(5),803-816. |
MLA | Xiong, Jin,et al."Metadata Distribution and Consistency Techniques for Large-Scale Cluster File Systems".IEEE TRANSACTIONS ON PARALLEL AND DISTRIBUTED SYSTEMS 22.5(2011):803-816. |
个性服务 |
查看访问统计 |
相关权益政策 |
暂无数据 |
收藏/分享 |
除非特别说明,本系统中所有内容都受版权保护,并保留所有权利。
修改评论