HScheduler: an optimal approach to minimize the makespan of multiple MapReduce jobs

doi:10.1007/s11227-016-1737-4

CORC > 重庆绿色智能技术研究院 > 中国科学院重庆绿色智能技术研究院 > 大数据挖掘及应用中心

	HScheduler: an optimal approach to minimize the makespan of multiple MapReduce jobs
	Tian, Wenhong1,3,4 ; Li, Guozhong 1; Yang, Wutong 1; Buyya, Rajkumar 2
刊名	JOURNAL OF SUPERCOMPUTING
	2016-06-01
卷号	72 期号:6 页码:2376-2393
关键词	Hadoop MapReduce Batch workloads Optimized schedule Minimized makespan
ISSN号	0920-8542
DOI	10.1007/s11227-016-1737-4
通讯作者	Tian, WH (reprint author), Univ Elect Sci & Technol China, Sch Informat & Software Engn, Chengdu 610054, Peoples R China. ; Tian, WH (reprint author), Univ Elect Sci & Technol China, Big Data Res Ctr, Chengdu 610054, Peoples R China. ; Tian, WH (reprint author), Chinese Acad Sci, Chongqing Inst Green & Intelligent Technol, Chongqing 400714, Peoples R China.
英文摘要	Large-scale MapReduce clusters that routinely process big data bring challenges to the cloud computing. One of the key challenges is to reduce the response time of these MapReduce clusters by minimizing their makespans. It is observed that the order in which these jobs are executed can have a significant impact on their overall makespans and resource utilization. In this work, we consider a scheduling model for multiple MapReduce jobs. The goal is to design a job scheduler that minimizes the makespan of such a set of MapReduce jobs. We exploit classical Johnson model and propose a novel framework HScheduler, which combines features of both classical Johnson's algorithm and MapReduce to minimize the makespan for both offline and online jobs. Our Offline HScheduler reaches the theoretical lower bound (optimum) and Online HScheduler is 2-competitive which is the best-known constant ratio for minimizing the makespan. Through extensive real data tests, we find that HScheduler has better performance than the best-known approach by 10.6-11.7 % on average for offline scheduling and 8-10 % on average for online scheduling. The HScheduler can be applied to improve responsive time, throughput and energy efficiency in cloud computing.
资助项目	China National Science Foundation (CNSF)[61450110440] ; Sichuan Province Technology Plan[2016GZ0322] ; Chongqing Research Program of Basic Research and Frontier Technology[cstc2015jcyjB0244]
WOS研究方向	Computer Science ; Engineering
语种	英语
出版者	SPRINGER
WOS记录号	WOS:000376650000015
内容类型	期刊论文
源URL	[http://119.78.100.138/handle/2HOD01W0/2530]
专题	大数据挖掘及应用中心
通讯作者	Tian, Wenhong
作者单位	1.Univ Elect Sci & Technol China, Sch Informat & Software Engn, Chengdu 610054, Peoples R China 2.Univ Melbourne, Parkville, Vic 3052, Australia 3.Univ Elect Sci & Technol China, Big Data Res Ctr, Chengdu 610054, Peoples R China 4.Chinese Acad Sci, Chongqing Inst Green & Intelligent Technol, Chongqing 400714, Peoples R China
推荐引用方式 GB/T 7714	Tian, Wenhong,Li, Guozhong,Yang, Wutong,et al. HScheduler: an optimal approach to minimize the makespan of multiple MapReduce jobs[J]. JOURNAL OF SUPERCOMPUTING,2016,72(6):2376-2393.
APA	Tian, Wenhong,Li, Guozhong,Yang, Wutong,&Buyya, Rajkumar.(2016).HScheduler: an optimal approach to minimize the makespan of multiple MapReduce jobs.JOURNAL OF SUPERCOMPUTING,72(6),2376-2393.
MLA	Tian, Wenhong,et al."HScheduler: an optimal approach to minimize the makespan of multiple MapReduce jobs".JOURNAL OF SUPERCOMPUTING 72.6(2016):2376-2393.