QQ客服 官方微博 反馈留言

	Prophet: Scheduling Executors with Time-varying Resource Demands on Data-Parallel Computation Frameworks
	Guoyao Xu; Cheng-Zhong Xu; Song Jiang
	2016
会议名称	ICAC 2016
会议地点	Germany
英文摘要	Efficiently scheduling execution instances of dataparallel computing frameworks, such as Spark and Dryad, on a multi-tenant environment is critical to applications' performance and systems' utilization. To this end, one has to avoid resource fragmentation and over-allocation so that both idleness and contention of resources can be minimized. To make effective scheduling decisions, a scheduler has to be informed of and exploit resource demands of individual execution instances, including both short-lived tasks and long-lived executors. The issue becomes particularly challenging when resource demands greatly vary over time within each instance. Prior studies often assume that a scheduling instance is either short lived or of gradually varying resource demands. However, when in-memory computing platforms, such as Spark, become increasingly popular, the assumption no longer holds. The execution instance for scheduling becomes executor, which executes an entire application once it is scheduled. Usually it is not short lived. Its resource demands are significantly timevarying. To address the inefficacy of current cluster schedulers, we propose a scheduling approach, namely Prophet, which takes resource demand variation within each executor into the scheduling decision. It leverages the fact that execution of a data-parallel application is pre-defined by a DAG structure and resource demands at various DAG stages are highly predictable. With this knowledge, Prophet schedules executors to minimize resource fragmentation and over-allocation. To deal with unexpected resource contention, Prophet adaptively backs off selected task(s) to reduce the contention. We have implemented Prophet in Apache Yarn running Spark. We evaluated it on a 16-server cluster, using 10 categories of a total of 90 application benchmarks. Compared to Yarn's default capacity and fair schedulers, Prophet reduces application makespan by up to 39% and reduces their median completion time by 23%.
收录类别	EI
语种	英语
内容类型	会议论文
源URL	[http://ir.siat.ac.cn:8080/handle/172644/10321]
专题	深圳先进技术研究院_数字所
作者单位	2016
推荐引用方式 GB/T 7714	Guoyao Xu,Cheng-Zhong Xu,Song Jiang. Prophet: Scheduling Executors with Time-varying Resource Demands on Data-Parallel Computation Frameworks[C]. 见:ICAC 2016. Germany.

个性服务

查看访问统计

相关权益政策

暂无数据

收藏/分享

除非特别说明，本系统中所有内容都受版权保护，并保留所有权利。