QQ客服 官方微博 反馈留言

	StroMAX: Partitioning-Based Scheduler for Real-Time Stream Processing System
	Jiang, Jiawei ; Zhang, Zhipeng ; Cui, Bin ; Tong, Yunhai ; Xu, Ning
	2017
关键词	Real-time stream processing Task allocation Workload scheduling Graph partition
英文摘要	With the increasing availability and scale of data from Web 2.0, the ability to efficiently and timely analyze huge amounts of data is important for industry success. A number of real-time stream processing platforms have been developed, such as Storm, S4, and Flume. A fundamental problem of these large scale decentralized stream processing systems is how to deploy the workload to each node so as to fully utilize the available resources and optimize the overall system performance. In this paper, we present StroMAX, a graph-partitioning based approach of workload scheduling for real-time stream processing systems. StroMAX uses two advanced generic schedulers to improve the performance of stream processing systems by reducing the inter-node communication cost while keeping the workload of nodes below a certain computational load threshold. The first scheduler analyzes the workload structure when a job is committed and uses the graph-partitioning result to determine the deployment of tasks. The second scheduler analyzes the statistical information of physical nodes, and dynamically reassigns the tasks during runtime to improve the overall performance. Besides, StroMAXcan be deployed to many other state-of-the-art real-time stream processing systems easily. We implemented StroMAX on Storm, a representative real-time stream processing system. Extensive experiments conducted with real-world workloads and datasets demonstrate the superiority of our approaches against the existing solutions.; National Natural Science Foundation of China [61572039]; Shenzhen Government Research [JCYJ20151014093505032]; 973 program [2014CB340405]; Tecent Research Grant (PKU); CPCI-S(ISTP); 269-288; 10178
语种	英语
出处	22nd International Conference on Database Systems for Advanced Applications (DASFAA)
DOI标识	10.1007/978-3-319-55699-4_17
内容类型	其他
源URL	[http://ir.pku.edu.cn/handle/20.500.11897/480766]
专题	信息科学技术学院
推荐引用方式 GB/T 7714	Jiang, Jiawei,Zhang, Zhipeng,Cui, Bin,et al. StroMAX: Partitioning-Based Scheduler for Real-Time Stream Processing System. 2017-01-01.

个性服务

查看访问统计

相关权益政策

暂无数据

收藏/分享

除非特别说明，本系统中所有内容都受版权保护，并保留所有权利。