A load balancing algorithm for distributed crawler system | |
Zhang ST(张树涛)1,2; Tan HB(谭海波)2; Chen LF(陈良锋)2; Lv B(吕波)2 | |
2019-06 | |
会议日期 | 2019-06-12 |
会议地点 | 中国重庆 |
关键词 | distributed crawler system load balancing runtime model approximate gradient descent constrained optimization |
期号 | 2472-8489 |
DOI | 10.1109/ICCSN.2019.8905323 |
英文摘要 | To address the problem of load balance in the distributed crawler system, we propose a new load balancing algorithm. First, we investigate the impact of different features among nodes of distributed crawler system on running time. Based on these features, a runtime model is established. Second, the minimum variance of running time of each node predicted by the model is regarded as the load-balancing objective function. Finally, we utilize an approximate gradient descent (AGD) to optimize the objective function and produce a sequence of task allocation to the nodes. The experimental results show an improvement in the running time of the distributed crawler system. |
会议录 | 2019 IEEE 11th International Conference on Communication Software and Networks
![]() |
语种 | 英语 |
内容类型 | 会议论文 |
源URL | [http://ir.hfcas.ac.cn:8080/handle/334002/43169] ![]() |
专题 | 合肥物质科学研究院_信息中心 |
作者单位 | 1.中国科学技术大学 2.中国科学院合肥物质科学研究院 |
推荐引用方式 GB/T 7714 | Zhang ST,Tan HB,Chen LF,et al. A load balancing algorithm for distributed crawler system[C]. 见:. 中国重庆. 2019-06-12. |
个性服务 |
查看访问统计 |
相关权益政策 |
暂无数据 |
收藏/分享 |
除非特别说明,本系统中所有内容都受版权保护,并保留所有权利。
修改评论