Stochastic Variance Reduced Gradient Methods Using a Trust-Region-Like Scheme

doi:10.1007/s10915-020-01402-x

	Stochastic Variance Reduced Gradient Methods Using a Trust-Region-Like Scheme
	Yu, Tengteng 1; Liu, Xin-Wei 2; Dai, Yu-Hong 4,5; Sun, Jie 2,3
刊名	JOURNAL OF SCIENTIFIC COMPUTING
	2021-02-17
卷号	87 期号:1 页码:24
关键词	Stochastic variance reduced gradient Trust region Barzilai-Borwein stepsizes Mini-batches Empirical risk minimization 90C06 90C30 90C90 90C25
ISSN号	0885-7474
DOI	10.1007/s10915-020-01402-x
英文摘要	Stochastic variance reduced gradient (SVRG) methods are important approaches to minimize the average of a large number of cost functions frequently arising in machine learning and many other applications. In this paper, based on SVRG, we propose a SVRG-TR method which employs a trust-region-like scheme for selecting stepsizes. It is proved that the SVRG-TR method is linearly convergent in expectation for smooth strongly convex functions and enjoys a faster convergence rate than SVRG methods. In order to overcome the difficulty of tuning stepsizes by hand, we propose to combine the Barzilai-Borwein (BB) method to automatically compute stepsizes for the SVRG-TR method, named as the SVRG-TR-BB method. By incorporating mini-batching scheme with SVRG-TR and SVRG-TR-BB, respectively, we further propose two extended methods mSVRG-TR and mSVRG-TR-BB. Linear convergence and complexity of mSVRG-TR are given. Numerical experiments on some standard datasets show that SVRG-TR and SVRG-TR-BB are generally better than or comparable to SVRG with best-tuned stepsizes and some modern stochastic gradient methods, while mSVRG-TR and mSVRG-TR-BB are very competitive with mini-batch variants of recent successful stochastic gradient methods.
资助项目	Chinese NSF[11671116] ; Chinese NSF[11701137] ; Chinese NSF[12071108] ; Chinese NSF[11631013] ; Chinese NSF[11991020] ; Chinese NSF[12021001] ; Major Research Plan of the NSFC[91630202] ; Beijing Academy of Artificial Intelligence (BAAI)
WOS研究方向	Mathematics
语种	英语
出版者	SPRINGER/PLENUM PUBLISHERS
WOS记录号	WOS:000620641500005
内容类型	期刊论文
源URL	[http://ir.amss.ac.cn/handle/2S8OKBNM/58194]
专题	中国科学院数学与系统科学研究院
通讯作者	Liu, Xin-Wei
作者单位	1.Hebei Univ Technol, Sch Artificial Intelligence, Tianjin 300401, Peoples R China 2.Hebei Univ Technol, Inst Math, Tianjin 300401, Peoples R China 3.Natl Univ Singapore, Sch Business, Singapore 119245, Singapore 4.Univ Chinese Acad Sci, Sch Math Sci, Beijing 100049, Peoples R China 5.Chinese Acad Sci, Acad Math & Syst Sci, Inst Computat Math & Sci Engn Comp, State Key Lab Sci & Engn Comp, Beijing 100190, Peoples R China
推荐引用方式 GB/T 7714	Yu, Tengteng,Liu, Xin-Wei,Dai, Yu-Hong,et al. Stochastic Variance Reduced Gradient Methods Using a Trust-Region-Like Scheme[J]. JOURNAL OF SCIENTIFIC COMPUTING,2021,87(1):24.
APA	Yu, Tengteng,Liu, Xin-Wei,Dai, Yu-Hong,&Sun, Jie.(2021).Stochastic Variance Reduced Gradient Methods Using a Trust-Region-Like Scheme.JOURNAL OF SCIENTIFIC COMPUTING,87(1),24.
MLA	Yu, Tengteng,et al."Stochastic Variance Reduced Gradient Methods Using a Trust-Region-Like Scheme".JOURNAL OF SCIENTIFIC COMPUTING 87.1(2021):24.