Stochastic Variance Reduction with Adaptive Optimization Methods

In this paper, we study the optimization technique known as Stochastic Variance Reduced Gradient (SVRG), a variant of Stochastic Gradient Descent (SGD), and analyze the effects of swapping its internal SGD evaluation with adaptive optimization algorithms, such as Adam and RMSProp on Machine Learning problems. We show that the direct application of this technique to neural networks does not result in improvements over SGD or Adaptive SGD methods.