This paper aims to explore seven commonly used optimization algorithms in deep learning: SGD, Momentum-SGD, NAG, AdaGrad, RMSprop, AdaDelta, and Adam. Based on an overview of their theories and ...