Theoretical Analysis of Learning Speed in Gradient Descent Algorithm Replacing Derivative with Constant

スポンサーリンク

概要

• 論文の詳細を見る

• In on-line gradient descent learning, the local property of the derivative term of the output function can slowly converge. Improving the derivative term, such as by using the natural gradient, has been proposed for speeding up the convergence. Beside this sophisticated method, we propose an algorithm that replaces the derivative term with a constant and show that this greatly increases convergence speed when the learning step size is less than 2.7, which is near the optimal learning step size. The proposed algorithm is inspired by linear perceptron learning and can avoid locality of the derivative term. We derived the closed deterministic differential equations by using a statistical mechanics method and show the validity of theoretical results by comparing them with computer simulation solutions. In real problems, the optimum learning step size is not given in advance. Therefore, the learning step size must be small. The proposed method is useful in this case.

著者

• Hara Kazuyuki

  College Of Industrial Technology Nihon University

• Katahira Kentaro

  Center for Evolutionary Cognitive Sciences, The University of Tokyo

• Hara Kazuyuki

  College of Industrial Technology, Nihon University

関連論文

• Statistical Mechanics of On-Line Node-Perturbation Learning
• Statistical Mechanics of On-line Node-perturbation Learning
• Statistical Mechanics of On-line Ensemble Teacher Learning through a Novel Perceptron Learning Rule
• Statistical Mechanics of Node-Perturbation Learning for Nonlinear Perceptron
• Theoretical Analysis of Learning Speed in Gradient Descent Algorithm Replacing Derivative with Constant

スポンサーリンク