Plateau and Quasi-Plateau Due to Singularity in Multilayer Perceptrons--Standard Gradient vs. Natural Gradient (特集統計的学習理論及び一般)

概要

論文の詳細を見る
In a hierarchical set of models, a low order model is included in the parameter space of a larger order model as a union of subspaces, which makes singularities in the parameter space. The complex singular structure is known to affect estimation performances and learning dynamics of the models. Although there have been a number of studies on properties of obtained estimators for the models, there are few studies on the dynamical properties of learning exploited for obtaining the estimators. Taking Statistical mechanical approach, we investigate influences of singularities on dynamics of standard gradient learning and natural gradient learning under various learning conditions for mutilayer percpetorns. In the standard gradient learning, we found a quasi-plateau phenomenon, which is severer than the well known plateau in some cases. The slow convergence due to the quasi-plateau and plateau becomes extremely serious when an optimal point is in a neighborhood of singularity. In the natural gradient learning, however, the quasi-plateau and plateau are not observed and convergence speed is hardly affected by singularity.
社団法人電子情報通信学会の論文
2003-07-21