On-Line Learning Dynamics of Multilayer Perceptrons with Unidentifiable Parameters
スポンサーリンク
概要
- 論文の詳細を見る
In the over-realizable scenario, in which the student network has larger number of hidden units than the true or optimal network, some of weight parameters are unidentifiable. In this case, the teacher network makes an optimal subspace in the paremeter space. Using statistical mechanics, we investigate the on-line learning dynamics in the over-realizable sceario with unidentifiable parameters. We show that the convergence speed strongly depends on initial conditions of learning parameters. We also show a plateau-like phenomenon, which is different from the well-known plateaus caused by the permutation symmetry. In addition, we discuss the property of the terminal point of learning, relating to the singular structures.
- 社団法人電子情報通信学会の論文
- 2003-03-11
著者
-
Park Hyeyoung
Riken Brain Science Institute
-
PARK Hyeyoung
Computer Science Dept., Kyungpook National Univ.
-
INOUE Masato
RIKEN Brain Science Institute
-
OKADA Masato
"Intelligent Coorperation and Control", PRESTO, JST
-
Inoue Masato
Department Of Electrical Engineering And Bioscience School Of Science And Engineering Waseda Univers
-
Okada Masato
Graduate School Of Frontier Sciences The University Of Tokyo
-
Okada Masato
"Intelligent Coorperation and Control", PRESTO, JST
-
OKADA Masato
RIKEN BSI:Japan Scientific Technology Corp.:Graduate School of Frontier Science, The University of Tokyo
-
OKADA Masato
Graduate School of Frontier Sciences, The University of Tokyo:RIKEN Brain Science Institute:Intelligent Cooperation and Control
関連論文
- Statistical Mechanics for Neural Spike Data Analysis Using Log-Linear Model
- Analysis Method Combining Monte Carlo Simulation and Principal Component Analysis : Application to Sourlas Code(General)
- A PCA Approach to Sourlas Code Analysis
- Statistical Mechanical Study of Code-Division Multiple-Access Multiuser Detectors : Analysis of Replica Symmetric and One-Step Replica Symmetry Breaking Solutions(General)
- Statistical Mechanical Analysis of CDMA Multiuser Detectors : AT Stability and Entropy of the RS Solution, and 1RSB Solution
- Retrieval Property of Attractor Network with Synaptic Depression(General)
- Analysis of Ensemble Learning Using Simple Perceptrons Based on Online Learning Theory
- Residual Energies after Slow Quantum Annealing(General)
- Slow Dynamics Due to Singularities of Hierarchical Learning Machines
- On-Line Learning Dynamics of Multilayer Perceptrons with Unidentifiable Parameters
- Multiple Stability of a Sparsely Encoded Attractor Neural Network Model for the Inferior Temporal Cortex(General)
- Naive Mean Field Approximation for Image Restoration
- Statistical Mechanics of Mexican-Hat-Type Horizontal Connection
- Ensemble Learning of Linear Perceptrons : On-Line Learning Theory(General)
- Neural Network Model of Spatial Memory: Associative Recall of Maps
- Statistical Mechanics of Mutual Learning with a Latent Teacher(General)
- Statistical Mechanics for Neural Spike Data Analysis Using Log-Linear Model
- Slow Dynamics Due to Singularities of Hierarchical Learning Machines
- Analysis of Ensemble Learning Using Simple Perceptrons Based on Online Learning Theory