Weak and Strong Potentiality Control for Interpreting Multi-layered Neural Networks
摘要
This paper aims to show that neural learning has a strong bias toward simplification, which can be realized through prototype learning followed by non-prototype learning. Prototype learning seeks to make network configurations as simple as possible, not based on input patterns but rather on the given network resources. The subsequent non-prototype learning then adjusts these configurations to accommodate detailed information from input patterns. However, prototype learning is not always easily identified in actual learning because it is deeply hidden within the surface network configurations. To detect the prototype, we previously proposed structural potentiality consumption to eliminate unnecessary information in the search for the prototype. However, this structural potentiality, which is defined for connection weights, was too strong and ultimately tended to consume all potentiality, even destroying necessary information. To moderate the excessive information reduction caused by structural potentiality consumption, we propose weak potentiality consumption, which is based on more general properties such as average values. This weak potentiality consumption allows connection weights to behave more flexibly, thereby mitigating the strong force of potentiality reduction. The method was applied to an artificial dataset with a limited number of inputs and both linear and non-linear relations for easier interpretation. The results showed that weak potentiality consumption effectively moderated strong potentiality consumption, leading to better generalization. The prototype was detected in the early stage of learning, and during subsequent non-prototype learning, a phase transition occurred, transforming the connection weights over the course of learning.