Influence of Digital Fluctuations on Behavior of Neural Networks
Igor V. Netay
Igor V. Netay, JSRPC Kryptonite and Intitute for Information Transmission Problems of Russian Academy of Sciences, Moscow, Russia.
Manuscript received on 18 November 2022 | Revised Manuscript received on 25 November 2022 | Manuscript Accepted on 15 December 2022 | Manuscript published on 30 December 2022 | PP: 1-7 | Volume-3 Issue-1, December 2022 | Retrieval Number: 100.1/ijainn.A1061123122 | DOI: 10.54105/ijainn.A1061.123122
Open Access | Ethics and Policies | Cite | Mendeley | Indexing and Abstracting
© The Authors. Published by Lattice Science Publication (LSP). This is an open access article under the CC-BY-NC-ND license (http://creativecommons.org/licenses/by-nc-nd/4.0/)
Abstract: This paper deals with effect of digital noise to numerical stability of neural networks. Digital noise arises from the inexactness of floating point values operations. Accumulated errors finally lead to the loss of significance. Experiments show that more redundant networks have higher noise influence. This effect is tested in both model and real world samples. As a result, one should exclude all the networks results from the beginning of fluctuations. Results of experiments allow us to hypothesize that minimal values of loss function preserving significance were achieved for the networks of size close to the complexity of the dataset. So, it is a reason to choose sizes of network layers in accordance with complexity of particular datasets and not universally for an architecture and general problem statement without relation to data. In the case of fine tuning this suggests that pruning of network layers can improve result accuracy and reliability of prediction due to decrease of numerical noise influence. Results of this article are based on analysis of numerical experiments with train of more than 50000 neural networks for thousands epochs for each network. Almost all the networks begin to fluctuate.
Keywords: Neural Network, Numerical Stability, Digital Noise, Digital Fluctuations, Fine-Tuning.
Scope of the Article: Neural Networks