Research ArticleOpen AccessGoogle Scholar indexed
A New Weight Initialization Method Using Cauchy’s Inequality Based on Sensitivity Analysis
- 1
- 2
Journal of Intelligent Learning Systems and Applications·Volume 03 (2011)·Pages 242–248·Published 29 November 2011·DOI10.4236/jilsa.2011.34027
Copy link · social · email
Abstract
In this paper, an efficient weight initialization method is proposed using Cauchy’s inequality based on sensitivity analy- sis to improve the convergence speed in single hidden layer feedforward neural networks. The proposed method ensures that the outputs of hidden neurons are in the active region which increases the rate of convergence. Also the weights are learned by minimizing the sum of squared errors and obtained by solving linear system of equations. The proposed method is simulated on various problems. In all the problems the number of epochs and time required for the proposed method is found to be minimum compared with other weight initialization methods.
KeywordsWeight InitializationBackpropagationFeedforward Neural NetworkCauchy’s InequalityLinear System of Equations
- R. Battiti, “First and Second Order Methods for Learning: Between Steepest Descent and Newton’s Method,” Neural Computation, Vol. 4, No. 2, 1992, pp. 141-166. doi:10.1162/neco.1992.4.2.141
- W. L. Buntine and A. S. Weigend, “Computing Second Derivatives in Feedforward Networks: A Review,” IEEE Transactions on Neural Networks, Vol. 5, No. 3, 1994, pp. 480-488. doi:10.1109/72.286919
- G. B. Orr and T. K. Leen, “Using Curvature Information for Fast Stochastic Search,” Neural Information Processing Systems, Vol. 9, 1996, pp. 606-612.
- N. N. Schrusolph, “Fast Curvature Matrix-Vector Products for Second Order Gradient Descent,” Neural Computation, Vol. 14, No. 7, 2002, pp. 1723-1738. doi:10.1162/08997660260028683
- F. Biegler-Konig and F. Barnmann, “A Learning Agorithm for Multilayered Neural Networks Based on Linear Least Squares Problems,” Neural Networks, Vol. 6, No. 1, 1993, pp. 127-131. doi:10.1016/S0893-6080(05)80077-2
- Y. F. Yam and T. W. S. Chow, “Determining Initial Weights of Feedforward Neural Networks Based on Least Squares Method,” Neural Processing Letters, Vol. 2, No. 2, 1995, pp. 13-17. doi:10.1007/BF02312350
- Y. F. Yam, T. W. S. Chow and C. T. Leung, “A New Method in Determining the Initial Weights of Feedforward Neural Networks for Training Enhancement,” Neurocomputing, Vol. 16, No. 1, 1997, pp. 23-32. doi:10.1016/S0925-2312(96)00058-6
- G. P .Drago amd S. Ridella, “Statiscally Controlled Activation Weight Initialization (SCAWI),” IEEE Transactions on Neural Networks, Vol. 3, No. 4, 1992, pp. 899-905. doi:10.1109/72.143378
- D. Nguyen and B. Widrow, “Improving the Learning Speed of 2-Layer Neural Networks by Choosing Initial Values of the Adaptive Weights,” Proceedings of the International Joint Conference on Neural Networks, San Diego, Vol. 3, 17-21 June 1990, pp. 21-26. doi:10.1109/IJCNN.1990.137819
- H. Shimodaira, “A Weight Value Initialization Method for Improved Learning Performance of the Back Propagation Algorithm in Neural Networks,” Proceedings of the sixth Internation Conference on Tools with Artificial Intelligence, New Orleans, 6-9 November 1994, pp. 672-675. doi:10.1109/TAI.1994.346429
- M. Lehtokangas, J. Saarinen, K. Kaski and P. Huuhtanen, “Initializing Weights of a Multilayer Perceptron Network by Using the Orthogonal Least Squares Problem,” Neural Computation, Vol. 7, No. 5, 1995, pp. 982-999. doi:10.1162/neco.1995.7.5.982