An Optimization of Neural Network Hyper-Parameter to Increase Its Performance
- 1 Cranbrook Kingswood Schools, Bloomfield Hills, MI, USA
Abstract
With the boost of artificial intelligence, the study of neural network intrigues scientists. Artificial neural network, which was first designed theoretically in 1943 based on understanding of human brains, demonstrated impressing computational and learning capabilities. In this paper, we investigated the neural network’s learning capability by using a feed-forward neural network to recognize human’s digit hand-writing. Controlled experiments were executed by changing the input values of different parameters, such as learning rates and hidden layer units. After investigating upon the effects of each parameter on the overall learning performance of the neural network, we concluded that, when an intermediate value of one given parameter was implemented, the neural network achieved the highest learning efficiency, and potential problems like over-fitting would be prevented.
- Thomas, M.S. and McClelland, J.L. (2008) Connectionist Models of Cognition. Cambridge handbook of Computational Cognitive Modelling, 23-58. https://doi.org/10.1017/CBO9780511816772.005
- McCulloch, W.S. and Pitts, W. (1943) A Logical Calculus of the Ideas Immanent in Nervous Activity. The Bulletin of Mathematical Biophysics, 5, 115-133. https://doi.org/10.1007/BF02478259
- Preparata, F.P. and Shamos, M.I. (1985) Computational Geometry: An Introduction. Springer, New York, 1-35. https://doi.org/10.1007/978-1-4612-1098-6
- Werbos, P.J. (1974) Beyond Regression: New Tools for Prediction and Analysis in the Behavioral Sciences. Doctoral Dissertation, Harvard University, Cambridge.
- Khosravi, H. and Kabir, E. (2007) Introducing a Very Large Dataset of Handwritten Farsi Digits and a Study on Their Varieties. Pattern Recognition Letters, 28, 1133-1141. https://doi.org/10.1016/j.patrec.2006.12.022
- Jin, W., Li, Z.J., Wei, L.S. and Zhen, H. (2000) The Improvements of BP Neural Network Learning Algorithm. 2000 5th International Conference on Signal Processing Proceedings, Beijing, 21-25 August 2000, 1647-1649.
- Attoh-Okine, N.O. (1999) Analysis of Learning Rate and Momentum Term in Backpropagation Neural Network Algorithm Trained to Predict Pavement Performance. Advances in Engineering Software, 30, 291-302. https://doi.org/10.1016/S0965-9978(98)00071-4
- Caruana, R., Lawrence, S. and Giles, C.L. (2001) Overfitting in Neural Nets: Backpropagation, Conjugate Gradient, and Early Stopping. Neural Information Processing Systems, 402-408.
- Tetko, I.V., Livingstone, D.J. and Luik, A.I. (1995) Neural Network Studies. 1. Comparison of Overfitting and Overtraining. Journal of Chemical Information and Computer Sciences, 35, 826-833. https://doi.org/10.1021/ci00027a006
- Krogh, A. and Hertz, J.A. (1992) A Simple Weight Decay Can Improve Generalization. Neural Information Processing Systems, 950-957.
- Baum, E.B. and Haussler, D. (1989) What Size Net Gives Valid Generalization? Neural Information Processing Systems, 81-90.
- Bishop, C.M. (2006) Pattern Recognition and Machine Learning. Springer, New York.
- Salakhutdinov, R., Mnih, A. and Hinton, G. (2007) Restricted Boltzmann Machines for Collaborative Filtering. Proceedings of the 24th International Conference on Machine Learning, Corvalis, 20-24 June 2007, 791-798. https://doi.org/10.1145/1273496.1273596