A Mathematical Analysis of the Backpropagation Algorithm in Artificial Neural Networks

Abstract

Any new mathematical concept in Machine Learning can often times prove difficult until it is approached in an algorithmic format, enabling manual step-by-step computations to be possible. This type of computation-based approach from first principles can greatly enhance our understanding of the subject. Backpropagation is very basic to Machine Learning; however, it may not be easy to fully comprehend how it functions. This article provides a detailed, step-by-step pedagogical exposition of the backpropagation algorithm for multilayer perceptrons (MLPs). Starting from first principles, we derive the necessary gradient expressions using the chain rule, illustrate the calculations manually on a small numerical example, and provide an annotated MATLAB implementation. The contribution is a self-contained tutorial that bridges the gap between mathematical formalism and executable code, enabling learners to trace forward and backward propagation without external references. Our main objective is to minimize the sum of squared errors involving the input values and the target values. Once this error is very close to or equal to zero, we conclude that the neural network has learned the given task. We demonstrate convergence of the sum-of-squares error over 10,000 iterations.