3.1 Neural networks
3.1.1 Backpropagation
In the vernacular of the machine learning literature the aim of the optimiza-tion procedure is to train the model to perform optimally on the regression, reconstruction or classification task at hand. Training the model requires the computation of the total derivative in equation 3.13. This is also where the
48 Deep learning theory Chapter 3
biological metaphor breaks down, as the brain is probably not employing gra-dient descent.
Backpropagation of errors by automatic differentiation, first described by Linnainmaa [28], is a method of computing the partial derivatives required to go from the gradient of the loss to individual parameter derivatives. Con-ceptually we wish to describe the slope of the error in terms of our model parameters, but having multiple layers complicate this somewhat.
The backpropagation algorithm begins with computing the total loss, here exemplified with the squared error function,
E=C(y,ˆ y) = 1 2n
∑
n
∑
j
(yˆnj−ynj)2. (3.14) The factor one half is included for practical reasons to cancel the exponent under differentiation. As the gradient is multiplied by the learning rateη, this is ineffectual on the training itself.
The sums over nand jenumerate the number of samples, and output di-mensions respectively. Finding the update for the parameters then starts with taking the derivative of equation 3.14 w.r.t the model outputyj =a[jl]
∂E
∂yj
=yˆj−a[jl]. (3.15) We have dropped the data index, as the differentiation is independent under the choice of data. In practice the derivative of each sample in the batch is averaged together for the gradient update of each parameter.
The activation function, f, has classically been the logistic sigmoid func-tion, but during the last decade the machine learning community has largely shifted to using the Rectified linear unit (ReLU). This shift was especially ap-parent after the success of Krizhevsky et al. [29]. In this section we then ex-emplify the backpropagation algorithm with a network with ReLU activation.
The ReLU function is defined in such a way that it is zero for all negative in-puts and the identity otherwise, i.e.
ReLU(x) = f(x) =
(x, ifx >0
0, otherwise. (3.16)
The ReLU is obviously monotonic and its derivative can be approximated with the Heaviside step-function which we denote with H(x) and is mathemati-cally expressed as
H(x) = f0(x) =
(1, ifx >0
0, otherwise. (3.17)
Section 3.1 Neural networks 49
Common to most neural network activations the computation of the deriva-tive is very light-weight. In the case of the ReLU function the computation of the derivative uses the mask needed to compute the activation itself, requiring no extra computing resources.
It is important to note that the cost and activation functions introduced in equations 3.14, 3.16 and 3.17 are not a be-all-end-all solution, but they are chosen for their ubiquitous nature in modern machine learning.
Returning to the optimization problem we start to unravel the backpropa-gation algorithm. We use equation 3.15 to find the derivatives in terms of the last parameters, i.e. Wij[n]andb[jn]
∂E
∂Wij[n]
= ∂E
∂yj
∂yj
∂z[jn]
∂z[jn]
∂Wij[n], (3.18)
= ∂E
∂yj
o0(z[jn]) 1
∂Wij[n]
a[in−1]Wij[n]+b[jn]
, (3.19)
= (yˆj−yj)o0(z[jn])a[in−1]. (3.20) The differentiation of the error w.r.t tobjcan be similarly derived to be
∂E
∂b[jn]
= (yˆj−yj)o0(a[jn]). (3.21) Repeating this procedure layer by layer is the process that defines the back-propagation algorithm. From equations 3.20 and 3.21 we discern a recursive pattern in the derivatives moving to the next layer. Before writing out the full backpropagation term we will introduce some more notation that makes bridging the gap to an implementation considerably simpler. From the repeat-ing structure in the aforementioned equations we define the first operation needed for backpropagation,
δnj = (yˆj−yj)o0(z[jn]). (3.22) Note that this is an element-wise Hadamard product and not an implicit sum-mation, expressed by the subscript index in ˆδnj. The element-wise product of two matrices or vectors is denoted as
a◦b. (3.23)
This short-hand lets us define equations 3.20 and 3.21 in a more compact way
50 Deep learning theory Chapter 3
∂E
∂w[ijn]
=δnja[in−1], (3.24)
∂E
∂b[jn]
=δnj. (3.25)
From the iterative nature of how we construct the forward pass we see that the last derivative in the chain for each layer, i.e. those in terms of the weights and biases, have the same form
∂z[jl]
∂w[ijl]
=a[il−1], (3.26)
∂z[jl]
∂b[jl]
=1. (3.27)
These derivatives together with a general expression for the recurrent term δlj are then the pieces we need to compute the parameter update rules. By summing up over the connecting nodes, k, to the layer, l, of interest δlj can be expressed as
δlj =
∑
k
∂E
∂a[kl+1]
∂a[kl+1]
∂z[kl+1]
∂z[kl+1]
∂a[jl]
∂a[jl]
∂z[jl], (3.28) δlj =
∑
k
δl+1∂z[kl+1]
∂a[jl]
∂a[jl]
∂z[jl]
. (3.29)
From the definitions of the z[jl] and a[jl] terms we can then compute the last derivatives. These are then inserted back into 3.29, giving a final expression forδjl,
δlj =
∑
k
δlk+1w[jkl+1]f0(z[jl]). (3.30) Finally the weight and bias update rules can then be written as
Section 3.1 Neural networks 51
∂E
∂w[jml]
=δlja[ml−1], (3.31)
∂E
∂b[jl]
=δlj. (3.32)
To finalize the discussion on the algorithm we illustrate how backpropagation might be implemented in algorithm 2.
Algorithm 2: Backpropagation of errors in a fully connected neural network for a single sample x.
Data: Iterablesa[l] z[l]W[l] b[l]∀ l ∈ [1, 2, . . . , n] Input: ∂E∂y, o0(z[n]), f0(·)
Result: Two iterables of the derivatives ∂E
∂wij[l] and ∂E
∂b[l]j
Initialization;
δnj ← ∂E∂y ◦o0(z[n]); Compute derivatives;
forl ∈ [n−1, . . . , 1]do
∂E
∂w[l]jm ←δˆlj+1a[ml];
∂E
∂w[l]jm ←δˆlj+1;
δlj+1←∑kδlk+1w[jkl+1]f0(z[jl]) return ∂E
∂wij[l] and ∂E
∂b[l]j
The backward propagation framework is highly generalizable to variations of activation functions and network architectures. The two major advance-ments in the theory of ANNs are both predicated on being fully trainable by the backpropagation of errors. Before we consider these improvements made by the introduction of recurrent neural networks (RNN) and convolutional neural networks (CNN), we remark again on the strength of this algorithm.
We are not only free to chose the activation function from a wide selection, the backpropagation algorithm also makes no assumptions on the transformation that constructszj. As long as it is once differentiable we are free to choose a different approach. This flexibility of the framework is part of the resurgent success of deep learning in the last decade.
52 Deep learning theory Chapter 3