Contenido del Curso

Introduction to Neural Networks

Backward Propagation

Backward propagation (backprop) is the process of computing how the loss function changes with respect to each parameter in the network. The objective is to update the parameters in the direction that reduces the loss.

To achieve this, we use the gradient descent algorithm and compute the derivatives of the loss with respect to each layer's pre-activation values (raw output values before applying the activation function) and propagate them backward.

Each layer contributes to the final prediction, so the gradients must be computed in a structured manner:

Perform forward propagation;
Compute the derivative of the loss with respect to the output pre-activation;
Propagate this derivative backward through the layers using the chain rule;
Compute gradients for weights and biases to update them.

Note

Gradients represent the rate of change of a function with respect to its inputs, meaning they are its derivatives. They indicate how much a small change in weights, biases, or activations affects the loss function, guiding the model's learning process through gradient descent.

Notation

To make explanation clearer, let's use the following notation:

$W^l$ is the weight matrix of layer $l$ ;
$b^l$ is the vector of biases of layer $l$ ;
$z^l$ is the vector of pre-activations of layer $l$ ;
$a^l$ is the vector of activation of layer $l$ ;

Therefore, setting $a^0$ to $x$ (the inputs), forward propagation in a perceptron with n layers can be described as the following sequence of operations:

\begin{aligned} a^0 &= x, & &... & &...\\ z^1 &= W^1 a^0 + b^1, & z^l &= W^l a^{l-1} + b^l, & z^n &= W^n a^{n-1} + b^n,\\ a^1 &= f^1(z^1), & a^l &= f^l(z^l), & a^n &= f^n(z^n),\\ &... & &... & \hat y &= a^n. \end{aligned}

To describe backpropagation mathematically, we introduce the following notations:

$da^l$ : derivative of the loss with respect to the activations at layer $l$ ;
$dz^l$ : derivative of the loss with respect to the pre-activations at layer $l$ (before applying the activation function);
$dW^l$ : derivative of the loss with respect to the weights at layer $l$ ;
$db^l$ : derivative of the loss with respect to the biases at layer $l$ .

Computing Gradients for the Output Layer

At the final layer $n$ , we first compute the gradient of the loss with respect to the activations of the output layer, $da^n$ . Next, using the chain rule, we compute the gradient of the loss with respect to output layer's pre-activations:

dz^n = da^n \odot f'^n(z^n)

Note

The $\odot$ symbol represents element-wise multiplication. Since we are working with vectors and matrices, the usual multiplication symbol $\cdot$ represents the dot product instead. $f'^n$ is the derivative of the activation function of the output layer.

This quantity represents how sensitive the loss function is to changes in the output layer's pre-activation.

Once we have $\text d z^n$ , we compute gradients for the weights and biases:

\begin{aligned} dW^n &= dz^n \cdot (a^{n-1})^T\\ db^n &= dz^n \end{aligned}

where $(a^{n-1})^T$ is the transposed vector of activation from the previous layer. Given that the original vector is a $n_{neurons} \times 1$ vector, the transposed vector is $1 \times n_{neurons}$ .

To propagate this backward, we calculate the derivative of the loss with respect to the activations of the previous layer:

da^{n-1} = (W^n)^T \cdot dz^n

Propagating Gradients to the Hidden Layers

For each hidden layer $l$ the procedure is the same. Given $da^l$ :

Compute the derivative of the loss with respect to the pre-activations;
Compute the gradients for the weights and biases;
Compute $da^{l-1}$ to propagate the derivative backward.

\begin{aligned} dz^l &= da^l \odot f'^l(z^l)\\ dW^l &= dz^l \cdot (a^{l-1})^T\\ db^l &= dz^l\\ da^{l-1} &= (W^l)^T \cdot dz^l \end{aligned}

This step repeats until we reach the input layer.

Updating Weights and Biases

Once we have computed the gradients for all layers, we update the weights and biases using gradient descent:

\begin{aligned} W^l &= W^l - \alpha \cdot dW^l\\ b^l &= b^l - \alpha \cdot db^l \end{aligned}

where $\alpha$ is the learning rate, which controls how much we adjust the parameters.

¿Todo estuvo claro?

¡Gracias por tus comentarios!

Sección 2. Capítulo 7

Pregunte a AI

Pregunte lo que quiera o pruebe una de las preguntas sugeridas para comenzar nuestra charla

Contenido del Curso

Introduction to Neural Networks

Backward Propagation

Each layer contributes to the final prediction, so the gradients must be computed in a structured manner:

Perform forward propagation;
Compute the derivative of the loss with respect to the output pre-activation;
Propagate this derivative backward through the layers using the chain rule;
Compute gradients for weights and biases to update them.

Note

Notation

To make explanation clearer, let's use the following notation:

$W^l$ is the weight matrix of layer $l$ ;
$b^l$ is the vector of biases of layer $l$ ;
$z^l$ is the vector of pre-activations of layer $l$ ;
$a^l$ is the vector of activation of layer $l$ ;

Therefore, setting $a^0$ to $x$ (the inputs), forward propagation in a perceptron with n layers can be described as the following sequence of operations:

\begin{aligned} a^0 &= x, & &... & &...\\ z^1 &= W^1 a^0 + b^1, & z^l &= W^l a^{l-1} + b^l, & z^n &= W^n a^{n-1} + b^n,\\ a^1 &= f^1(z^1), & a^l &= f^l(z^l), & a^n &= f^n(z^n),\\ &... & &... & \hat y &= a^n. \end{aligned}

To describe backpropagation mathematically, we introduce the following notations:

$da^l$ : derivative of the loss with respect to the activations at layer $l$ ;
$dz^l$ : derivative of the loss with respect to the pre-activations at layer $l$ (before applying the activation function);
$dW^l$ : derivative of the loss with respect to the weights at layer $l$ ;
$db^l$ : derivative of the loss with respect to the biases at layer $l$ .

Computing Gradients for the Output Layer

dz^n = da^n \odot f'^n(z^n)

Note

This quantity represents how sensitive the loss function is to changes in the output layer's pre-activation.

Once we have $\text d z^n$ , we compute gradients for the weights and biases:

\begin{aligned} dW^n &= dz^n \cdot (a^{n-1})^T\\ db^n &= dz^n \end{aligned}

To propagate this backward, we calculate the derivative of the loss with respect to the activations of the previous layer:

da^{n-1} = (W^n)^T \cdot dz^n

Propagating Gradients to the Hidden Layers

For each hidden layer $l$ the procedure is the same. Given $da^l$ :

Compute the derivative of the loss with respect to the pre-activations;
Compute the gradients for the weights and biases;
Compute $da^{l-1}$ to propagate the derivative backward.

\begin{aligned} dz^l &= da^l \odot f'^l(z^l)\\ dW^l &= dz^l \cdot (a^{l-1})^T\\ db^l &= dz^l\\ da^{l-1} &= (W^l)^T \cdot dz^l \end{aligned}

This step repeats until we reach the input layer.

Updating Weights and Biases

Once we have computed the gradients for all layers, we update the weights and biases using gradient descent:

\begin{aligned} W^l &= W^l - \alpha \cdot dW^l\\ b^l &= b^l - \alpha \cdot db^l \end{aligned}

where $\alpha$ is the learning rate, which controls how much we adjust the parameters.

¿Todo estuvo claro?

¡Gracias por tus comentarios!

Sección 2. Capítulo 7