Learn Challenge: Training the Perceptron | Neural Network from Scratch

Before proceeding with training the perceptron, keep in mind that it uses the binary cross-entropy loss function discussed earlier. The final key concept before implementing backpropagation is the formula for the derivative of this loss function with respect to the output activations, $a^n$ . Below are the formulas for the loss function and its derivative:

\begin{aligned} L &= -(y \log(\hat{y}) + (1-y) \log(1 - \hat{y}))\\ da^n &= \frac {\hat{y} - y} {\hat{y}(1 - \hat{y})} \end{aligned}

where $a^n = \hat{y}$

To verify that the perceptron is training correctly, the fit() method also prints the average loss at each epoch. This is calculated by averaging the loss over all training examples in that epoch:

for epoch in range(epochs):
    loss = 0

    for i in range(training_data.shape[0]):
        loss += -(target * np.log(output) + (1 - target) * np.log(1 - output))

average_loss = loss[0, 0] / training_data.shape[0]
print(f'Loss at epoch {epoch + 1}: {average_loss:.3f}')

L = -\frac1N \sum_{i=1}^N (y_i \log(\hat{y}_i) + (1 - y_i) \log(1 - \hat{y}_i))

Finally, the formulas for computing gradients are as follows:

\begin{aligned} dz^l &= da^l \odot f'^l(z^l)\\ dW^l &= dz^l \cdot (a^{l-1})^T\\ db^l &= dz^l\\ da^{l-1} &= (W^l)^T \cdot dz^l \end{aligned}

The sample training data (X_train) along with the corresponding labels (y_train) are stored as NumPy arrays in the utils.py file. Additionally, instances of the activation functions are also defined there:

relu = ReLU()
sigmoid = Sigmoid()

Task

Swipe to start coding

Your goal is to complete the training process for a multilayer perceptron by implementing backpropagation and updating the model parameters.

Follow these steps carefully:

Implement the backward() method in the Layer class:
- Compute the following gradients:
  - dz: derivative of the loss with respect to the pre-activation values, using the derivative of the activation function;
  - d_weights: gradient of the loss with respect to the weights, calculated as the dot product of dz and the transposed input vector;
  - d_biases: gradient of the loss with respect to the biases, equal to dz;
  - da_prev: gradient of the loss with respect to the activations of the previous layer, obtained by multiplying the transposed weight matrix by dz.
- Update the weights and biases using the learning rate.
Complete the fit() method in the Perceptron class:
- Compute the model output by calling the forward() method;
- Calculate the loss using the cross-entropy formula;
- Compute $da^n$ — the derivative of the loss with respect to the output activations;
- Loop backward through the layers, performing backpropagation by calling each layer's backward() method.
Check the training behavior:
- If everything is implemented correctly, the loss should steadily decrease with each epoch when using a learning rate of 0.01.

Solution

Everything was clear?

Thanks for your feedback!

Section 2. Chapter 10

single

Ask AI

Ask anything or try one of the suggested questions to begin our chat

Awesome!

Completion rate improved to 4

Swipe to show menu

\begin{aligned} L &= -(y \log(\hat{y}) + (1-y) \log(1 - \hat{y}))\\ da^n &= \frac {\hat{y} - y} {\hat{y}(1 - \hat{y})} \end{aligned}