Learn Challenge: Training the Perceptron | Neural Network from Scratch

Before proceeding with training the perceptron, keep in mind that it uses the binary cross-entropy loss function discussed earlier. The final key concept before implementing backpropagation is the formula for the derivative of this loss function with respect to the output activations, $a^n$ . Below are the formulas for the loss function and its derivative:

\begin{aligned} L &= -(y \log(\hat{y}) + (1-y) \log(1 - \hat{y}))\\ da^n &= \frac {\hat{y} - y} {\hat{y}(1 - \hat{y})} \end{aligned}

where $a^n = \hat{y}$

To verify that the perceptron is training correctly, the fit() method also prints the average loss at each epoch. This is calculated by averaging the loss over all training examples in that epoch:

for epoch in range(epochs):
    loss = 0

    for i in range(training_data.shape[0]):
        loss += -(target * np.log(output) + (1 - target) * np.log(1 - output))

average_loss = loss[0, 0] / training_data.shape[0]
print(f'Loss at epoch {epoch + 1}: {average_loss:.3f}')

L = -\frac1N \sum_{i=1}^N (y_i \log(\hat{y}_i) + (1 - y_i) \log(1 - \hat{y}_i))

Finally, the formulas for computing gradients are as follows:

\begin{aligned} dz^l &= da^l \odot f'^l(z^l)\\ dW^l &= dz^l \cdot (a^{l-1})^T\\ db^l &= dz^l\\ da^{l-1} &= (W^l)^T \cdot dz^l \end{aligned}

The sample training data (X_train) along with the corresponding labels (y_train) are stored as NumPy arrays in the utils.py file. Additionally, instances of the activation functions are also defined there:

relu = ReLU()
sigmoid = Sigmoid()

Task

Swipe to start coding

Compute the following gradients: dz, d_weights, d_biases, and da_prev in the backward() method of the Layer class.
Compute the output of the model in the fit() method of the Perceptron class.
Compute da ( $da^n$ ) before the loop, which is the gradient of the loss with respect to output activations.
Compute da and perform backpropagation in the loop by calling the appropriate method for each of the layers.

If you implemented training correctly, given the learning rate of 0.01, the loss should steadily decrease with each epoch.

Solution

Everything was clear?

Thanks for your feedback!

Section 2. Chapter 10

single

Ask AI

Ask anything or try one of the suggested questions to begin our chat

Swipe to show menu

\begin{aligned} L &= -(y \log(\hat{y}) + (1-y) \log(1 - \hat{y}))\\ da^n &= \frac {\hat{y} - y} {\hat{y}(1 - \hat{y})} \end{aligned}

where $a^n = \hat{y}$

for epoch in range(epochs):
    loss = 0

    for i in range(training_data.shape[0]):
        loss += -(target * np.log(output) + (1 - target) * np.log(1 - output))

average_loss = loss[0, 0] / training_data.shape[0]
print(f'Loss at epoch {epoch + 1}: {average_loss:.3f}')

L = -\frac1N \sum_{i=1}^N (y_i \log(\hat{y}_i) + (1 - y_i) \log(1 - \hat{y}_i))

Finally, the formulas for computing gradients are as follows:

\begin{aligned} dz^l &= da^l \odot f'^l(z^l)\\ dW^l &= dz^l \cdot (a^{l-1})^T\\ db^l &= dz^l\\ da^{l-1} &= (W^l)^T \cdot dz^l \end{aligned}

relu = ReLU()
sigmoid = Sigmoid()

Task

Swipe to start coding

Compute the following gradients: dz, d_weights, d_biases, and da_prev in the backward() method of the Layer class.
Compute the output of the model in the fit() method of the Perceptron class.
Compute da ( $da^n$ ) before the loop, which is the gradient of the loss with respect to output activations.
Compute da and perform backpropagation in the loop by calling the appropriate method for each of the layers.

If you implemented training correctly, given the learning rate of 0.01, the loss should steadily decrease with each epoch.

Solution

Switch to desktop for real-world practiceContinue from where you are using one of the options below

Everything was clear?

Thanks for your feedback!

Section 2. Chapter 10

single

Challenge: Training the Perceptron

Solution

Awesome!

Challenge: Training the Perceptron

Solution

Awesome!