## General Approach

In **forward propagation**, each layer **l** takes the outputs from the previous layer, al-1, as inputs and computes its own outputs. Therefore, the `forward()` method of the `Layer` class takes the **vector of previous outputs** as its only parameter, while the rest of the needed information is stored within the class. 

In **backward propagation**, each layer **l** only needs dal to compute the respective gradients and return dal-1, so the `backward()` method takes the dal vector as its parameter. The rest of the required information is already stored in the `Layer` class.

## Activation Functions Derivatives

Since derivatives of activation functions are needed for backpropagation, activation functions like **ReLU** and **sigmoid** should be structured as **classes** instead of standalone functions. This allows us to define both:
  
1. **The activation function itself** (implemented via the `__call__()` method), allowing it to be applied to an input in the format: `activation(z)`;  
2. **Its derivative** (implemented via the `derivative()` method), which is used for backpropagation in the format: `activation.derivative(z)`.  

By structuring activation functions as **objects**, we can easily pass them to the `Layer` class and use them dynamically.

### ReLu

The derivative of **ReLU activation function** is as follows, where zi is an element of vector of pre-activations `z`:

```python
class ReLU:
    def __call__(self, z):
        return np.maximum(0, z)

    def derivative(self, z):
        return (z > 0).astype(float)
```

### Sigmoid

The derivative of **sigmoid activation function** is as follows:

```python
class Sigmoid:
    def __call__(self, x):
        return 1 / (1 + np.exp(-z))

    def derivative(self, z):
        sig = self(z)
        return sig * (1 - sig)
```

For both of these activation functions, we apply them to the **entire vector** `z`, and the same goes for their derivatives. **NumPy** internally applies the operation to each element of the vector. For example, if the vector `z` contains **3** elements, the derivation is as follows:

## The backward() Method

The `backward()` method is responsible for **computing the gradients** using the formulas below:

al-1 and zl are stored as the `inputs` and `outputs` attributes in the `Layer` class, respectively. The activation function **f** is stores as the `activation` attribute.

Once all the required gradients are computed, the **weights** and **biases** can be updated since they are no longer needed for further computation:

Therefore, `learning_rate` (η) is another parameter of this method.

```python
def backward(self, da, learning_rate):
    dz = ...
    d_weights = ...
    d_biases = ...
    da_prev = ...

    self.weights -= learning_rate * d_weights
    self.biases -= learning_rate * d_biases

    return da_prev
```

Neural networks are powerful algorithms inspired by the structure of the human brain that are used to solve complex machine learning problems. You will build your own Neural Network from scratch to understand how it works. After this course, you will be able to create neural networks for solving classification and regression problems using the scikit-learn library.

First, we will discuss what a neural network is and how it works. And also consider the scope of its application.

Next, we will try to build our own neural network and see how efficiently it copes with learning. We will also consider a ready-made solution from the scikit-learn library.

Finally, we will give you some additional useful information on how to understand which model to use and what types of neural networks there are. To complete the course, you will be tested on your acquired knowledge.

Introduction to Neural Networks

Backpropagation Implementation

General Approach

Activation Functions Derivatives

ReLu

Sigmoid

The backward() Method