Artificial Neural Networks (ANNs) are the backbone of modern Generative AI. They are designed to recognize patterns, learn representations, and generate data that mimics real-world distributions. You'll gain a concise and comprehensive overview of ANNs, emphasizing their significance in Generative AI.

Structure of Neural Networks

Neurons and Layers

A neural network consists of interconnected units called neurons, which are organized into layers:

• Input Layer: receives raw data (e.g., images, text, numerical inputs);

• Hidden Layers: process and transform data using weighted connections;

• Output Layer: produces predictions or classifications.

Each neuron applies a weighted sum to its inputs and passes the result through an activation function:

where:

• $x_i$ are input values;

• $\omega_i$ are weights;

• $b$ is the bias term;

• $z$ is the weighted sum passed to the activation function.

Activation Functions

Activation functions introduce non-linearity, enabling networks to learn complex patterns. Common activation functions include:

• Sigmoid, used for probabilities: $\sigma(z)=\dfrac{1}{1+e^{-z}}$

• ReLU (Rectified Linear Unit), commonly used in deep networks: $f(z)=\max(0,z)$

• Tanh, useful for zero-centered outputs: $\tanh(z)=\dfrac{e^z-e^{-z}}{e^z+e^{-z}}$

Forward and Backward Propagation

Forward Propagation

Forward propagation refers to passing inputs through the network to compute the output. Each neuron computes:

where $f(z)$ is the activation function.

Backpropagation and Gradient Descent

To improve predictions, ANNs adjust weights using backpropagation, which minimizes error using gradient descent. The weight update rule in gradient descent is:

where:

• $\eta$ is the learning rate;

• $L$ is the loss function;

• $\frac{\partial L}{\partial \omega_i}$ is the gradient of the loss with respect to $\omega_i$.

Loss Functions and the Training Process

Loss Functions

Loss functions measure the difference between predicted and actual values. Common loss functions include:

• Mean Squared Error (MSE) (for regression):

• Cross-Entropy Loss (for classification):

where:

• $y_i$ is the true label;

• $\hat{y}_i$ is the predicted probability.

Training Process

1. Initialize weights randomly;

2. Perform forward propagation to compute predictions;

3. Compute the loss using the chosen loss function;

4. Use backpropagation to compute weight updates;

5. Update weights using gradient descent;

6. Repeat for multiple epochs until the network converges.

The Universal Approximation Theorem and Deep Learning

Universal Approximation Theorem

The Universal Approximation Theorem states that a neural network with at least one hidden layer can approximate any continuous function, given sufficient neurons and proper weights. This justifies why ANNs can model highly complex relationships.

Deep Learning and Its Significance

Deep Learning extends ANNs by adding many hidden layers, allowing them to:

• Extract hierarchical features (useful in image processing and NLP);

• Model complex probability distributions (critical for Generative AI);

• Learn without manual feature engineering (as seen in self-supervised learning).

Conclusion

This chapter introduced the core principles of ANNs, emphasizing their structure, learning process, and significance in deep learning. These concepts lay the foundation for advanced Generative AI techniques like GANs and VAEs, which rely on neural networks to generate realistic data.