Gradient Descent

Gradient Descent is a fundamental optimization algorithm in machine learning used to minimize functions. It iteratively adjusts model parameters in the direction of steepest descent, allowing models to learn from data efficiently.

Understanding Gradients

The gradient of a function represents the direction and steepness of the function at a given point. It tells us which way to move to minimize the function.

For a simple function:

J(\theta) = \theta^2

The derivative (gradient) is:

\nabla J(\theta) = \frac{d}{d \theta}\left(\theta^2\right)= 2\theta

This means that for any value of $θ$ , the gradient tells us how to adjust $θ$ to descend toward the minimum.

Gradient Descent Formula

The weight update rule is:

\theta \larr \theta - \alpha \nabla J(\theta)

Where:

$\theta$ - model parameter;
$\alpha$ - learning rate (step size);
$\nabla J(\theta)$ - gradient of the function we're aiming to minimize.

For our function:

\theta_{\text{new}} = \theta_{\text{old}} - \alpha\left(2\theta_{old}\right)

This means we update $θ$ iteratively by subtracting the scaled gradient.

Stepwise Movement – A Visual

Example with start values: $\theta = 3$ , $\alpha = 0.3$

$\theta_1 = 3 - 0.3(2 \times 3) = 3 - 1.8 = 1.2;$
$\theta_2 = 1.2 - 0.3(2 \times 1.2) = 1.2 - 0.72 = 0.48;$
$\theta_3 = 0.48 - 0.3(2\times0.48) = 0.48 - 0.288 = 0.192;$
$\theta_4 = 0.192 - 0.3(2 \times 0.192) = 0.192 - 0.115 = 0.077.$

After a few iterations, we move toward $θ=0$ , the minimum.

Learning Rate – Choosing α Wisely

Too large $\alpha$ - overshoots, never converges;
Too small $\alpha$ - converges too slowly;
Optimal $\alpha$ - balances speed & accuracy.

When Does Gradient Descent Stop?

Gradient descent stops when:

\nabla J (\theta) \approx 0

This means that further updates are insignificant and we've found a minimum.

1. What is the primary goal of gradient descent?

2. What happens if the learning rate $α$ is too large?

3. If the gradient $∇J(θ)$ is zero, what does this mean?

What is the primary goal of gradient descent?

Select the correct answer

To maximize a function.

To minimize a function.

To find random values for parameters.

To increase functions over time.

What happens if the learning rate $α$ is too large?

Select the correct answer

The algorithm converges instantly.

The algorithm moves too slowly.

The algorithm oscillates and never converges.

The gradient becomes zero.

If the gradient $∇J(θ)$ is zero, what does this mean?

Select the correct answer

The function is increasing.

The function is at its minimum or maximum.

The learning rate becomes infinite.

The model stops learning.

Tutto è chiaro?

Come possiamo migliorarlo?

Grazie per i tuoi commenti!

Sezione 3. Capitolo 9

Chieda ad AI

Chieda pure quello che desidera o provi una delle domande suggerite per iniziare la nostra conversazione

Awesome!

Completion rate improved to 1.89

Gradient Descent

Scorri per mostrare il menu

Understanding Gradients

The gradient of a function represents the direction and steepness of the function at a given point. It tells us which way to move to minimize the function.

For a simple function:

J(\theta) = \theta^2

The derivative (gradient) is:

\nabla J(\theta) = \frac{d}{d \theta}\left(\theta^2\right)= 2\theta

This means that for any value of $θ$ , the gradient tells us how to adjust $θ$ to descend toward the minimum.

Gradient Descent Formula

The weight update rule is:

\theta \larr \theta - \alpha \nabla J(\theta)

Where:

$\theta$ - model parameter;
$\alpha$ - learning rate (step size);
$\nabla J(\theta)$ - gradient of the function we're aiming to minimize.

For our function:

\theta_{\text{new}} = \theta_{\text{old}} - \alpha\left(2\theta_{old}\right)

This means we update $θ$ iteratively by subtracting the scaled gradient.

Stepwise Movement – A Visual

Example with start values: $\theta = 3$ , $\alpha = 0.3$

$\theta_1 = 3 - 0.3(2 \times 3) = 3 - 1.8 = 1.2;$
$\theta_2 = 1.2 - 0.3(2 \times 1.2) = 1.2 - 0.72 = 0.48;$
$\theta_3 = 0.48 - 0.3(2\times0.48) = 0.48 - 0.288 = 0.192;$
$\theta_4 = 0.192 - 0.3(2 \times 0.192) = 0.192 - 0.115 = 0.077.$

After a few iterations, we move toward $θ=0$ , the minimum.

Learning Rate – Choosing α Wisely

Too large $\alpha$ - overshoots, never converges;
Too small $\alpha$ - converges too slowly;
Optimal $\alpha$ - balances speed & accuracy.

When Does Gradient Descent Stop?

Gradient descent stops when:

\nabla J (\theta) \approx 0

This means that further updates are insignificant and we've found a minimum.

1. What is the primary goal of gradient descent?

2. What happens if the learning rate $α$ is too large?

3. If the gradient $∇J(θ)$ is zero, what does this mean?

What is the primary goal of gradient descent?

Select the correct answer

To maximize a function.

To minimize a function.

To find random values for parameters.

To increase functions over time.

What happens if the learning rate $α$ is too large?

Select the correct answer

The algorithm converges instantly.

The algorithm moves too slowly.

The algorithm oscillates and never converges.

The gradient becomes zero.

If the gradient $∇J(θ)$ is zero, what does this mean?

Select the correct answer

The function is increasing.

The function is at its minimum or maximum.

The learning rate becomes infinite.

The model stops learning.

Tutto è chiaro?

Come possiamo migliorarlo?

Grazie per i tuoi commenti!

Sezione 3. Capitolo 9

Gradient Descent

Understanding Gradients

Gradient Descent Formula

Stepwise Movement – A Visual

Learning Rate – Choosing α Wisely

When Does Gradient Descent Stop?

1. What is the primary goal of gradient descent?

2. What happens if the learning rate ααα is too large?

3. If the gradient ∇J(θ)∇J(θ)∇J(θ) is zero, what does this mean?

Awesome!

Gradient Descent

Understanding Gradients

Gradient Descent Formula

Stepwise Movement – A Visual

Learning Rate – Choosing α Wisely

When Does Gradient Descent Stop?

1. What is the primary goal of gradient descent?

2. What happens if the learning rate ααα is too large?

3. If the gradient ∇J(θ)∇J(θ)∇J(θ) is zero, what does this mean?

2. What happens if the learning rate $α$ is too large?

3. If the gradient $∇J(θ)$ is zero, what does this mean?

2. What happens if the learning rate $α$ is too large?

3. If the gradient $∇J(θ)$ is zero, what does this mean?