Notice: This page requires JavaScript to function properly.
Please enable JavaScript in your browser settings or update your browser.Lære Gradient Descent | Mathematical Analysis
Mathematics for Data Science

bookGradient Descent

Gradient Descent is a fundamental optimization algorithm in machine learning used to minimize functions. It iteratively adjusts model parameters in the direction of steepest descent, allowing models to learn from data efficiently.

Understanding Gradients

The gradient of a function represents the direction and steepness of the function at a given point. It tells us which way to move to minimize the function.

For a simple function:

J(θ)=θ2J(\theta) = \theta^2

The derivative (gradient) is:

∇J(θ)=ddθ(θ2)=2θ\nabla J(\theta) = \frac{d}{d \theta}\left(\theta^2\right)= 2\theta

This means that for any value of θθ, the gradient tells us how to adjust θθ to descend toward the minimum.

Gradient Descent Formula

The weight update rule is:

θ←θ−α∇J(θ)\theta \larr \theta - \alpha \nabla J(\theta)

Where:

  • θ\theta - model parameter;
  • α\alpha - learning rate (step size);
  • ∇J(θ)\nabla J(\theta) - gradient of the function we're aiming to minimize.

For our function:

θnew=θold−α(2θold)\theta_{\text{new}} = \theta_{\text{old}} - \alpha\left(2\theta_{old}\right)

This means we update θθ iteratively by subtracting the scaled gradient.

Stepwise Movement – A Visual

Example with start values: θ=3\theta = 3, α=0.3\alpha = 0.3

  1. θ1=3−0.3(2×3)=3−1.8=1.2;\theta_1 = 3 - 0.3(2 \times 3) = 3 - 1.8 = 1.2;
  2. θ2=1.2−0.3(2×1.2)=1.2−0.72=0.48;\theta_2 = 1.2 - 0.3(2 \times 1.2) = 1.2 - 0.72 = 0.48;
  3. θ3=0.48−0.3(2×0.48)=0.48−0.288=0.192;\theta_3 = 0.48 - 0.3(2\times0.48) = 0.48 - 0.288 = 0.192;
  4. θ4=0.192−0.3(2×0.192)=0.192−0.115=0.077.\theta_4 = 0.192 - 0.3(2 \times 0.192) = 0.192 - 0.115 = 0.077.

After a few iterations, we move toward θ=0θ=0, the minimum.

Learning Rate – Choosing α Wisely

  • Too large α\alpha - overshoots, never converges;
  • Too small α\alpha - converges too slowly;
  • Optimal α\alpha - balances speed & accuracy.

When Does Gradient Descent Stop?

Gradient descent stops when:

∇J(θ)≈0\nabla J (\theta) \approx 0

This means that further updates are insignificant and we've found a minimum.

1. What is the primary goal of gradient descent?

2. What happens if the learning rate αα is too large?

3. If the gradient ∇J(θ)∇J(θ) is zero, what does this mean?

question mark

What is the primary goal of gradient descent?

Select the correct answer

question mark

What happens if the learning rate αα is too large?

Select the correct answer

question mark

If the gradient ∇J(θ)∇J(θ) is zero, what does this mean?

Select the correct answer

Var alt klart?

Hvordan kan vi forbedre det?

Tak for dine kommentarer!

Sektion 3. Kapitel 9

Spørg AI

expand

Spørg AI

ChatGPT

Spørg om hvad som helst eller prøv et af de foreslåede spørgsmål for at starte vores chat

Suggested prompts:

Can you explain how to choose the best learning rate for gradient descent?

What happens if the gradient never reaches zero?

Can you give more examples of cost functions used with gradient descent?

Awesome!

Completion rate improved to 1.89

bookGradient Descent

Stryg for at vise menuen

Gradient Descent is a fundamental optimization algorithm in machine learning used to minimize functions. It iteratively adjusts model parameters in the direction of steepest descent, allowing models to learn from data efficiently.

Understanding Gradients

The gradient of a function represents the direction and steepness of the function at a given point. It tells us which way to move to minimize the function.

For a simple function:

J(θ)=θ2J(\theta) = \theta^2

The derivative (gradient) is:

∇J(θ)=ddθ(θ2)=2θ\nabla J(\theta) = \frac{d}{d \theta}\left(\theta^2\right)= 2\theta

This means that for any value of θθ, the gradient tells us how to adjust θθ to descend toward the minimum.

Gradient Descent Formula

The weight update rule is:

θ←θ−α∇J(θ)\theta \larr \theta - \alpha \nabla J(\theta)

Where:

  • θ\theta - model parameter;
  • α\alpha - learning rate (step size);
  • ∇J(θ)\nabla J(\theta) - gradient of the function we're aiming to minimize.

For our function:

θnew=θold−α(2θold)\theta_{\text{new}} = \theta_{\text{old}} - \alpha\left(2\theta_{old}\right)

This means we update θθ iteratively by subtracting the scaled gradient.

Stepwise Movement – A Visual

Example with start values: θ=3\theta = 3, α=0.3\alpha = 0.3

  1. θ1=3−0.3(2×3)=3−1.8=1.2;\theta_1 = 3 - 0.3(2 \times 3) = 3 - 1.8 = 1.2;
  2. θ2=1.2−0.3(2×1.2)=1.2−0.72=0.48;\theta_2 = 1.2 - 0.3(2 \times 1.2) = 1.2 - 0.72 = 0.48;
  3. θ3=0.48−0.3(2×0.48)=0.48−0.288=0.192;\theta_3 = 0.48 - 0.3(2\times0.48) = 0.48 - 0.288 = 0.192;
  4. θ4=0.192−0.3(2×0.192)=0.192−0.115=0.077.\theta_4 = 0.192 - 0.3(2 \times 0.192) = 0.192 - 0.115 = 0.077.

After a few iterations, we move toward θ=0θ=0, the minimum.

Learning Rate – Choosing α Wisely

  • Too large α\alpha - overshoots, never converges;
  • Too small α\alpha - converges too slowly;
  • Optimal α\alpha - balances speed & accuracy.

When Does Gradient Descent Stop?

Gradient descent stops when:

∇J(θ)≈0\nabla J (\theta) \approx 0

This means that further updates are insignificant and we've found a minimum.

1. What is the primary goal of gradient descent?

2. What happens if the learning rate αα is too large?

3. If the gradient ∇J(θ)∇J(θ) is zero, what does this mean?

question mark

What is the primary goal of gradient descent?

Select the correct answer

question mark

What happens if the learning rate αα is too large?

Select the correct answer

question mark

If the gradient ∇J(θ)∇J(θ) is zero, what does this mean?

Select the correct answer

Var alt klart?

Hvordan kan vi forbedre det?

Tak for dine kommentarer!

Sektion 3. Kapitel 9
some-alt