Course Content

Introduction to TensorFlow

1. Tensors

Welcome to TensorFlow Introduction to Tensors Tensor Properties Applications of Tensors Batches Creating Tensors Data Types Basic Operations: Arithmetic Basic Operations: Linear Algebra Challenge: Creating a Neural Network Layer Transformations Reduction Operations Quiz: TensorFlow Basics

2. Basics of TensorFlow

Gradient Tape Graph Execution Neural Network Implementation Quiz: Conclusion Summary

Gradient Tape

Having now grasped the fundamental tensor operations, we can progress to streamlining and accelerating these processes using built-in TensorFlow features. The first of these advanced tools we'll explore is Gradient Tape.

What is Gradient Tape?

In this chapter, we'll delve into one of the fundamental concepts in TensorFlow, the Gradient Tape. This feature is essential for understanding and implementing gradient-based optimization techniques, particularly in deep learning.

Gradient Tape in TensorFlow is a tool that records operations for automatic differentiation. When you perform operations inside a Gradient Tape block, TensorFlow keeps track of all the computations that happen. This is particularly useful for training machine learning models, where gradients are needed to optimize model parameters.

Note

Essentially, a gradient is a set of partial derivatives.

Usage of Gradient Tape

To use Gradient Tape, follow these steps:

Create a Gradient Tape block: Use with tf.GradientTape() as tape:. Inside this block, all the computations are tracked;
Define the computations: Perform operations with tensors within the block (e.g. define a forward pass of a neural network);
Compute gradients: Use tape.gradient(target, sources) to calculate the gradients of the target with respect to the sources.

Simple Gradient Calculation

Let's go through a simple example to understand this better.


              123456789101112131415
            
import tensorflow as tf

# Define input variables
x = tf.Variable(3.0)

# Start recording the operations
with tf.GradientTape() as tape:
    # Define the calculations
    y = x * x
    
# Extract the gradient for the specific input (`x`)
grad = tape.gradient(y, x)

print(f'Result of y: {y}')
print(f'The gradient of y with respect to x is: {grad.numpy()}')

This code calculates the gradient of y = x^2 at x = 3. This is the same as the partial derivative of y with respect to x.

Several Partial Derivatives

When output is influenced by multiple inputs, we can compute a partial derivative with respect to each of these inputs (or just a selected few). This is achieved by providing a list of variables as the sources parameter.

The output from this operation will be a corresponding list of tensors, where each tensor represents the partial derivative with respect to each of the variables specified in sources.


              1234567891011121314151617
            
import tensorflow as tf

# Define input variables
x = tf.Variable(tf.fill((2, 3), 3.0))
z = tf.Variable(5.0)

# Start recording the operations
with tf.GradientTape() as tape:
    # Define the calculations
    y = tf.reduce_sum(x * x + 2 * z)
    
# Extract the gradient for the specific inputs (`x` and `z`)
grad = tape.gradient(y, [x, z])

print(f'Result of y: {y}')
print(f"The gradient of y with respect to x is:\n{grad[0].numpy()}")
print(f"The gradient of y with respect to z is: {grad[1].numpy()}")

This code computes the gradient of the function y = sum(x^2 + 2*z) for given values of x and z. In this example, the gradient of x is shown as a 2D tensor, where each element corresponds to the partial derivative of the respective value in the original x matrix.

Note

For additional insights into Gradient Tape capabilities, including higher-order derivatives and extracting the Jacobian matrix, refer to the official TensorFlow documentation.

Task

Swipe to show code editor

Your goal is to compute the gradient (derivative) of a given mathematical function at a specified point using TensorFlow's Gradient Tape. The function and point will be provided, and you will see how to use TensorFlow to find the gradient at that point.

Consider a quadratic function of a single variable x, defined as:

f(x) = x^2 + 2x - 1

Your task is to calculate the derivative of this function at x = 2.

Steps

Define the variable x at the point where you want to compute the derivative.
Use Gradient Tape to record the computation of the function f(x).
Calculate the gradient of f(x) at the specified point.

Note

Gradient can only be computed for values of floating-point type.

Switch to desktop for real-world practiceContinue from where you are using one of the options below

Everything was clear?

Thanks for your feedback!

Section 2. Chapter 1

Gradient Tape

What is Gradient Tape?

Note

Essentially, a gradient is a set of partial derivatives.

Usage of Gradient Tape

To use Gradient Tape, follow these steps:

Create a Gradient Tape block: Use with tf.GradientTape() as tape:. Inside this block, all the computations are tracked;
Define the computations: Perform operations with tensors within the block (e.g. define a forward pass of a neural network);
Compute gradients: Use tape.gradient(target, sources) to calculate the gradients of the target with respect to the sources.

Simple Gradient Calculation

Let's go through a simple example to understand this better.


              123456789101112131415
            
import tensorflow as tf

# Define input variables
x = tf.Variable(3.0)

# Start recording the operations
with tf.GradientTape() as tape:
    # Define the calculations
    y = x * x
    
# Extract the gradient for the specific input (`x`)
grad = tape.gradient(y, x)

print(f'Result of y: {y}')
print(f'The gradient of y with respect to x is: {grad.numpy()}')

This code calculates the gradient of y = x^2 at x = 3. This is the same as the partial derivative of y with respect to x.

Several Partial Derivatives

The output from this operation will be a corresponding list of tensors, where each tensor represents the partial derivative with respect to each of the variables specified in sources.


              1234567891011121314151617
            
import tensorflow as tf

# Define input variables
x = tf.Variable(tf.fill((2, 3), 3.0))
z = tf.Variable(5.0)

# Start recording the operations
with tf.GradientTape() as tape:
    # Define the calculations
    y = tf.reduce_sum(x * x + 2 * z)
    
# Extract the gradient for the specific inputs (`x` and `z`)
grad = tape.gradient(y, [x, z])

print(f'Result of y: {y}')
print(f"The gradient of y with respect to x is:\n{grad[0].numpy()}")
print(f"The gradient of y with respect to z is: {grad[1].numpy()}")

Note

For additional insights into Gradient Tape capabilities, including higher-order derivatives and extracting the Jacobian matrix, refer to the official TensorFlow documentation.

Task

Swipe to show code editor

Consider a quadratic function of a single variable x, defined as:

f(x) = x^2 + 2x - 1

Your task is to calculate the derivative of this function at x = 2.

Steps

Define the variable x at the point where you want to compute the derivative.
Use Gradient Tape to record the computation of the function f(x).
Calculate the gradient of f(x) at the specified point.

Note

Gradient can only be computed for values of floating-point type.

Switch to desktop for real-world practiceContinue from where you are using one of the options below

Everything was clear?

Thanks for your feedback!

Section 2. Chapter 1

Switch to desktop for real-world practiceContinue from where you are using one of the options below