Course Content

Introduction to Reinforcement Learning

1. RL Core Theory

What is RL?RL vs Other Learning Paradigms Markov Decision Process Episodes and Returns Model, Policy, and Values Exploration vs Exploitation Gymnasium Basics Challenge: Setting Up an Environment

2. Multi-Armed Bandit Problem

Problem Introduction Action Values Epsilon-Greedy Algorithm Upper Confidence Bound Algorithm Gradient Bandits Algorithm Challenge: Multi-Armed Bandits

3. Dynamic Programming

What is Dynamic Programming?Bellman Equations Optimality Conditions Policy Evaluation Policy Improvement Generalized Policy Iteration Policy Iteration Value Iteration Challenge: Dynamic Programming

4. Monte Carlo Methods

What are Monte Carlo Methods?Value Function Estimation Monte Carlo Control Exploration Approaches On-Policy Monte Carlo Control Off-Policy Monte Carlo Control Incremental Implementations Challenge: Monte Carlo Methods

5. Temporal Difference Learning

What is Temporal Difference Learning?TD(0): Value Function Estimation SARSA: On-Policy TD Learning Q-Learning: Off-Policy TD Learning Generalization of TD Learning Challenge: Temporal Difference Learning

Markov Decision Process

Definition

Markov decision process (MDP) is a mathematical framework used to model decision-making problems where an agent interacts with an environment over time.

Reinforcement learning problems are often framed as MDPs, which provide a structured way to define the problem. MDPs describe the environment using four key components: states, actions, transitions, and rewards. These components work together under the Markov property, which ensures that the future state depends only on the current state and action, not on past states.

The Four Components

State

Definition

A state $s$ is a representation of the environment at a specific point in time. The set of all possible states is called a state space $S$ .

A state is typically represented by a set of parameters that capture the relevant features of the environment. These parameters can include various aspects such as the position, velocity, rotation, etc.

Action

Definition

An action $a$ is a decision or a move made by the agent to influence the environment. The set of all possible actions is called an action space $A$ .

The set of possible actions usually depends on the current state.

Transition

Definition

Transition describes how the environment's state changes in response to the agent's action. The transition function $p$ specifies the probability of moving from one state to another, given a specific action.

In many cases, environments can be either deterministic or stochastic, meaning that the transition may be predictable or may involve some degree of randomness.

Reward

Definition

A reward $r$ is a numerical value received by the agent after taking an action in a particular state. The function that maps transitions to expected rewards is called the reward function $R$ .

Rewards steer the agent toward desirable behavior, and can be either positive or negative. Reward engineering is complex, as the agent may attempt to exploit the rewards.

Markov Property

The Markov property in a Markov decision process states that the next state and reward depend only on the current state and action, not on past information. This ensures a memoryless framework, simplifying the learning process.

Mathematically, this property can be described by this formula:

\begin{aligned} &P(R_{t+1} = r, S_{t+1} = s' | S_t, A_t)=\\ =&P(R_{t+1} = r, S_{t+1} = s' | S_0, A_0, R_1,..., S_{t-1}, A_{t-1}, R_t, S_t, A_t) \end{aligned}

where:

$S_t$ is a state at a time $t$ ;
$A_t$ is an action taken at a time $t$ ;
$R_t$ is a reward at a time $t$ .

Note

The memoryless nature of MDP doesn't mean past observations are ignored. The current state should encode all relevant historical information.

Everything was clear?

Thanks for your feedback!

Section 1. Chapter 3

Ask AI

Ask anything or try one of the suggested questions to begin our chat

Course Content

Introduction to Reinforcement Learning

1. RL Core Theory

What is RL?RL vs Other Learning Paradigms Markov Decision Process Episodes and Returns Model, Policy, and Values Exploration vs Exploitation Gymnasium Basics Challenge: Setting Up an Environment

2. Multi-Armed Bandit Problem

Problem Introduction Action Values Epsilon-Greedy Algorithm Upper Confidence Bound Algorithm Gradient Bandits Algorithm Challenge: Multi-Armed Bandits

3. Dynamic Programming

What is Dynamic Programming?Bellman Equations Optimality Conditions Policy Evaluation Policy Improvement Generalized Policy Iteration Policy Iteration Value Iteration Challenge: Dynamic Programming

4. Monte Carlo Methods

5. Temporal Difference Learning

Markov Decision Process

Definition

Markov decision process (MDP) is a mathematical framework used to model decision-making problems where an agent interacts with an environment over time.

The Four Components

State

Definition

A state $s$ is a representation of the environment at a specific point in time. The set of all possible states is called a state space $S$ .

Action

Definition

An action $a$ is a decision or a move made by the agent to influence the environment. The set of all possible actions is called an action space $A$ .

The set of possible actions usually depends on the current state.

Transition

Definition

In many cases, environments can be either deterministic or stochastic, meaning that the transition may be predictable or may involve some degree of randomness.

Reward

Definition

Rewards steer the agent toward desirable behavior, and can be either positive or negative. Reward engineering is complex, as the agent may attempt to exploit the rewards.

Markov Property

Mathematically, this property can be described by this formula:

\begin{aligned} &P(R_{t+1} = r, S_{t+1} = s' | S_t, A_t)=\\ =&P(R_{t+1} = r, S_{t+1} = s' | S_0, A_0, R_1,..., S_{t-1}, A_{t-1}, R_t, S_t, A_t) \end{aligned}

where:

$S_t$ is a state at a time $t$ ;
$A_t$ is an action taken at a time $t$ ;
$R_t$ is a reward at a time $t$ .

Note

The memoryless nature of MDP doesn't mean past observations are ignored. The current state should encode all relevant historical information.

Everything was clear?

Thanks for your feedback!

Section 1. Chapter 3