Academic Databases
Explore Our Semantic Databases
STEM & Tech
Math, Physics, CS & Engineering...
Economics
Macro, Micro, Econometrics...
Business
Management, Marketing, Finance...
Health & Life Sciences
Biology, Medicine, Neuroscience...
Humanities & Social Sciences
Psychology, Law, Philosophy, History...
General Studies
General & interdisciplinary studies...
Explore Topics
Features

Proximal Policy Optimization: RL Algorithm Explained | PPO Tutorial

Added: 2026-08-07

Deep Dive

Fundamental Reinforcement Learning concepts, including Markov Decision Processes (MDPs), states, actions, rewards, and the balance of exploration vs. exploitation.

doc V. Lisý: Introduction to Artificial Intelligence (BE4B36ZUI) — 05 [18. 3. 2025, LS 24/25]

doc V. Lisý: Introduction to Artificial Intelligence (BE4B36ZUI) — 05 [18. 3. 2025, LS 24/25]

352

The Markov Decision Process (MDP) is the fundamental formal model for reinforcement learning, consisting of states, actions, rewards, environment dynamics, and the Markov property (future depends only on current state). This framework allows any real-world problem to be modeled for RL application. In Atari games, states are screen observations, rewards are scores, actions are joystick movements, and dynamics are the game engine. In robotics, states include joint angles and positions, rewards are for desired behaviors, and actions are movements. The simplest MDP has one state with multiple actions, capturing the essential exploration-exploitation trade-off.

Click to watch →

RL 6: Policy iteration and value iteration - Reinforcement learning

RL 6: Policy iteration and value iteration - Reinforcement learning

59K

Markov Decision Processes (MDPs) provide the mathematical framework for reinforcement learning, consisting of five core components: agents (decision-makers), environments (transition dynamics), states (situations the agent encounters), actions (choices available to the agent), and rewards (feedback signals). A policy is a function mapping states to actions, which can be deterministic or probabilistic. The environment provides transition dynamics as a joint probability distribution of next state and reward given current state and action. Discounted reward introduces a discount factor γ (0<γ<1) that weights future rewards exponentially, giving more importance to immediate rewards. The value of a state under a policy represents the expected total discounted reward achievable from that state onward.

Click to watch →

Markov Decision Processes Explained | The Foundation of Reinforcement Learning

Markov Decision Processes Explained | The Foundation of Reinforcement Learning

1.2K

A Markov Decision Process (MDP) is a mathematical framework for modeling sequential decision-making problems in reinforcement learning, consisting of four core components: states (the environment's configuration), actions (choices available to the agent), transition probabilities (likelihood of moving between states given actions), and reward functions (immediate feedback received after actions). The defining Markov property ensures that future states depend only on the present state and action, not on the sequence of events that preceded them, enabling efficient computation and modeling of complex environments. MDPs enable AI agents to learn optimal policies through trial and error by balancing exploration and exploitation to maximize cumulative rewards over time, with applications spanning robotics, finance, healthcare, and game AI.

Click to watch →

Lec 69 Introduction to Reinforcement Learning : RL

Lec 69 Introduction to Reinforcement Learning : RL

107

The Markov Decision Process (MDP) provides the mathematical foundation for reinforcement learning. An MDP consists of five key components: (1) State space (S) - all possible states the agent can be in; (2) Action space (A) - all possible actions the agent can take; (3) Transition kernel - probability of transitioning between states given an action; (4) Reward function - maps state-action pairs to reward values (can be negative); (5) Discount factor (γ) - a number between 0 and 1 that determines how much future rewards are valued. The transition kernel does not depend on the reward function.

Click to watch →

Lecture 1: Foundations of Reinforcement Learning: Introduction and MDP Basics

Lecture 1: Foundations of Reinforcement Learning: Introduction and MDP Basics

8.8K

Reinforcement learning is a mathematical framework for sequential decision-making problems where an agent interacts with an environment through states, actions, and rewards. The core mathematical model underlying reinforcement learning is the Markov Decision Process (MDP), which consists of a state space S, action space A, reward function R, transition probability P, and horizon H. An MDP captures how actions influence future states through probabilistic transitions, embodying the Markov property that the future depends only on the current state and action. Key concepts include policies (strategies mapping states to actions), value functions (expected cumulative rewards), and Q-values (value of state-action pairs). The fundamental objective is to find an optimal policy that maximizes cumulative reward, balancing exploration versus exploitation trade-offs and addressing challenges like large state spaces and partial observability.

Click to watch →

Deep Dive

Policy Gradient Methods, specifically how algorithms like REINFORCE update policy parameters directly using gradient ascent.

Reinforcement Learning: A (practical) introduction

Reinforcement Learning: A (practical) introduction

828

REINFORCE is a policy-based RL method that directly optimizes the policy parameters theta to maximize expected return. The objective function J(theta) represents the average total return across all possible trajectories under policy pi_theta. The update rule uses gradient ascent: theta_new = theta_old + alpha × gradient(J). For each action-reward pair, the gradient term is reward multiplied by the log-probability of the action under the current policy, effectively increasing parameters that led to good decisions and decreasing those that led to bad ones.

Click to watch →

Sutton and Barto Reinforcement Learning Chapter 13: Policy Gradient Methods Introduction

Sutton and Barto Reinforcement Learning Chapter 13: Policy Gradient Methods Introduction

207

REINFORCE is the first Monte Carlo policy gradient algorithm. It uses the policy gradient theorem expression and replaces the expected value with actual samples collected from following the policy. For each trajectory, the algorithm computes the total discounted return G_t at each time step, multiplies it by the eligibility vector (gradient of log-policy), and updates parameters using gradient ascent: θ ← θ + α ∑_t G_t ∇_θ log π(a_t|s_t). Since it requires full episodes to compute returns, it only works for episodic problems.

Click to watch →

RL Debugging and Diagnostics | Stanford CS229: Machine Learning Andrew Ng - Lecture 20 (Autumn 2018)

RL Debugging and Diagnostics | Stanford CS229: Machine Learning Andrew Ng - Lecture 20 (Autumn 2018)

75.9K

The REINFORCE algorithm is a direct policy search method in reinforcement learning that performs gradient ascent on the expected total payoff by updating policy parameters using the formula θ ← θ + α × payoff × ∇_θ log π_θ(a|s), where each update direction is a random sample whose expected value equals the true gradient of the objective function, making it a theoretically sound stochastic gradient ascent algorithm despite suffering from high variance that requires many iterations for convergence.

Click to watch →

L9: Policy Gradient Methods (P5-Gradient-based algorithms&REINFORCE) —Mathematical Foundations of RL

L9: Policy Gradient Methods (P5-Gradient-based algorithms&REINFORCE) —Mathematical Foundations of RL

1.1K

Policy gradient methods optimize reinforcement learning policies by computing the gradient of the objective function and applying gradient ascent, where the gradient is approximated using stochastic gradients to handle the unknown state distribution; REINFORCE is the simplest policy gradient algorithm that uses Monte Carlo methods to estimate action values, and the algorithm balances exploitation (increasing probability of high-value actions) and exploration (increasing probability of low-probability actions) through the βt term, which is proportional to the estimated action value and inversely proportional to the current policy probability.

Click to watch →

Policy Gradients are Easy in Tensorflow 2 | Complete Deep Reinforcement Learning Tutorial |

Policy Gradients are Easy in Tensorflow 2 | Complete Deep Reinforcement Learning Tutorial |

9.9K

Policy gradients are a Monte Carlo-based reinforcement learning method that uses deep neural networks to approximate an agent's policy—a probability distribution over actions given states—by maximizing the expected cumulative reward through gradient ascent; the REINFORCE algorithm implements this by calculating the discounted sum of future rewards for each action, computing the gradient of the log-probability of actions taken multiplied by these rewards, and updating the policy network parameters using gradient ascent to encourage profitable actions while maintaining exploration through non-zero probabilities for all actions.

Click to watch →

Deep Dive

The Actor-Critic Architecture, understanding how a policy network (the actor) and a value network (the critic) collaborate to evaluate and improve actions.

Fidle - Séquence 13 (Live)

Fidle - Séquence 13 (Live)

1.3K

Actor-critic combines two networks: (1) The actor (policy network) outputs actions based on the current state; (2) The critic (value network) estimates the value of states or state-action pairs. The actor uses the critic's value estimates to update its policy, while the critic learns from the actor's actions. This architecture enables TD learning instead of Monte Carlo, allowing incremental updates without waiting for complete episodes.

Click to watch →

Lecture 7 | Advanced Methods | Spring 25 (Screen Record)

Lecture 7 | Advanced Methods | Spring 25 (Screen Record)

126

The Actor-Critic architecture consists of two neural networks: the actor network (policy network) that takes a state and outputs action probabilities, and the critic network (value network) that takes a state and outputs a value estimate. Training data consists of state-action-reward-next state tuples. The target for the critic network is: target = R(s, a) + gamma * V(s'). The target for the actor network is the advantage function: A(s, a) = Q(s, a) - V(s). The actor network is updated to maximize the advantage function, which encourages actions that are better than average and discourages actions that are worse than average.

Click to watch →

Landing a SpaceX Falcon Heavy Rocket

Landing a SpaceX Falcon Heavy Rocket

50.8K

Actor-critic architectures use two neural networks: the actor (policy network) selects actions, while the critic (value network) evaluates action quality. This dual-network approach typically outperforms single-network methods. Policy gradient methods directly optimize policies by computing gradients of expected reward with respect to policy parameters, using episode outcomes as learning signals. Trust Region Policy Optimization (TRPO) improves stability by constraining policy updates to prevent large, destabilizing changes. Proximal Policy Optimization (PPO) implements TRPO principles more efficiently using surrogate loss functions and KL divergence penalties.

Click to watch →

Naive Actor Critic With Experience Replay | When Great Ideas Go Horribly Wrong

Naive Actor Critic With Experience Replay | When Great Ideas Go Horribly Wrong

6.3K

An actor-critic network has common lower layers that split into two streams: one for the policy (actor) and one for the value function (critic). The actor outputs action probabilities (policy), while the critic outputs state values. The actor tells the agent what to do, and the critic evaluates how good the action was. This architecture differs from value-based methods like DQN which have a single Q-value output.

Click to watch →

【強化学習】Actor-Critic - 行動者と評価者を同時に学習！【方策勾配法⑤】RL vol. 26 #202 #VRアカデミア #ReinforcementLearning

【強化学習】Actor-Critic - 行動者と評価者を同時に学習！【方策勾配法⑤】RL vol. 26 #202 #VRアカデミア #ReinforcementLearning

3.3K

The Actor-Critic method is a reinforcement learning algorithm that simultaneously learns a policy (actor) and a value function (critic) by using Temporal Difference (TD) error to guide both components; the actor updates its parameters using the gradient of the policy multiplied by the TD error to improve action selection, while the critic updates its value function parameters using the TD error multiplied by the gradient of the value function to improve state-value estimation, combining policy gradient methods with TD learning for efficient credit assignment.

Click to watch →

Deep Dive

Basic Deep Learning and Neural Network optimization, including loss functions, backpropagation, and learning rates.

MIT 6.S191 (2021): Introduction to Deep Learning

MIT 6.S191 (2021): Introduction to Deep Learning

565.9K

Training neural networks requires quantifying prediction error through loss functions: cross-entropy for classification, mean squared error for regression. The empirical loss averages error across all training examples. Gradient descent optimizes weights by iteratively computing gradients (direction of steepest ascent) and taking steps in the opposite direction, scaled by a learning rate. Backpropagation computes these gradients using the chain rule recursively, propagating errors backward through the network. This enables efficient computation of gradients for all weights in deep networks. The learning rate determines step size—too low causes slow convergence, too high causes divergence. Adaptive optimizers adjust learning rates dynamically based on observed gradients and weight magnitudes.

Click to watch →

ATPESC 2022 8.2 Deep Learning Methods | Tanwi Mallick

ATPESC 2022 8.2 Deep Learning Methods | Tanwi Mallick

105

Loss functions measure prediction error: binary cross entropy for classification (heavily penalizes wrong predictions), mean squared error for regression (squared differences). Gradient descent minimizes loss by: (1) randomly initializing weights, (2) computing gradients (partial derivatives of loss w.r.t. weights), (3) updating weights in the opposite direction of the gradient, (4) repeating until convergence. The learning rate controls step size - too high causes instability, too low causes slow convergence or local minima. Backpropagation applies the chain rule to compute gradients across all layers. Training challenges include complex optimization problems where convergence is not guaranteed. The MNIST dataset (handwritten digits 0-9, 28x28 pixels) is a standard benchmark for demonstrating neural network training.

Click to watch →

MIT Introduction to Deep Learning (2024) | 6.S191

MIT Introduction to Deep Learning (2024) | 6.S191

1M

This section covers the complete training methodology for neural networks. It begins with loss functions that quantify prediction accuracy: cross-entropy for binary classification and mean squared error for regression. The core optimization algorithm, gradient descent, is explained as an iterative process that navigates the loss landscape by computing gradients and taking steps in the direction of steepest descent. The critical technique of backpropagation is introduced, demonstrating how gradients are computed using the chain rule from calculus, propagating backward through each layer to determine how much each weight contributes to overall loss. The section emphasizes that while theoretically simple, practical optimization of deep neural networks is challenging due to high-dimensional loss landscapes with many local minima. The learning rate parameter is explained as controlling step size, with too small causing slow convergence and too large causing overshooting or divergence.

Click to watch →

Introduction to Neural Networks | What is Neural Network | Deep Learning | Re-live | Ivy Pro School

Introduction to Neural Networks | What is Neural Network | Deep Learning | Re-live | Ivy Pro School

313

This section covers the mathematical machinery behind neural network training: loss functions quantify prediction error (MSE for regression, cross-entropy for classification); gradient descent finds minimum error by moving opposite to gradient direction; learning rates control step size - too large causes overshooting, too small slows convergence; backpropagation computes gradients using chain rule across network layers; this optimization framework transforms initial random weights into effective representations through iterative refinement. The section explains why these mathematical principles are essential for anyone serious about deep learning.

Click to watch →

What is a neural network and its fundamentals? | in Urdu/ Hindi

What is a neural network and its fundamentals? | in Urdu/ Hindi

1.3K

The loss function measures the difference between predicted and actual outputs, guiding the network's learning process. For classification tasks, cross-entropy loss is preferred, while mean squared error is used for regression. Backpropagation is the mechanism by which the network learns from errors: after forward propagation, the network calculates the loss and adjusts weights to minimize it. The learning rate controls how much weights change during each iteration—too high causes divergence, too low causes slow convergence. The goal is to reach the global minimum of the loss function, where the model performs optimally.

Click to watch →

Core Concepts

Training Overview

Loss & Advantage

Detailed Loss

Policy Update

Final Summary

Core Concepts

0:00

Playing Section

1
Explains policy and value networks, their roles, and outputs.
2
Policy network outputs action probabilities; value network outputs Q-values.

Deep Dive

Reinforcement Learning from Human Feedback (RLHF), exploring how PPO is utilized to align Large Language Models (LLMs) like ChatGPT with human preferences.

Reinforcement Learning from Human Feedback: From Zero to chatGPT

Reinforcement Learning from Human Feedback: From Zero to chatGPT

186.7K

Reinforcement Learning from Human Feedback (RLHF) is a three-phase framework for training large language models that addresses the challenge of encoding complex human values (such as safety, helpfulness, and ethical considerations) into machine learning systems. The process begins with pre-training a large language model on internet text, followed by training a reward model using human-labeled preference data where humans rank different model responses to the same prompt. Finally, the language model is fine-tuned using reinforcement learning (typically PPO algorithm) to maximize the reward predicted by the reward model while maintaining KL divergence constraints to prevent catastrophic forgetting. This approach enables models like ChatGPT to generate more aligned and human-like responses compared to purely unsupervised training.

Click to watch →

Modern Reinforcement Learning (RL), Part 1: How RL Powers Generative AI

Modern Reinforcement Learning (RL), Part 1: How RL Powers Generative AI

114

RLHF follows a traditional RL pipeline for aligning LLMs with human preference. It consists of three stages: supervised fine-tuning on curated high-quality responses to establish behavioral foundations; reward model training where human annotators compare response pairs to train a model that scores satisfaction and alignment; and policy optimization using algorithms like PPO to maximize learned reward while staying close to original behavior through KL divergence penalty. PPO has become the industry standard due to its stability - it offers controlled optimization preventing aggressive updates that could destabilize training, is robust against noisy human feedback, uses clipping mechanisms to keep updates within safe bounds, and scales well for billion-parameter models.

Click to watch →

Intro to Fine-Tuning Large Language Models

Intro to Fine-Tuning Large Language Models

51.8K

Proximal Policy Optimization (PPO) is a reinforcement learning algorithm used in RLHF to optimize model performance. The process involves: receiving a prompt, generating multiple possible responses, having the reward model rank these responses from best to worst, and using PPO to adjust the model's parameters based on this feedback. PPO strikes a balance between exploration (trying new responses) and exploitation (improving based on past feedback), preventing drastic changes that might degrade performance.

Click to watch →

LLMs for Everything and Everyone! - Sebastian Raschka - Lightning AI

LLMs for Everything and Everyone! - Sebastian Raschka - Lightning AI

7.3K

RLHF is a three-step alignment process: (1) Humans write responses to sampled prompts; (2) Supervised fine-tuning creates a model from these responses; (3) Humans rank multiple responses, training a reward model that scores responses. Proximal Policy Optimization (PPO) then uses the reward model to refine the supervised fine-tuned model. This process, while labor-intensive, significantly improves model alignment with human preferences and produces better responses than baseline models.

Click to watch →

Stanford CME295 Transformers & LLMs | Autumn 2025 | Lecture 5 - LLM tuning

Stanford CME295 Transformers & LLMs | Autumn 2025 | Lecture 5 - LLM tuning

22.2K

This lecture covers preference tuning and reinforcement learning from human feedback (RLHF) for aligning large language models with human preferences. The process involves collecting pairwise preference data where humans rate which model response is better, then training a reward model using the Bradley-Terry formulation to score responses. The second stage uses reinforcement learning algorithms like Proximal Policy Optimization (PPO) with clipping and KL-divergence penalties to align the model with these preferences without deviating too far from the base model. An alternative approach called Direct Preference Optimization (DPO) provides a simpler supervised method that directly optimizes model weights using preference pairs, avoiding the complexity of maintaining separate reward and value functions.

Click to watch →

Deep Dive

Generalized Advantage Estimation (GAE), a crucial variance-reduction technique often paired with PPO to compute policy advantages.

Stanford CS234 Reinforcement Learning I Policy Search 3 I 2024 I Lecture 7

Stanford CS234 Reinforcement Learning I Policy Search 3 I 2024 I Lecture 7

10.4K

Generalized Advantage Estimation (GAE) is a technique used in Proximal Policy Optimization (PPO) that combines multiple k-step advantage estimators into a single expression using a weighted combination with parameter λ, enabling more efficient and stable policy gradient updates by balancing bias and variance trade-offs between temporal difference and Monte Carlo estimation methods.

Click to watch →

PufferLib - the hard problem in RL

PufferLib - the hard problem in RL

196

Generalized Advantage Estimation (GAE) is the key technique that makes on-policy RL work effectively. The speaker explains that GAE reduces variance by rolling forward trajectory segments and computing advantage estimates that combine multiple time steps. While average reward is an unbiased objective, the variance reduction from GAE is essential for practical RL performance. The speaker notes that GAE is the workhorse that enables on-policy algorithms to achieve good results, and without it, the high variance of Monte Carlo estimates makes training unstable. The speaker discusses how GAE works by exponentially weighting bootstraps over trajectory segments, providing a balance between bias and variance.

Click to watch →

PPO Implementation With PyTorch | Reinforcement Learning Tutorial

PPO Implementation With PyTorch | Reinforcement Learning Tutorial

PPO introduces policy ratio clipping to prevent unstable updates: the ratio π_θ/π_θ_old is constrained within [1-ε, 1+ε] (e.g., 0.8-1.2), and the actor loss uses the minimum of clipped and unclipped values. Generalized Advantage Estimation (GAE) calculates advantage as the sum of γ^k * λ * Δ_t, where Δ_t = r_t + γV(s_{t+1}) - V(s_t). The discount factor γ (typically 0.99) and smoothing parameter λ (typically 0.95) balance bias-variance trade-offs in advantage estimation.

Click to watch →

Lecture 2 Policy Gradient / On Policy Reinforcement Learning, Deep Robotic Learning @ UW

Lecture 2 Policy Gradient / On Policy Reinforcement Learning, Deep Robotic Learning @ UW

301

GAE provides a spectrum of advantage estimators parameterized by λ ∈ [0,1]. When λ=0, we get the one-step advantage estimator A_t^0 = r_{t+1} + γV(s_{t+1}) - V(s_t), which is biased but low-variance. When λ=1, we get the Monte Carlo return estimator A_t^1 = R̂_t^T - V(s_t), which is unbiased but high-variance. GAE computes a weighted combination: A_t^λ = (1-λ)∑_{k=0}^∞ λ^k [γ^k A_t^k], where A_t^k is the k-step advantage estimator. Choosing λ ≈ 0.95 empirically balances bias-variance tradeoff well in practice.

Click to watch →

Reinforcement Learning (PPO) Football Agent | Part 3: Generalized Advantage Estimation

Reinforcement Learning (PPO) Football Agent | Part 3: Generalized Advantage Estimation

4.7K

Generalized Advantage Estimation (GAE) is an algorithm used in reinforcement learning to calculate the advantage of taking a specific action by combining immediate rewards with discounted future rewards, using a backward loop through time steps to compute Delta values (reward + gamma * value of next state - value of current state) and accumulating them with a smoothing parameter (lambda) to reduce variance, ultimately producing normalized advantage estimates that help train actor-critic models like PPO.

Click to watch →

Deep Dive

Hands-on implementation of PPO using Deep Learning frameworks (PyTorch or TensorFlow) or standard RL libraries like Stable-Baselines3.

3 Train a Unity RL Env using Stable Baselines3!

3 Train a Unity RL Env using Stable Baselines3!

2.9K

This section covers the complete workflow for training a Proximal Policy Optimization (PPO) agent using Stable Baselines3. First, import necessary modules including PPO from stable_baselines3.ppo.ppo. Configure the PPO trainer by specifying the policy type (MlpPolicy for simple neural networks), the wrapped environment, and hyperparameters such as verbose level and entropy coefficient (ent_coef). The entropy coefficient controls exploration, typically set between 0.01-0.1. Wrap the environment with Monitor from stable_baselines3.common.monitor to automatically track episode statistics like average length and reward. Instantiate the PPO object with these configurations and call learn(total_timesteps=...) to begin training. The training process involves the agent interacting with the environment, receiving rewards, and iteratively improving its policy through gradient ascent on the PPO objective function.

Click to watch →

PyTorch 2.0 Q&A: TorchRL

PyTorch 2.0 Q&A: TorchRL

3.4K

PPO (Proximal Policy Optimization) is one of the major RL algorithms. The tutorial demonstrates implementing PPO with TorchRL, showing various library components. The environment wrapper handles device placement (CPU/GPU) and data transformation. Transforms are composed as a list rather than nested wrappers, making it easy to add, remove, or modify preprocessing steps. Transforms can access the parent environment to initialize statistics like mean and standard deviation for normalization.

Click to watch →

Part 1 of 3 — Proximal Policy Optimization Implementation: 11 Core Implementation Details

Part 1 of 3 — Proximal Policy Optimization Implementation: 11 Core Implementation Details

63.2K

PPO is a deep reinforcement learning algorithm proposed by OpenAI in 2017 that has become one of the most popular RL algorithms. This video covers building a PPO implementation in PyTorch from scratch, covering 11 general implementation details. The target audience is intermediate to advanced RL practitioners familiar with PyTorch and basic PPO concepts. Key resources recommended include OpenAI Spinning Up for beginners and Lil'Log's policy gradient blog post for deeper understanding.

Click to watch →

Reinforcement Learning with Stable Baselines 3 - Introduction (P.1)

Reinforcement Learning with Stable Baselines 3 - Introduction (P.1)

110.8K

Stable Baselines 3 is a Python library that simplifies reinforcement learning by abstracting away complex algorithm implementations, similar to how scikit-learn simplifies general machine learning; it allows users to quickly experiment with various RL algorithms on custom or pre-built environments (like those from OpenAI Gym) by defining the environment, selecting an appropriate algorithm based on whether the action space is discrete or continuous, and training the model through repeated interactions where the agent receives observations, takes actions, and receives rewards until the episode ends.

Click to watch →

SuperFast Reinforcement Learning Tutorial: A2C, DQN, PPO, TD3 with Stable-Baselines3 - AIML 101

SuperFast Reinforcement Learning Tutorial: A2C, DQN, PPO, TD3 with Stable-Baselines3 - AIML 101

496

DQN extends Q learning by using deep neural networks to approximate Q values, enabling scaling to complex environments with large/continuous state spaces. Q learning is model-free, updating Q value tables for state-action pairs using the Bellman equation, balancing exploration and exploitation. DQN stores experiences in replay buffers and uses target networks for training stability. In Lunar Lander, the agent takes four actions (do nothing, left thruster, right thruster, main engine) and learns to land safely. PPO (Proximal Policy Optimization), introduced by John Schulman's team in 2017, makes learning stable and efficient by making small controlled policy updates through a clipped objective function, preventing large destabilizing changes. PPO is simpler than TRPO while maintaining stability. For Cart Pole, PPO can achieve high scores after 24,000 time steps and maximum reward after 1 million steps.

Click to watch →

Deep Dive

Advanced off-policy and maximum-entropy algorithms, such as Soft Actor-Critic (SAC) and Deep Deterministic Policy Gradient (DDPG).

Soft Actor-Critic | Lecture 83 (Part 3) | Applied Deep Learning

Soft Actor-Critic | Lecture 83 (Part 3) | Applied Deep Learning

4.4K

Soft Actor-Critic (SAC) is an off-policy maximum entropy deep reinforcement learning algorithm that combines policy optimization and value optimization frameworks by incorporating entropy maximization directly into the objective function, resulting in a stochastic policy that achieves better exploration-exploitation balance and faster convergence compared to deterministic alternatives like DDPG and PPO.

Click to watch →

L5 DDPG and SAC (Foundations of Deep RL Series)

L5 DDPG and SAC (Foundations of Deep RL Series)

31.2K

DDPG and SAC are actor-critic methods for continuous control in deep reinforcement learning; DDPG uses deterministic policy gradients with Q-learning updates and replay buffers for off-policy learning, while SAC extends this by incorporating maximum entropy regularization into the objective function, which enhances exploration and reduces overfitting, making SAC more robust and sample-efficient for complex robotic control tasks.

Click to watch →

CS885 Module 2: Maximum Entropy Reinforcement Learning

CS885 Module 2: Maximum Entropy Reinforcement Learning

9.6K

Maximum entropy reinforcement learning introduces an entropy regularizer to the reward function, encouraging stochastic policies that are more robust, generalize better across environments, and avoid local optima; this approach leads to the Soft Actor-Critic algorithm, which uses a softmax policy derived from the Q-function and achieves superior performance compared to standard Q-learning and policy gradient methods through off-policy updates with a replay buffer.

Click to watch →

Sergey Levine: Control as Inference and Soft Deep RL

Sergey Levine: Control as Inference and Soft Deep RL

5.6K

Two practical algorithms implement maximum entropy RL: (1) Soft Q-learning updates Q functions using soft max backups and trains sampling distributions for continuous actions, but doesn't directly optimize policy; (2) Soft Actor-Critic (SAC) directly learns Q^π and updates policy via KL minimization to exp(Q - V), corresponding to variational inference. SAC is substantially more stable and scalable. Experiments show SAC outperforms prior methods on humanoid locomotion tasks and demonstrates remarkable stability across runs, unlike deterministic versions without entropy regularization.

Click to watch →

Advanced Deep Reinforcement Learning Algorithms | PPO, TRPO, DDPG, TD3, SAC Explained | L-09

Advanced Deep Reinforcement Learning Algorithms | PPO, TRPO, DDPG, TD3, SAC Explained | L-09

391

This lecture introduces five advanced deep reinforcement learning algorithms: PPO (Proximal Policy Optimization) balances exploration and exploitation using a clipped objective function; TRPO (Trust Region Policy Optimization) ensures stable updates through KL divergence constraints; DDPG (Deep Deterministic Policy Gradient) handles continuous action spaces with actor-critic architecture; TD3 (Twin Delayed DDPG) improves upon DDPG by addressing overestimation through twin Q-networks and delayed updates; and SAC (Soft Actor-Critic) maximizes entropy for robust exploration in continuous control tasks. Each algorithm offers unique strengths suited for different problem requirements.

Click to watch →

Core Concepts

Training Overview

Loss & Advantage

Detailed Loss

Policy Update

Final Summary

Core Concepts

0:00

Playing Section

1
Explains policy and value networks, their roles, and outputs.
2
Policy network outputs action probabilities; value network outputs Q-values.

41.2K views1.4K13:26@CodeEmporiumOriginal Release: 2023-12-04

Proximal Policy Optimization (PPO) is a reinforcement learning algorithm that enables AI agents to modify their decision-making by using two neural networks—a policy network that outputs a probability distribution over possible actions given a state, and a value function network that outputs Q-values quantifying how good each action was. The algorithm works by collecting experience data (state, action, reward, action probability) during episodes, then computing advantages as the difference between actual future rewards and predicted Q-values. These advantages are used to calculate losses for both networks: the value function loss minimizes the squared advantage across the batch, while the policy loss uses clipped probability ratios multiplied by advantages to ensure stable updates. PPO balances effective policy improvements with cautious updates, making it suitable for training complex models like ChatGPT to produce safe, factual, and non-toxic responses.

[00:00:00]greetings fellow Learners now before we embark on our journey towards understanding proximal policy optimization I have a question for you how do you modify your decision making for a certain activity that you do or what does it take to modify your decision- making for a certain activity can you express this at all in words now this could be your decision- making on how you invest money or decision- making on your exercise routine or just any anything else share your thoughts in the comments down below and I would love to hear them in this video we're going to delve into a popular method that an AI uses to modify its decision making called proximal policy optimization this video is going to be divided into three passes where in the first pass we'll start by introducing some Concepts and definitions and as we go down into the later passes we'll dive into further details about the algorithm itself pay attention because I'm going to quiz you along the way so let's get to it the Poo algorithm makes use of two main architectures a policy Network and a value function Network they are both neural networks that take an input and return an output the policy network will take a stateus input and it will produce an action as output from an architecture standpoint the output layer this network has a number of neurons equal to the number of possible actions it can take and each neuron is a probability that an action is taken when the agent is in the input State the value function takes a state and an action as input and it will output a real number that quantifies how good was this decision this number is known as a q value from an architecture standpoint as was with the policy netor Network the output layer of this network has the number of neurons equal to the number of possible actions it can take and so technically this value function neural network will take in a state as input and for every action it is going to determine a q value and this will quantify how good was this specific output action for this specific State quiz time have you been paying attention let's quiz you to find out what is the primary function of the output layer in the policy network of this PO algorithm a it outputs Q values for the given State and action B it produces a probability distribution over possible actions given the input State C it calculates the advantage estimates for policy updates or D it generates random actions for exploration comment your answer down below and let's just have a discussion and if you think I deserve it and you love learning please consider hitting that like button now that's going to do for quiz time and pass one of our explanation but I'll be back so pay attention to help us out let's bring out Frank our lovely friend say hi Frank hello what a cutie now Frank is in this grid world and he needs to know how to get to that plus 10 reward spot now to do so he can take the actions of left right up or down and in this pass we're going to give an overview of how Frank can learn to navigate this world using po so let's talk about the overview of the training process Frank starts at some random State the state is passed into a policy Network the policy network will determine the probability of generating each action this is then a probability distribution and we sample from this to determine the actual next action Frank then takes that next action and will receive some reward we then stored the quadruple of the state action reward and action probability into a data store this information will be used ful when training the policy Network and the value function Network and we'll talk about this later we now repeat the sequence of steps for the episode or some fixed number of time steps within the episode now we have data stored in this episode as a batch and we can collect multiple batches if we choose but let's just stop over here thanks for the help Frank we'll take it from here and we're going to help you learn okay nice now let's talk about training the two networks take the batch of the state action reward and action probability now the state in action are used with the value function Network to give us a q value and like we said before this will quantify how good do we expect this action to be we then determine the total future reward for every time step using the data that we stored and this will quantify how good did we actually perform we take the difference between these two values and this difference is known as the advantage we use this advantage in order to compute a loss this loss is then back back propagated through the value function Network and so it learns we then use the same advantage and probabilities that we stored previously to determine the loss for the policy Network the loss is back propagated through the policy Network so its parameters are updated and then we repeat this process for all batches of data effectively the policy Network and the value function Network get better over time as they learn over time together and Frank learns to make better and better decisions so that he can get to that plus 10 reward square and this is an overview of the one iteration for proximal policy optimization quiz time I'm back back have you been paying attention let's quiz you to find out what is the main purpose of the Q values produced by the value function Network in po a to represent the probabilities of different actions given a state B to calculate the advantage estimates using policy updates C to generate probability distribution over possible actions or D to produce random actions for exploration comment your answer down below and let's have a discussion that'll do it for quiz time for now and also P two but keep paying attention because I will be back to quiz you so in past two we saw that the value function Network and the policy Network are trained together and the overview of the steps is basically we compute the LW loss for the value function Network we compute the loss for the policy Network we update both networks together and repeat and then Frank becomes better and better as a decision maker let's now take the overview of the process that we discussed in pass two but we're going to add a few more details specifically along the loss function calculation so let's start with the value function Network and generating its loss so we get the batch of data for the episode that we stored in past two for each time step we compute the actual future reward with the data gathered and this is done by taking the sum of discounted future rewards and then we can compute the expected future reward by passing the state into the value function neural network and it produces a q value for every action and then we look at the Q value for the specific action neuron in our Tuple so for every every time step we have two numbers we take the difference between these two numbers to get the advantage we Square the advantage for every time step and we can take the average of this across the batch this final number is a loss that we back propagate through the value function Network and so the neural network learns and at the same time all of this happens we are also training the policy Network so let's see how that's going now if we write it in mathematical form the policy Network loss looks like this very cumbersome but let's explain what's going on first we'll get the batch of data that we stored next pass the batch of states to the policy Network to get the probabilities of actions in each case we only consider the probability of the action taken when we gathered the data we then divide two numbers that is a probability for the specific action that we have now divided by the probability we collected previously and this is a probability ratio we'll multiply this ratio with the advantage computed in this time step and so for every time step in the episode we have a number let's keep this number aside and hold on to this Frank for me okay okay nice next we take the probability ratio and we'll clip it to ensure that we're not changing the network too much we multiply this by the advantage and so now for every time step we have two values we'll take the minimum of these values and then when we take the average of the values across the batch we'll get a single number this single number is our loss that is back propagated through the policy Network so this is quite a bit of steps but overall the loss function strikes a balance between making effective policy updates to improve per performance and making cautious policy updates to improve stability and effectively the value function Network and the policy Network are trained together and that is po quiz time this is going to be a fun one have you been paying attention let's quiz you to find out what is the primary purpose of computing the advantage in the loss function for the value function Network a to determine the ratio of probabilities between the old and new policies B to calculate the expected future rewards and guide updates to the value function C to click the probability ratio and ensure stable policy updates or D to compute the loss for the policy Network during training comment your answer down below and let's just have a discussion and once again if you do think I deserve this and you haven't done so yet please consider hitting that like button thank you so much and that will do it for quiz time for this video but before we go let's get a [Music] summary proximal policy optimization algorithm is used to learn a policy directly now the PO algorithm makes use of two architectures a policy Network and a value function Network the polic Network predicts a probability distribution of actions whereas the value function Network predicts Q values for every action taken from a state the PO algorithm involves training a policy Network and the value function Network iteratively and together and just as a new tidbit here but this algorithm is actually used by chat GPT and other large language models today to ensure that the responses that they give are safe factual and non-toxic now that's all we have for today and if you like this video please do consider giving it a like and if you want to look at some similar videos here check out this video on deep Q networks and just caution I'm going to be quizzing you a lot in this video too so if you like all that stuff please do check it out and I will see you in the next one bye-bye

Up Next

Train a Humanoid to Walk: Stable Baselines3 & Soft Actor-Critic

@johnnycode

16.5K views•2023-07-20

Building Real-Time ML Pipelines with Feature Stores and MLOps Frameworks

@ODSCAI

5.1K views•2022-02-20

Q-Learning Explained: A Guide to Reinforcement Learning Basics

@CodeEmporium

82.4K views•2023-11-07

Neural Networks Explained: Math, Layers, and Learning Fundamentals

@3blue1brown

21.9M views•2017-10-05

Related Study Plans & Knowledge Roadmaps

Structured learning paths in Artificial Intelligence

View All Artificial Intelligence Topics

artificial-intelligence

Building Recommenders: Two-Tower & Vector DBs

01Foundations of Python & Machine Learning Basics

02Introduction to Recommender Systems

03Deep Learning & Representation Learning

+ 3 more learning modules

Explore Roadmap

artificial-intelligence

Latent Diffusion: PyTorch, UNet & CLIP

01Deep Learning & PyTorch Foundations

02Convolutional Networks & Autoencoders

03The Math and Mechanics of Diffusion Models

+ 3 more learning modules

Explore Roadmap

artificial-intelligence

NeRFs: 3D Reconstruction & Instant-NGP

01Math & Python Foundations for 3D Vision

02Photogrammetry & Camera Pose Estimation

03Neural Radiance Fields (NeRF) Core Concepts

+ 2 more learning modules

Explore Roadmap

artificial-intelligence

TFT Forecasting: Energy Demand & Deep Learning

01Programming & Data Science Foundations

02Introduction to Machine Learning & Time Series

03Deep Learning & Sequential Models

+ 3 more learning modules

Explore Roadmap