Training Deep RL Agents: SB3 & Custom Envs

Learning Goal: Develop and train deep reinforcement learning agents using Stable-Baselines3 to solve custom simulated decision-making environments.

Prerequisites: Basic mathematical understanding (high school level algebra and introductory probability) and elementary Python programming.

Estimated Total Study Time: 40 Hours

---

Module 1: Programming & Mathematics Foundations

To build functional and mathematically sound reinforcement learning agents, you must first master Python scripting, linear algebra, and gradient-based optimization. This module establishes a strong foundation in programmatic structures and the mathematical frameworks that underpin neural network parameter updates.

Recommended Videos

Why this video is valuable: Deep reinforcement learning requires writing clean, debugging-friendly Python code. This comprehensive tutorial takes you from absolute beginner fundamentals (variables, logic, loops) to advanced design patterns like Object-Oriented Programming (OOP) and class inheritance. Understanding classes and inheritances is critical because custom environments must inherit from the Farama Gymnasium base class.

Why this video is valuable: Machine learning relies heavily on linear algebra (vectors, matrices, dot products) and calculus. This video condenses these complex mathematical concepts into digestible lessons, providing the visual intuition behind dimensions, matrix transformations, and scalar multiplication—the operations that process observations and outputs within your reinforcement learning neural networks.

Why this video is valuable: Reinforcement learning agents use neural networks as functional approximators. This classic 3Blue1Brown animation explains gradient descent—the backbone of neural network training. By visualizing cost surfaces and backpropagation, you will intuitively understand how neural network weights and biases are iteratively updated to minimize error and optimize agent policies.

Module 1 Knowledge Checkpoint

• Implement conditional logic, loops, custom function definitions, and basic class structures with inheritance in Python.
• Describe how high-dimensional observation vectors are mathematically mapped to actions via matrix multiplication inside a neural network.
• Explain the concept of gradient descent and how step sizes (learning rates) influence the optimization process of a model's weights.

---

Module 2: Introduction to Reinforcement Learning Basics

This module transitions you from static machine learning into the dynamic world of sequential decision-making. You will learn the core framework of Reinforcement Learning: the interaction loop between agents and environments formalized through Markov Decision Processes (MDPs), alongside basic tabular methods such as Q-learning.

Recommended Videos

Why this video is valuable: UC Berkeley's CS188 lecture provides an academic and intuitive breakdown of Markov Decision Processes (MDPs). It introduces how states, actions, transition dynamics, and discount factors ($\gamma$) establish a mathematically sound way of representing environments with stochastic or deterministic outcomes.

Why this video is valuable: This video breaks down Q-learning step-by-step. It explains how agents maintain and update a lookup table (Q-table) of quality values using the temporal-difference (TD) error and the fundamental Bellman equation. This serves as the conceptual stepping stone to Deep Q-Networks (DQN).

Module 2 Knowledge Checkpoint

• Define the five major components of an MDP: State ($S$), Action ($A$), Transition Probability ($P$), Reward ($R$), and Discount Factor ($\gamma$).
• Explain how the Bellman Equation computes state-action values ($Q(s, a)$) by balancing immediate rewards with discounted future rewards.
• Describe the difference between exploration (trying unknown actions) and exploitation (using known lucrative actions) using an $\epsilon$-greedy strategy.

---

Module 3: Deep Reinforcement Learning Algorithms

When environments scale to millions of states, traditional Q-tables become computationally impossible to store. This module explores Deep Reinforcement Learning, focusing on how neural networks approximate value functions (DQN) and policy gradients (PPO) to tackle continuous, high-dimensional spaces.

Recommended Videos

Why this video is valuable: This lecture introduces Deep Q-Networks (DQN). It highlights the core issues that arise when combining neural networks with reinforcement learning (such as non-stationary targets and correlated data) and explains how experience replay and target networks stabilize training.

Why this video is valuable: PPO is the industry workhorse of policy gradient methods. While we will be using Stable-Baselines3's implementation, walking through a full raw PyTorch implementation of PPO exposes the inner mechanics of actor-critic structures, importance sampling ratios, and surrogate objective clipping.

Why this video is valuable: This high-level, clear conceptual lecture walks through the mathematical motivation of PPO. It visualizes how clipping prevents the agent from making destructively large policy updates during training iterations, guaranteeing monotonic improvements.

Module 3 Knowledge Checkpoint

• Explain how a DQN differs from tabular Q-learning and how an experience replay buffer breaks temporal correlation in training batches.
• Contrast value-based methods (DQN) with policy gradient methods (PPO).
• Describe the purpose of the clipped surrogate objective function in PPO and why it prevents performance collapses during optimization steps.

---

Module 4: Getting Started with Stable-Baselines3

Stable-Baselines3 (SB3) is a powerful, reliable set of reinforcement learning implementations built on PyTorch. This module teaches you how to instantiate, train, save, load, and benchmark standard SB3 models using built-in environments.

Recommended Videos

Why this video is valuable: This is a clean, practical introduction to utilizing Stable-Baselines3. You will learn the code syntax required to set up an environment, instantiate algorithms (like PPO or DQN), run the training steps via a single execution line, and analyze basic training output logs.

Why this video is valuable: Real-world training can take hours or even days. Learn how to save your trained weights, reload models from disk for continued training or deployment, and extract logs to run evaluations on the agent's performance.

Module 4 Knowledge Checkpoint

• Install Stable-Baselines3 alongside Gymnasium inside a virtual environment.
• Write a script to instantiate a default SB3 PPO model, train it on CartPole-v1 for 10,000 timesteps, and evaluate its average reward.
• Save a trained policy to a local directory as a .zip file and reload it to perform inference in real-time visualization steps.

---

Module 5: Designing Custom Gymnasium (Farama) Environments

Most real-world optimization problems do not have ready-made pre-built environments. This module focuses on building custom simulations using the modern Farama Foundation Gymnasium API, defining state representations, continuous/discrete action spaces, and custom reward boundaries.

⚠️ API Transition Note (Legacy Gym vs Farama Gymnasium): Many historical internet tutorials refer to the legacy OpenAI gym API. In 2022, maintenance transitioned to the Farama Foundation under the name gymnasium.

The most critical change is the signature of the environment's step() method.

• Legacy Gym: Returned 4 variables: obs, reward, done, info
• Modern Farama Gymnasium: Returns 5 variables: obs, reward, terminated, truncated, info
  • terminated is True if the agent reaches an end state (e.g., dying, hitting a goal).
  • truncated is True if an external limit (like a step/time limit) is reached.

Ensure your custom classes always implement the 5-return structure!

Recommended Videos

Why this video is valuable: This video walks through implementing a custom Grid/Snake-style simulation environment under Gymnasium. It shows you how to design explicit observations and map actions directly into your custom state machine.

Why this video is valuable: This tutorial emphasizes state tracking and modular execution of actions. It helps you design code that tracks environment resets, updates states internally based on agent choices, and computes immediate rewards.

Module 5 Knowledge Checkpoint

• Define the inheritance of a custom environment: class CustomEnv(gym.Env): and implement the abstract methods __init__, reset, step, and render.
• Declare appropriate observation and action spaces using Gymnasium types (spaces.Discrete, spaces.Box, spaces.Dict).
• Correctly implement the modern 5-value return inside your step() function: return self._get_obs(), reward, terminated, truncated, info.

---

Module 6: Training, Tuning, and Evaluating in Custom Envs

The final step is connecting your custom Farama Gymnasium environment to Stable-Baselines3. You will learn to validate your environment for standard conformity, vectorize training across multiple CPU cores to accelerate sample efficiency, and automate hyperparameter tuning.

Recommended Videos

Why this video is valuable: Reinforcement learning is highly sensitive to hyperparameters (learning rate, batch size, clip range, network architecture). This video demonstrates how to systematically automate parameter evaluation to discover optimal agent configurations.

Why this video is valuable: A quick look at a highly complex custom learning environment (a robotic walking dog). It provides visual proof of how custom state constraints translate back to physics engine step execution under SB3 training loops.

Gap Solutions & Deep Dives

To address specialized gaps in available video guides, study the implementations below to validate and scale your environments.

Environment Validation (Using check_env)

Before sending your environment to an SB3 algorithm, you must validate its space dimensions, datatypes, and boundaries. Stable-Baselines3 provides an internal checker tool:

from stable_baselines3.common.env_checker import check_env from my_custom_env import CustomEnv

Instantiate your custom Gymnasium environment

env = CustomEnv()

check_env will verify that:

1. Action/Observation spaces match return shapes

2. step() returns 5 elements (obs, reward, terminated, truncated, info)

3. reset() returns (obs, info)

4. Values stay within lower/upper boundaries

check_env(env, warn=True) print("Environment checked successfully!")

Vectorizing Environments for Speed

Deep RL algorithms require millions of samples. Instead of running one simulation, you can run multiple environments concurrently on separate threads or processes.

• DummyVecEnv: Runs environments sequentially on a single thread (useful for debugging, light computations).
• SubprocVecEnv: Runs environments in parallel on individual CPU cores using multiprocessing (crucial for scaling heavy simulations).

from stable_baselines3 import PPO from stable_baselines3.common.vec_env import SubprocVecEnv, DummyVecEnv from my_custom_env import CustomEnv

def make_env(): return CustomEnv()

if name == "main": # Create 8 parallel instances of your environment num_envs = 8 env = SubprocVecEnv([make_env for _ in range(num_envs)])

# Train PPO using the vectorized parallel environments model = PPO("MlpPolicy", env, verbose=1, tensorboard_log="./tb_logs/") model.learn(total_timesteps=100_000) print("Parallelized training complete!")

Module 6 Knowledge Checkpoint

• Run the check_env diagnostic script on your custom environment and resolve all returned system warnings and errors.
• Implement DummyVecEnv and SubprocVecEnv wrappers to scale up simulation collections during training.
• Connect TensorBoard logs to monitor training trends (specifically ep_rew_mean and loss curves).

---

Course Map

---

Key People Index

• Richard Sutton: Professor at University of Alberta and Distinguished Research Scientist at DeepMind. Widely considered the "father" of modern reinforcement learning; co-author of the seminal textbook Reinforcement Learning: An Introduction.
• David Silver: Professor at UCL and Principal Research Scientist at Google DeepMind. He led the historic AlphaGo project that defeated world champion Lee Sedol, proving the massive potential of Deep RL.
• John Schulman: Co-founder of OpenAI and primary author of the Proximal Policy Optimization (PPO) paper. His research enabled stable policy updates across various domains, including robotics and LLM alignment (RLHF).
• Antonin Raffin: Lead maintainer of the Stable-Baselines3 library and Research Engineer at DLR. His contributions make deep reinforcement learning highly accessible, structured, and reproducible for global researchers.

---

Final Self-Assessment

Complete this comprehensive checklist to verify your proficiency in Deep Reinforcement Learning.

• Python Foundations: You can write class-based structures that cleanly leverage Python inheritance concepts.
• Math & RL Foundations: You can explain how an environment's state transition function $P(s' | s, a)$ influences action optimization strategies.
• DQN vs. PPO: You understand the operational difference between value-based neural approximations and direct actor-critic policy mappings.
• Farama Gymnasium API: Your custom environments fully conform to Gymnasium patterns, utilizing correct 5-variable step() returns (terminated and truncated values separated).
• SB3 Environment Validation: Your custom environment successfully passes the check_env utility suite without warnings or errors.
• Vectorization: You can wrap custom environment structures in both DummyVecEnv and SubprocVecEnv layers to scale training metrics.
• TensorBoard Analysis: You can track agent progress using local TensorBoard sessions, confirming learning through an increasing average episodic reward curve over training steps.
• Hyperparameter Strategy: You can articulate how learning rates, clip bounds, and discount factors can be optimized to stabilize policy updates.