Academic Databases
Explore Our Semantic Databases
STEM & Tech
Math, Physics, CS & Engineering...
Economics
Macro, Micro, Econometrics...
Business
Management, Marketing, Finance...
Health & Life Sciences
Biology, Medicine, Neuroscience...
Humanities & Social Sciences
Psychology, Law, Philosophy, History...
General Studies
General & interdisciplinary studies...
Explore Topics
Features

Stable Baselines 3: Saving, Loading & Tracking Models in RL

Added: 2026-08-07

Deep Dive

Fundamental concepts of Reinforcement Learning (RL), including agents, environments (Gym/Gymnasium API), states, actions, and rewards.

MIT 6.S191 (2019): Deep Reinforcement Learning

MIT 6.S191 (2019): Deep Reinforcement Learning

106.4K

Reinforcement learning involves five core components: (1) Agent - the learner or decision-maker (e.g., a drone making deliveries, Super Mario navigating a game); (2) Environment - the world where the agent operates; (3) Action - a possible move the agent can make, typically chosen from a discrete subset of possible actions; (4) State - the concrete situation the agent finds itself in at any given time, defined by sensor inputs like vision, sound, and touch; (5) Reward - feedback measuring the success or failure of an action taken in the environment. Upon taking an action, the environment responds with a new state and an associated reward.

Click to watch →

Reinforcement Learning: Crash Course AI #9

Reinforcement Learning: Crash Course AI #9

244.5K

Reinforcement learning operates within a structured framework involving four key components: (1) An Agent is the AI learner that makes decisions; (2) The Environment encompasses everything external to the agent, including the world it interacts with; (3) A State represents the agent's current situation, containing all relevant information about the environment; (4) Actions are the choices the agent can make to change its state. The agent observes its state, selects an action, and receives feedback in the form of rewards or penalties. This interaction loop enables learning. For example, a robot navigating a room has states representing its position and observations, and actions representing movement directions. Rewards signal task success, allowing the agent to retrospectively evaluate which actions contributed to achieving goals.

Click to watch →

MIT 6.S191 (2023): Reinforcement Learning

MIT 6.S191 (2023): Reinforcement Learning

148.8K

In reinforcement learning, an agent is the learner/decision-maker (like a robot or game character), the environment is the world the agent operates in, actions are the commands the agent sends to the environment, states are the observations the environment returns describing the current situation, and rewards are feedback signals measuring the success of actions taken.

Click to watch →

MIT 6.S191 (2024): Reinforcement Learning

MIT 6.S191 (2024): Reinforcement Learning

123.9K

An agent is an entity that can take actions in an environment. The environment is the world in which the agent operates and provides observations back to the agent. A state (s) is a concrete, immediate situation the agent finds itself in—essentially what the agent observes from the environment. An action (a) is a decision the agent makes based on its current state. Rewards (r) are numerical feedback signals provided by the environment that measure the success, failure, or penalties of the agent's actions at each time step.

Click to watch →

Introduction to Reinforcement Learning | DigiKey

Introduction to Reinforcement Learning | DigiKey

47.8K

Reinforcement learning (RL) is one of three main machine learning subfields, where agents learn through trial and error by interacting with environments to maximize total rewards. The field traces back to Richard Bellman's work in the 1950s, gaining prominence in the 1980s through biology-inspired research. Key concepts include: the environment (what the agent interacts with), the agent (decision-making software), observations (numerical data about the environment state), rewards (signals incentivizing/punishing actions), action spaces (discrete or continuous permissible actions), and episodes (complete runs until termination or truncation). The cartpole environment demonstrates these concepts—a pole attached to a moving cart that the agent must balance upright. Practical implementation uses libraries like Gymnasium for standardized environments and Stable Baselines 3 for algorithm wrappers, requiring careful version management due to frequent interface changes.

Click to watch →

Deep Dive

Basic theoretical understanding of policy gradient algorithms, specifically Proximal Policy Optimization (PPO) and Advantage Actor-Critic (A2C).

Let's Code Proximal Policy Optimization

Let's Code Proximal Policy Optimization

17.4K

This section establishes the theoretical foundation of Proximal Policy Optimization (PPO), a reinforcement learning method known for its data efficiency among on-policy approaches. Key concepts include the actor-critic architecture where a single network outputs both policy and value estimates, the CartPole environment as a benchmark with four observations and two actions, and the necessity of collecting five data types during rollouts: observations, actions, rewards, values, and action log probabilities. The PPO trainer class is introduced with critical hyperparameters: ppo_clip_value (default 0.2), target_kl_divergence (default 0.01), and learning rates for policy and value functions. Generalized Advantage Estimation (GAE) is explained as a method for computing advantage estimates by combining multiple-step returns with discounting, forming the mathematical backbone for policy gradient computation.

Click to watch →

AI Learns To Swing Like Spiderman

AI Learns To Swing Like Spiderman

7.1M

PPO (Proximal Policy Optimization) is a deep reinforcement learning algorithm that trains actor-critic networks through iterative improvement. The algorithm uses the value function as a baseline for performance, calculating the advantage function by subtracting the value function from actual discounted returns. Positive advantages encourage actions, while negative advantages discourage them. The actor is trained by multiplying advantage values with action probabilities. A critical innovation is the clamping mechanism, which prevents the actor from changing too much between training sessions, preventing catastrophic forgetting and ensuring stable learning.

Click to watch →

Lecture 24: Advantage Actor-Critic. Trust Regions. Proximal Policy Optimization.

Lecture 24: Advantage Actor-Critic. Trust Regions. Proximal Policy Optimization.

877

This lecture covers the evolution of reinforcement learning algorithms, starting with policy gradient methods that directly optimize policy parameters through gradient ascent on expected return, and Q-learning as a value-based approach using the Bellman equation for self-consistent Q-function estimation. The lecture introduces Advantage Actor-Critic (A2C) as a hybrid approach that combines policy gradient with value-based methods by using the advantage function (A(s,a) = Q(s,a) - V(s)) to reduce variance in policy updates. Finally, it presents Trust Region Policy Optimization (TRPO) and Proximal Policy Optimization (PPO) as state-of-the-art methods that constrain policy updates to prevent destructive changes, with PPO using a clipped objective function that is simpler and more computationally efficient than TRPO's quadratic approximation.

Click to watch →

Синильщиков И.В. | "Обучение с подкреплением" | Лекция 9 | ВМК МГУ

Синильщиков И.В. | "Обучение с подкреплением" | Лекция 9 | ВМК МГУ

235

Policy gradient methods in reinforcement learning use the log-derivative trick to compute gradients of expected rewards by expressing them as mathematical expectations over trajectories, enabling Monte Carlo estimation; the Advantage Actor-Critic (A2C) algorithm combines an actor (policy network) and critic (value network) to efficiently learn policies by using advantage functions that measure how much better specific actions are compared to the average action in each state, and while A2C is an on-policy algorithm requiring fresh data per update, PPO (Proximal Policy Optimization) improves data efficiency by allowing multiple learning epochs on collected data before discarding it.

Click to watch →

Proximal Policy Optimization (PPO) - How to train Large Language Models

Proximal Policy Optimization (PPO) - How to train Large Language Models

77K

Proximal Policy Optimization combines two neural networks: the value network approximates state values using a square loss function trained on gains (actual minus predicted values). The policy network approximates action probabilities using a clipped surrogate objective function that incorporates momentum through probability ratios and prevents large updates via clipping. Both networks train simultaneously, with the value network guiding the policy toward high-value states while the policy network refines value estimates through exploration-exploitation balance.

Click to watch →

Deep Dive

Proficiency in Python programming and basic familiarity with deep learning frameworks like PyTorch.

Build a Deep Learning Project with Python & PyTorch From Scratch | Hands-on Tutorial

Build a Deep Learning Project with Python & PyTorch From Scratch | Hands-on Tutorial

61.2K

This hands-on workshop guides participants through building a deep learning project from scratch in 2.5–3 hours using Python and PyTorch. The process begins with selecting an unstructured dataset—such as images, text, or audio—from sources like Kaggle, including Chest X-Ray Images, Fruits 360, Flowers Recognition, Malaria Cell Images, Intel Image Classification, Best Artworks of All Time, and CelebA. Participants must identify the problem type: regression, classification, or generative modeling. Based on the data and objective, they choose an appropriate neural network architecture—fully connected, convolutional (CNN), or recurrent. The dataset is then prepared with batching, augmentations, and transforms. A network architecture is defined and a training loop is implemented. The model is trained and evaluated using validation and test sets. Participants experiment with different architectures, hyperparameters, and regularization techniques to improve performance. Finally, the project is documented and published via a Jupyter notebook or blog post. Additional resources include a free certification course, “Deep Learning with PyTorch: Zero to GANs,” and inspiration projects such as blindness detection, GAN-based art generation, bounding box prediction, and environmental audio classification. Datasets are accessible via Jovian’s Open Datasets GitHub repository.

Click to watch →

(Old) Recitation 0 (1/2) | Python Primer

(Old) Recitation 0 (1/2) | Python Primer

4.4K

This section covers essential Python concepts for deep learning: (1) Course prerequisites including Python syntax, virtual environments, and OOP concepts, (2) Importing modules and packages using 'import' and 'from...import' syntax, (3) Essential libraries: NumPy for numeric data, Matplotlib for visualization, PyTorch and TensorFlow as deep learning frameworks, (4) Common data storage formats: NPZ for numeric arrays, CSV for plain text, H5 for heavy data like videos, and Pickle for serializing Python objects including model weights and parameters.

Click to watch →

قسمت ۱: دوره برای چه کسانی هست؟ پیش نیازهای دوره جامع صفر تا صد 👨‍🏫

قسمت ۱: دوره برای چه کسانی هست؟ پیش نیازهای دوره جامع صفر تا صد 👨‍🏫

214

This comprehensive PyTorch deep learning course is designed for beginners in machine learning, artificial intelligence, or deep learning who want to learn PyTorch through practical, hands-on projects. The course requires basic Python programming knowledge (3-6 months experience), familiarity with data types (integers, floats, booleans, strings, lists, dictionaries), control structures (if statements, loops), and functions. Students should also have foundational knowledge of NumPy for array operations and Matplotlib for data visualization. Additionally, a basic understanding of machine learning theory is necessary to understand the code being written. The course uses Google Colab as the learning environment, which provides a cloud-based computing environment that handles system requirements and internet access issues automatically.

Click to watch →

03 Redes neuronales y aprendizaje profundo

03 Redes neuronales y aprendizaje profundo

122

Deep learning frameworks simplify implementation: (1) Low-level (NumPy, C++): maximum flexibility; (2) Mid-level (PyTorch, TensorFlow, JAX): balance of flexibility and ease; (3) High-level (Keras, PyTorch Lightning): rapid prototyping. PyTorch dominates research, while Keras provides user-friendly interfaces.

Click to watch →

Семинар. Основы PyTorch. Работа с тензорами.

Семинар. Основы PyTorch. Работа с тензорами.

20.9K

PyTorch - это популярный фреймворк для глубокого обучения, использующий динамические вычислительные графы, где тензоры (многомерные массивы) могут быть перенесены на GPU для ускорения вычислений. Основные операции включают создание тензоров с различными типами данных (float, double), изменение их формы через view(), индексацию, broadcasting, матричные операции, а также автоматическое вычисление градиентов через механизм autograd, который отслеживает операции над тензорами и позволяет вычислять производные для оптимизации нейронных сетей.

Click to watch →

Deep Dive

The general concept of model serialization (saving and loading model weights) in machine learning.

France DevOps - Quelle approche DevOps pour un projet de Machine Learning ? Avec Paul Peton

France DevOps - Quelle approche DevOps pour un projet de Machine Learning ? Avec Paul Peton

218

Model serialization converts trained models into storable formats. Two main approaches exist: (1) Package-based serialization (pickle, RDS) - saves the entire model object including structure; (2) Parameter-only serialization - saves only numerical weights and biases. Framework-specific formats (XGBoost, H2O, TensorFlow) vary in portability and file size. For very large models like GPT-3 (175 billion parameters), parameter-only serialization creates large files. Portable formats like ONNX (Open Neural Network Exchange) enable cross-framework deployment, reducing vendor lock-in and enabling easier movement between different cloud providers and environments.

Click to watch →

Intro to Practical Machine Learning: Create and Serve a Model

Intro to Practical Machine Learning: Create and Serve a Model

473

This section covers saving and loading trained machine learning models. You learn to use the pickle module to serialize (save) a trained classifier to a file using pickle.dump(). The file extension (.pkl) is just a convention. To use the saved model later, you load it using pickle.load() with the file opened in binary read mode ('rb'). This process is essential for deploying trained models in production environments where you need to use the model without retraining it.

Click to watch →

Packaging the ML Models for Production - MLOps

Packaging the ML Models for Production - MLOps

745

Model serialization (saving) and deserialization (loading) are essential for production deployment. The save_model function uses pickle (joblib.dump) to save the trained model to a specified path, constructed by joining the package root with a save model path and model name. The load_model function uses pickle (joblib.load) to load the saved model from the specified path. This enables the model to be deployed and used for predictions in production environments.

Click to watch →

Package ML Model as an Artifact for Production | MLOps

Package ML Model as an Artifact for Production | MLOps

102

Serialization is the process of converting a trained machine learning model into a file format that can be stored or transmitted. During serialization, the model is written to disk using methods like pickle.dump(). Deserialization is the reverse process where the saved model file is read back into memory using methods like pickle.load(). These processes function similarly to compression and decompression—serialization compresses the model into a storable format, while deserialization reconstructs the original model object from the saved file.

Click to watch →

Deep Reinforcement Learning Tutorial for Python in 20 Minutes

Deep Reinforcement Learning Tutorial for Python in 20 Minutes

269K

Save trained model weights using save_weights() method to files (e.g., .h5 format). To reload, rebuild the model architecture, then load_weights() from saved file. This enables deployment in production environments. The saved weights preserve learned parameters without requiring retraining, allowing model reuse across sessions.

Click to watch →

Setting Up Saving

Efficient Training

Model Comparison

Metrics Overview

Model Selection

Visualizing Agent

Learning Check

Final Training

Agent Evaluation

Setting Up Saving

0:00

Playing Section

1
Configure paths for models and logs to persist training data.
2
Use timesteps and loops to save checkpoints during the learning process.

Deep Dive

Implementing custom Callbacks in Stable Baselines 3 for advanced monitoring, evaluation, and early stopping during training.

RLVS 2021 - Day 6 - RL in practice: tips & tricks and practical session with stable-baselines3

RLVS 2021 - Day 6 - RL in practice: tips & tricks and practical session with stable-baselines3

23.5K

Callbacks extend RL training behavior without modifying core agent code. They receive events (training start/end, step execution) and access model/environment states. Key applications include checkpoint saving (periodically storing model progress), early stopping (pruning unpromising trials), and logging custom metrics. The CheckpointCallback example demonstrates saving models at specified intervals.

Click to watch →

Hyperparameter Optimization for Reinforcement Learning using Meta’s Ax | DigiKey

Hyperparameter Optimization for Reinforcement Learning using Meta’s Ax | DigiKey

3.2K

Custom callbacks extend stable-baselines3's BaseCallback class to monitor training progress, saving checkpoints periodically and evaluating agent performance. The evaluate_agent function computes average episode length, reward, and step time across multiple test episodes. Weights & Biases integration logs metrics automatically using wb.log() with commit=True to upload results. Hyperparameter derivation transforms raw parameter values into usable forms: mini-batch size as factor of total steps, steps per update as 2^N (powers of two help with computational efficiency), and neural network layer sizes with constraints (layer2 > layer1). Each trial represents one complete hyperparameter configuration executed through the do_trial() function, which sets random seeds, creates W&B runs, assigns custom loggers, derives hyperparameters, creates PPO models, attaches callbacks, trains the agent, finds the checkpoint with maximum reward, loads that model, evaluates it, logs final metrics, and returns the average episode reward.

Click to watch →

Build a Chrome Dino Game AI Model with Python | AI Learns to Play Dino Game

Build a Chrome Dino Game AI Model with Python | AI Learns to Play Dino Game

46.1K

Callbacks in Stable Baselines 3 automate common training tasks: (1) Saving model checkpoints at regular intervals (e.g., every 1000 steps), (2) Logging training progress to TensorBoard for visualization, (3) Implementing early stopping criteria. The callback is created with a save path and interval, then passed to the model.learn() method. This enables automated model management without manual intervention during long training runs.

Click to watch →

Data collection in SB3

Data collection in SB3

280

Stable Baselines 3 collects data for reinforcement learning algorithms through a fundamental agent-environment interaction loop using OpenAI Gym environments, with two main mechanisms for instrumenting this loop: wrappers (which modify the environment's reset and step methods to add processing like data logging or observation transformation) and callbacks (which are classes that inherit from a base class and are called at specific points in the training process, such as at the start of training, at each step, or at the end of episodes, allowing for logging, checkpointing, and evaluation between training periods).

Click to watch →

How to Save Best Model & Auto-Stop Training in Stable Baselines3

How to Save Best Model & Auto-Stop Training in Stable Baselines3

1.7K

Stable Baselines3 provides callback functions that automatically stop training when predefined thresholds are met, such as when the average reward reaches a target value or when the model shows no improvement after a specified number of evaluations, preventing unnecessary computation and overfitting.

Click to watch →

Deep Dive

Hyperparameter tuning for RL agents using optimization frameworks like Optuna integrated with the RL Baselines3 Zoo.

Learning to Race in One Hour (Best of Twitch) - DonkeyCar Simulator

Learning to Race in One Hour (Best of Twitch) - DonkeyCar Simulator

1.8K

This section introduces Stable Baseline 3 as a comprehensive RL library providing reliable algorithm implementations with user-friendly interfaces. The tutorial demonstrates importing algorithms, instantiating environments, and using model.predict() for action selection. It covers the RL Baseline Zoo framework for streamlined training, including hyperparameter tuning, automatic logging, and checkpoint management. The section explains how to configure training scripts with keyword arguments for algorithms like TQC, set environment IDs, control parallel evaluation, and specify checkpoint frequencies. Practical demonstrations show modifying hyperparameters on-the-fly and understanding the trade-offs between exploration and exploitation during training.

Click to watch →

Hyperparameter Optimization for Reinforcement Learning using Meta’s Ax | DigiKey

Hyperparameter Optimization for Reinforcement Learning using Meta’s Ax | DigiKey

3.2K

Hyperparameters are configuration settings controlling model training but not part of the model itself, including learning rate, batch size, and network architecture. In reinforcement learning, minor hyperparameter changes can determine whether an agent succeeds or fails entirely. Three main optimization approaches exist: grid search (exhaustive testing of all combinations), random search (sampling subsets with mathematical estimation), and Bayesian optimization (iteratively applying Bayes' theorem). The pendulum environment (Pendulum-v1) represents a classic RL challenge where the goal is swinging up and balancing a pendulum upright. Create environments using gym.make('Pendulum-v1') with render mode set to 'rgb_array' for visualization. Observations consist of three values: X and Y position of the pendulum's free end and angular velocity. A testing function evaluates trained models by resetting the environment, initializing metrics, letting the agent interact, collecting rewards, optionally recording video frames, and returning final performance metrics. Custom callbacks extend stable-baselines3's BaseCallback class to monitor training progress, saving checkpoints periodically and evaluating agent performance. Weights & Biases integration logs metrics automatically using wb.log() with commit=True to upload results. Hyperparameter derivation transforms raw parameter values into usable forms: mini-batch size as factor of total steps, steps per update as 2^N (powers of two help with computational efficiency), and neural network layer sizes with constraints (layer2 > layer1).

Click to watch →

Automatic Hyperparameter Optimization @ ICRA 22 | Tools for Robotic RL 6/8

Automatic Hyperparameter Optimization @ ICRA 22 | Tools for Robotic RL 6/8

4.3K

Practical implementation with Optuna involves: defining search space (categorical, integer, continuous variables with appropriate distributions), defining objective function with callbacks for intermediate evaluation, configuring sampler (e.g., TPE with initial trials) and pruner (e.g., median pruner with startup trials), creating the study, and calling optimize(). Key challenges in RL include noisy evaluation due to random seed dependence (solved by multiple evaluation or post-processing top trials), search space size considerations, and budget transfer (tuning on half actual budget often transfers to longer budgets). Best practices include using automatic optimization instead of manual tuning, avoiding grid search, and combining automatic optimization with manual refinement when needed. The most impactful parameters typically include learning rate, replay buffer size, and network architecture.

Click to watch →

RLVS 2021 - Day 6 - RL in practice: tips & tricks and practical session with stable-baselines3

RLVS 2021 - Day 6 - RL in practice: tips & tricks and practical session with stable-baselines3

23.5K

Effective hyperparameter tuning requires automated approaches rather than manual adjustment. Key parameters include network architecture (width/depth), learning rate, discount factor (gamma), entropy coefficient (exploration-exploitation balance), and gradient clipping (training stability). Optuna combines Bayesian optimization sampling with median-based pruning to eliminate unpromising trials early. Successful tuning typically requires 20-40 trials across multiple random seeds to identify robust configurations.

Click to watch →

Hyperparameter Optimization: This Tutorial Is All You Need

Hyperparameter Optimization: This Tutorial Is All You Need

114.3K

Optuna provides a modern, user-friendly framework for hyperparameter optimization with a clean API. Unlike other libraries, Optuna defines parameter spaces directly within the objective function using trial.suggest_* methods: suggest_categorical for categorical choices, suggest_int for integers, and suggest_float for continuous parameters. The study.optimize method manages the optimization process, tracking trials and their results automatically. Optuna supports various optimization algorithms and integrates well with different machine learning frameworks, offering flexibility for both simple and complex hyperparameter search problems while maintaining simplicity in implementation.

Click to watch →

Deep Dive

Integrating advanced experiment tracking platforms like Weights & Biases (W&B) or MLflow for comprehensive multi-run analysis.

Using W&B beyond experiment tracking

Using W&B beyond experiment tracking

2.4K

Weights & Biases (W&B) extends beyond basic experiment tracking to provide a comprehensive ML workflow platform that integrates with major cloud providers, hardware vendors, and ML frameworks; its advanced features include Sweeps for automated hyperparameter optimization with parallel coordinates visualization, interactive Tables for collaborative data analysis combining rich media with performance metrics, Artifacts for tracking data lineage and versioning model assets, and Reports for sharing live dashboards with stakeholders to track model improvements and project milestones.

Click to watch →

[ML News] AI-generated patent approved | Germany gets an analog to OpenAI | ML cheats video games

[ML News] AI-generated patent approved | Germany gets an analog to OpenAI | ML cheats video games

16.7K

Weights & Biases is a machine learning experiment tracking platform that allows researchers to track experiments, save models, recreate old experiments, share work with others, and analyze results. With a single line of code, it tracks execution runs, results, saved models, and checkpoints, uploading everything to a central profile for analysis and visualization. The platform integrates with deep learning frameworks like PyTorch, TensorFlow, and Hugging Face. A new feature called 'Tables' allows users to view datasets, outputs, and any artifacts from runs, with capabilities to sort, group, filter, and visualize data. The platform offers free personal accounts with unlimited experiments, team accounts for larger organizations, and self-hosting options for regulatory compliance.

Click to watch →

Modeling and Tracking Experiments with PyTorch & W&B

Modeling and Tracking Experiments with PyTorch & W&B

678

CinemaML uses a stack-based architecture with orchestrator, artifact store, and experiment tracker components. To integrate Weights & Biases, obtain entity, project, and API key credentials, then register the tracker using cinema CLI commands. For TensorBoard, use SummaryWriter to log metrics. Modify step decorators to specify experiment trackers, then call logging methods within steps. Both tools automatically capture model gradients and hyperparameters, enabling comprehensive experiment monitoring and visualization.

Click to watch →

Better tracking for your deep learning training - Wandb.ai (Weights & Biases)

Better tracking for your deep learning training - Wandb.ai (Weights & Biases)

54.1K

Weights and Biases is a free machine learning experiment tracking platform addressing common TensorBoard limitations. It solves problems like managing long log directory names containing hyperparameters, losing configuration details when scripts are modified, and the difficulty of comparing multiple experiments. The core integration workflow involves: importing wandb and framework-specific callbacks, initializing wandb with a project name using wandb.init(), and adding the wandb callback to model.fit(). The platform supports TensorFlow, PyTorch, Hugging Face, FastAI, Scikit-learn, and Stable Baselines 3. Hyperparameter tracking uses a config dictionary passed during initialization, replacing the practice of embedding parameters in filenames. Additional features include remote run management (stopping experiments from any device), automatic console log storage, and TensorBoard synchronization for legacy compatibility.

Click to watch →

StanConnect 2021: Ecology (part 1).

StanConnect 2021: Ecology (part 1).

517

MLflow is a framework for tracking machine learning experiments that integrates well with Bayesian workflows. An experiment in MLflow consists of: (1) logging metadata about the experiment setup (priors, likelihoods, model structure); (2) running the model (sampling); (3) computing and logging metrics (ELPD, runtime, diagnostics, R-hat values); (4) saving artifacts (model objects, diagnostic plots). This provides a principled way to track the iterative process of model refinement. The overhead of learning MLflow is manageable once installed, with the main challenge being deciding what metrics to track. The framework enables easy comparison of different model runs through its user interface.

Click to watch →

Deep Dive

Designing custom Gymnasium environments to train and deploy RL models on specialized domain-specific tasks.

Обучение с подкреплением - гибкий подход для сложных задач. Создаем собственные окружения

Обучение с подкреплением - гибкий подход для сложных задач. Создаем собственные окружения

132

This extensive section covers the practical implementation of custom reinforcement learning environments using the Gymnasium library. The lecture explains why standard environments are insufficient for real-world applications and demonstrates how to create custom environments tailored to specific problems. Key topics include: (1) Understanding the Gymnasium library structure and its role as a standardized platform for RL development; (2) Implementing the Environment class with proper inheritance and method overrides; (3) Defining observation_space and action_space using Discrete types for countable states/actions or Box types for continuous domains with bounded ranges; (4) Critical implementation details including random initialization in reset() to ensure generalization, proper handling of termination and truncation flags in step(), and maintaining consistent interface signatures; (5) Demonstrating concrete implementations for discrete state/discrete action environments, continuous state/discrete action environments, and continuous state/continuous action environments. The section emphasizes that reward function design is one of the most challenging aspects of environment creation, directly determining what behaviors the agent will learn.

Click to watch →

Distributed Deep RL on Azure ML using Ray's RLLIB and Custom GYM environments.

Distributed Deep RL on Azure ML using Ray's RLLIB and Custom GYM environments.

1.4K

Creating custom Gym environments involves subclassing gym.Env and implementing key components. For the Contoso Cabs example: the action space contains tuples representing source-destination pairs (5 cities × 5 destinations = 25 actions), while the observation space encodes state using city, hour, and day of week. The time matrix defines travel durations between cities. Key methods include __init__ (initializing parameters), step (applying actions, computing rewards, returning next state and done flag), reset (resetting to default state), and render (optional visualization). Rewards are calculated based on revenue and cost functions, with penalties for invalid actions like same-source-destination requests.

Click to watch →

Deep-Dive Into Isaac Lab Workflows | Isaac Lab Office Hours

Deep-Dive Into Isaac Lab Workflows | Isaac Lab Office Hours

9.4K

After registering an environment with Gymnasium, wrapping converts it to interfaces expected by different RL libraries (Stable Baselines3, RLlib, RoboMimic, etc.). Wrappers handle data format conversions between the environment and learning libraries. Training scripts support single-GPU laptops and cloud deployment via Isaac Automator (AWS, GCP, Alibaba Cloud). Training efficiency is demonstrated with quadruped walking trained on desktop GPU achieving competence in ~20 minutes using simple neural networks.

Click to watch →

Build a Custom Gymnasium Reinforcement Learning Environment & Train w Q-Learning & Stable Baselines3

Build a Custom Gymnasium Reinforcement Learning Environment & Train w Q-Learning & Stable Baselines3

18.9K

This comprehensive section covers the complete workflow for building a custom RL environment using the warehouse robot navigation problem. The process involves three key modules: the Problem Module defining underlying mechanics (grid-based movement, random target generation), the Wrapper Module converting this into a Gymnasium-compatible environment, and the Solution Module for training. Key implementation details include defining action spaces (discrete movements), observation spaces (state vectors with robot and target positions), implementing reset() for episode initialization, and render() for visualization. The environment is registered using gym.register() with proper ID formatting. The video demonstrates defining metadata, spaces using gym.spaces.Discrete and gym.spaces.Box, implementing required methods, and validating the environment using check_env(). Common errors like missing functions or incorrect parameters are addressed.

Click to watch →

How to create a custom Open-AI Gym environment? with codes and example

How to create a custom Open-AI Gym environment? with codes and example

4.2K

To create a custom OpenAI Gym environment, you must define a class that inherits from gym.Env and implements four essential functions: __init__ (constructor) for initializing environment parameters, reset() for resetting the environment to its initial state, step() for processing agent actions and returning new state, reward, and done status, and render() for visualizing the environment; once implemented, the environment can be registered using gym.register() with a unique ID, entry point, and arguments, allowing it to be used with standard reinforcement learning algorithms.

Click to watch →

Deep Dive

Sim-to-Real transfer techniques and deploying trained PyTorch-based RL models into production or physical robotic control systems.

Ilya Sutskever: OpenAI Meta-Learning and Self-Play | MIT Artificial General Intelligence (AGI)

Ilya Sutskever: OpenAI Meta-Learning and Self-Play | MIT Artificial General Intelligence (AGI)

To transfer policies from simulation to physical robots, train policies under randomized physics conditions without telling them how the physics was randomized. This teaches the policy to infer physical properties during deployment. The policy learns to adapt to unknown environments by trying actions and observing responses, making it robust to systematic differences between simulation and reality.

Click to watch →

Training a Unitree G1 to Walk w/ Reinforcement Learning

Training a Unitree G1 to Walk w/ Reinforcement Learning

27.7K

The sim-to-real challenge represents a fundamental barrier in robotics where neural networks trained in simulation fail catastrophically in the real world. This occurs because real-world physics, gravity, friction, and environmental conditions create out-of-distribution data that neural networks struggle to handle. Successful transfer requires addressing two critical controller design issues: implicit PD controllers in physics engines require privileged data and future information, creating gaps between simulation and reality; explicit PD controllers implemented as Python/Torch code ensure identical control logic across domains. Additionally, linear velocity estimation becomes necessary since real robots lack direct velocity measurements. Before attempting real-world deployment, researchers must first achieve successful sim-to-sim training using identical code and hardware interfaces to identify pipeline inconsistencies.

Click to watch →

Josh Tobin — Productionizing ML Models

Josh Tobin — Productionizing ML Models

3.5K

Domain randomization addresses the sim-to-real transfer problem by massively randomizing all non-critical aspects of physics simulators during training—colors, object positions, camera positions, backgrounds—to produce unrealistic-looking data. The key insight is that when neural networks must solve tasks across diverse randomized conditions, they learn to ignore superficial details and focus on invariant features that transfer to reality. This technique has been successfully applied to object manipulation, fish detection, cloth folding, and robotic hands solving Rubik's cubes. The Rubik's cube demonstration illustrates both progress and limitations: achieving basic functionality (solving the cube approximately 20% of the time) represents progress, but the system relied on hardcoded solvers while neural networks only controlled finger movements. This highlights the gap between impressive demonstrations and reliable production systems.

Click to watch →

Control Legged Robots Reinforcement Learning, Jemin Hwangbo, KAIST, PhD at ETH 2019 Marco Hutter

Control Legged Robots Reinforcement Learning, Jemin Hwangbo, KAIST, PhD at ETH 2019 Marco Hutter

835

Sim-to-real transfer enables training robot control policies in simulation and deploying them on physical hardware. Five critical factors determine success: active modeling (creating accurate system representations), accurate simulation (realistic physics-based environments), accurate state estimation, adaptive control, and dynamic randomization. The reality gap—the discrepancy between simulated and real systems—arises from communication delays, gearbox friction, lubrication effects, and unmodeled dynamics. Traditional cascaded modeling approaches fail because small errors multiply across multiple modules. Hybrid simulation combines physics-based Newtonian mechanics (fundamentally correct) with learned neural networks to capture complex actuator dynamics. Physics engine selection matters: hard contact simulation prevents surface penetration providing consistent behavior, while soft contact requires parameter tuning and produces inconsistent results. Rising physics engine is widely adopted for its hard contact capabilities and reinforcement learning utilities.

Click to watch →

Lecture 22 Sim2Real and Domain Randomization -- CS287-FA19 Advanced Robotics at UC Berkeley

Lecture 22 Sim2Real and Domain Randomization -- CS287-FA19 Advanced Robotics at UC Berkeley

10.3K

Domain randomization is a technique for bridging the gap between simulation and real-world robotics by training models on highly randomized simulated environments rather than trying to create an accurate simulator; this approach works because exposing models to extreme variation during training forces them to learn generalizable features that can adapt to the unpredictable real world, as demonstrated by successful applications in robotic grasping, manipulation, and computer vision tasks.

Click to watch →

Setting Up Saving

Efficient Training

Model Comparison

Metrics Overview

Model Selection

Visualizing Agent

Learning Check

Final Training

Agent Evaluation

Setting Up Saving

0:00

Playing Section

1
Configure paths for models and logs to persist training data.
2
Use timesteps and loops to save checkpoints during the learning process.

50.1K views89833:06@sentdexOriginal Release: 2022-02-06

In Stable Baselines 3, you can save and load trained reinforcement learning models using the `model.save()` method with a filename, and load them back using the appropriate algorithm's `load()` function (e.g., `PPO.load()`), which is essential for tracking model performance, resuming training, and deploying trained agents; additionally, TensorBoard logging provides visualization of training metrics like reward averages and episode lengths to monitor algorithm performance over time.

[00:00:00]what is going on everybody and welcome to part two of the reinforcement learning with stable bass lines three tutorial series in this video we're going to be talking about saving loading and tracking the performance of your models so this is the code that we left off on we definitely learned that something is happening when we train it for x number of steps the problem is how many steps do we train for and as we saw for sure with a2c i didn't even look through the ppo but we definitely saw where performance at one point was actually in the positives but it ended at 100 000 steps with a reward in the negatives so how do we like deal with this so so that's what we're going to be focusing on uh here so first off uh we will go ahead and we will import os and then we're going to specify a models directory and this will just be we'll make models models slash and in this case depending on which one we're using so in this case we're using ppo so first for now i'll just say ppo but if we change the algorithm change change that name don't be like me and forget i'm sure i will so that's modelsdirt and then we'll also have a loggeder for whoops i didn't mean to run that i have to what what have i done logder equals logs so uh so once we've specified those we want to make sure they exist so if not os dot path dot exists and models der we want to os dot make durs models der and then we're going to do the exact same thing for the log directory so paste paste cool so that'll make them if they don't already exist these lines are totally fine and now what i'd like to do is set time steps so this could be in the form of a thousand time steps or ten thousand it really is going to depend on like how quickly does your environment step in my opinion so this just comes down to how free essentially how frequently do you actually want to save your model so for now i'm going to also comment this out because this was to actually run it and we'll get to that on the load here in a little bit but for now what we want to address is training and saving so so in this case let's say we're going to train every ten thousand steps you can change this to be five thousand ten thousand two thousand five hundred whatever whatever the heck you wanna use um put that number in i'll go with ten thousand since that's relatively quickly so let's say what we really want to do and we'll actually make a constant here time steps we'll say 10 000.

[00:02:41]so model learn for those time steps and then we'll say for i in range and then how many time steps do you want to do so in reality we might actually make this a wow true loop but just so it'll be easy for me to kind of compare the two well again we'll be comparing a2c and ppo here so i'll just limit it in some way but for i in range 30 what we're going to do is model.learn for that many time steps now if we just leave it this way the model will reset every time the number of time steps that it's like already taken the model object i mean so every time you do model.learn it would reset the time steps that it's at but to stop that we can say reset num time steps and we'll set that to be false and then we'll also set a tb log name and again that will set that to be ppo um i feel like we need to have like a constant probably um i think we'll be okay with with two places to edit but if we add any more i think i should we should probably have a variable for that but anyways whatever model.learn uh so yeah so the so this reset time steps will help both with the output that we see in the console but also in the tensorboard itself so model.learn and then basically all we're going to add is model.save and we'll use an f string here models underscore oops we wanted to use those variables models dir slash and then in this case it will be uh we're just gonna save uh time steps times i so we'll say um whoops not a capital i dude there we go so go ahead and save that so now essentially this will be time step zero uh the first time through and in fact maybe we'll do a range one to thirty to make this actually actually accurate otherwise we'll be ten thousand steps off so model that save so now it will save that model every 10 000 steps so i'll go ahead save that and in fact i think we probably want to run this in its own little console so we'll go ahead and run this python3 p2 uh save dot pi let's make this a wee bit larger and let's go ahead and run that make sure we don't hit any errors so far so good okay and that's off that is training uh our model for ppo now what i'm gonna go ahead and do is let's just shift uh i'll put this over here i guess so i'll move that over i wish it told you somewhere in that output what type of model it was but that's okay so we got ppo training now i'm going to go ahead and toss in uh a2c to kill two birds one stone at a time here so let's make sure that we make all the changes that we're supposed to make ooh find and replace probably would have made sense but that's okay uh a2c uh a2c wasn't there one more spot yes there was not that it'd be an interesting model uh okay so i think i think we're all set we'll go ahead and save that let's confirm that uh we got models or logs so this will be tensorboard here models ppo so yeah we have our ten thousand twenty thousand and thirty thousand steps models already in there um whoops and i think we're good so let's just double check make sure i didn't screw anything up in fact let's go ppo found that one it's a good thing we didn't do a find and replace isn't it anyways okay cool so now whoops now we'll save and let's go ahead and i will run that one as well so open in terminal python 3 p2 save dot pi so again this should be running for a2c in our case so again we'll make it a little bigger and i will wait for that to at least start to output something for us let me check a2c cool and let's go ahead and check logs logs is empty why would logs be empty is it possible why would logs be empty maybe it's every hundred thousand steps is that what it's going to do to me i would think it would update the logs every 10 000 steps oh i know what i've done okay so i forgot to in model i told you we weren't going to be adding anything else but we are so so in here we're going to say tensorboard tensorboard underscore log equals log der and then this is the tb log name dang it okay so okay i'm gonna break these two that are training right now uh and come back over here i just want to clean everything up so actually i'm just going to delete that okay coming back here a2c a2c cool i wish i would have checked that earlier anyways let's rerun that so this will be a2c going and we'll make sure that we're actually logging this time okay that's training we'll come over here well let's confirm before we waste our time shall we so a2c underscore zero cool models a2c very well okay so now what i want to do is switch this to ppo copy paste paste paste and just to be sure a2c very good we don't have a2c and will we run ppo now i'll slap this over here also a little curious um nvidia s am i curious where i'm at gpu-wise okay my gpu yeah we're comfy uh thanks puget so um okay so now we have our things training we should have tensorboard logging happening so now to actually view the tensorboard what we'll do is uh do actually we need to be in that directory so we'll come down here open a terminal and we'll go uh tensorboard logder logder equals logs very well so okay so here although the graphs are very very small admittedly let's make this a little bigger so the main graph that you're probably going to care about the most is the reward average okay um and in this case ppo is our blue and a2c is our orange so you probably care about the the average reward the most that is typically the thing that we're trying to create here but the length of the episode also can carry quite a bit of significance um and maybe more on that later uh we'll see so ppo blue a2c is orange okay so what i'm going to do i think is i'll make i think i can do this no that's the wrong button i never remember which one which ones are the right buttons so i'm going to let these things train for the 300 000 steps hopefully i got that right it should take somewhere before that we should be able to solve these models these are very simple models okay obviously this is still training and i will continue to let it go for the 300 000 steps but eventually we'll get to 300 000 and either way while you're training like i said typically you're going to probably be using a wow true loop rather than you any known ending so now what we want to do is looking at our reward mean we might say okay we want to take ppo at you know maybe i don't know which one will we do 180 or maybe even 175 but we wouldn't have that saved so it'd either be 170 000 or 180 000 either one we can just grab one of those let's say we want to load that and run that also interestingly enough i'm kind of surprised that uh we've actually seen we actually see a2c overtaking ppo this is not that's not nothing is what i've expected so far in this series like i expected um our random agent in part one to be way more wild but actually our random agent part one was i think better than the 100 000 age that's just like what the heck man trying to make a tutorial here um let's go look at some of these metrics though so one thing that should be pretty clear by looking at these um they are different algorithms so take these take this with a grain of salt but in general the reality or the thing that i was hoping to be able to show even with the reward here is but you can also see it with the reward um ppo is a smoother algorithm in general it does it doesn't have as much volatility and one thing you will grow to really hate as you train these reinforcement learning algorithms is how volatile they can they can become right um especially like when i was doing ddpg for example with the robot dog um it was very frustrating to have a model learn learn learn learn learn and then just tanks it just explodes and it's like no good anymore it's like it just it really guts you so um in general i like something that is a little smoother and we can see that basically in all of these other metrics like entropy loss all the all basically all of these metrics across the board um ppo is the smoother metric and so for that reason it is a little more less stressful to use something like ppo also in general the performance is much better in this case it is comical to see a2c is actually is it still winning it is still winning real hard okay live demos so the other thing to think about too is a lot of models can be dictated by just randomness so where did you randomly initialize and where did you start taking steps from there so you can do two things you can either set your random seed specifically or i kind of just like to just run a few models randomly like to test truly test a model test your environment i would run like three to five something like that i've seen huge variation in exactly the same code and if and in fact maybe what i'll do before we leave it'll probably be an exact match but before the end of this this tutorial um i'll run one more ppo and i mean in fact we can just run it um i think we can get away ppo maybe make the models dirt could be a dash two and actually i don't want to overload the gpu so i'll wait until one of them is done but i'll show you at least another tensorboard before this is over um and you can hopefully see some of the variants my guess anyways these environments so basic that maybe there won't be much but anyways let's get to loading and kind of seeing one of our examples in action so for example let's say we want to take ppo and i forget already what was a good ppo one which one is ppo it's our blue so let's take ppo at 180 000.

[00:14:52]so while things are training we probably want to see where we're at along the way because again you'll probably wind up doing it for just a while true loop and you might want to actually visually see it so sometimes you'll see the reward is what you want or maybe the episode length is what you want but nothing beats visually seeing it so again for example with the robot dog sometimes the reward looks really good but then you load it up and either one the uh the actual agent is somehow defying the laws of physics like it has found an actual like bug in the environment which is pretty common or two maybe it's doing something really dumb that will never be recovered from so one example that i've seen again from the robot dog scenario is it would like jump and do like this kind of roll but get stuck on its back and it just it was probably never going to get out of that that was a really short term kind of gain in reward but then actually recovering from that i've never seen it happen it's possible that it could like jump and roll and then jump and roll and then jump and roll or something like that but um in general it's good to see it visually so you have an idea of like what's actually going on in your environment uh to cause that reward even though it might look good on paper in practice it might not look good at all so so we want to see what's going on so to do that we still need our environment we do need a model but we're going to use our one of our saved models so i'm going to copy the models dir paste it over here and the first thing that we want to do is we're going to specify our model path model path and that will be um that will be the uh we'll make that an f string and it will be models der and then slash whatever that file name is in this case we'll do 170 000.zip that one was looking good at least to start and now what we want to do is actually load that model so we're going to say model equals and this will be ppo dot load and we will load in the model path and then we specify the environment so to just be m vehicles m now we want to actually run with the model so episodes 10 that's totally fine we get the observation yeah yeah cool cool basically what we want to do is swap out this action here so we're going to cut this out and we're going to say we want to take an action now what is that action going to be so to make that action we're going to say action equals model dot predict and actually uh it's gonna be action and then we'll just say like underscore it in this case it also the model that predict is returning um some states so if you are using we're not i'm not touching this right now but if you're using a recurrent algorithm this will this will be of use to you um but for now we're not doing that so we really only care about what is the action so we're going to say models.predict or model.predict rather and that observation so then we'll do the step based on our actual model's action we will get all of this other information um and then basically continue the loop so let's go ahead and save that and we can run that maybe yes okay so here we have i guess because i've changed the name it's all itty bitty let me fix that beautiful so here we can see our actual model in action this is our ppo model um so the episode length is starting to make a little bit of sense now it's not terrible right it's not crashing but the goal is to land in between these two flags and it's definitely not doing that right right now but this is better than random and it's better than the 10 000 model but clearly we need to possibly continue training so let's go ahead and check let's see here if anybody got even better no our best model in this case was actually around maybe 190 but also interestingly ppo and e2c performed identically it's this is so frustrating yeah you do like definitely ppo is supposed to win here i can't believe this and ppo also just turned to to crud by the end this is hilarious um so what i think i'm gonna do now is i'm gonna train i'm gonna train like i guess we'll do we'll do another run for each of them to be fair so i'll run another ppo and another a2c but also while we're at it let's go ahead and check an a2c at i guess 190k so we'll come over here we will oh it already died it died before i could do it so we'll change this to 190 and then we'll change this to a a2c we'll change this to a2c a2c cool uh and let's just go ahead and import ppo and a2c we don't need to be switching that around every single time cool cool run that wait for it so here's our a2c it looks pretty good too it's very comparable to our i think that i don't think we ever saw ppo actually succeed in any way this is interesting when i was like making the text based version of this tutorial this is this is like nothing like i was seeing it's funny um classic so okay so now what i want to do is i'm going to train both of these again and maybe maybe i'm going to walk away while i train these so i'm going to do maybe i'll do like a million steps just because i am surprised at our results thus far so cool so what i'm going to do is we'll just kill this and p2 save let's do models do ppo so the only thing if i want to run a bunch of ppos would be here and maybe what i'll do is import time let's get real fancy about this super fancy i will do an int time dot time cool and [Music] 30 let's do do i really want to do a million times kind of i kind of want to see the a better result here what was it so uh ten thousand so we'll do a hundred time steps and i think maybe i'll do like two and two so let's save that p2 save so what i'll do is come over here i'll run p2 save once okay it was trying to rewrite to the uh the other tensorboard logs so i guess we'll also say logs and then we'll do this logs slash ppo time.time what did i screw up ah boom okay try that again what i really want to see like it was trying to overwrite each other so here i'll show you what i what tipped me off to that one okay well maybe it was this one i might have made the logs a little messy but yeah i saw logging to logs and then it was just ppo underscore zero for both of them obviously we don't want that um because then we'd have crazy logs so hopefully we didn't totally murder well we did we murdered our other log but that's okay um that's all right okay that one is running so i will run one more uh ppo and just to clean up the logs i will come into here and i will delete maybe we'll leave the original a2c but this one's been ruined so i'll just delete it and now what i'll do is i'll also add an a2c take that pasta pasta a2c a2c i bet we could have fixed it with that we could that would have been the more proper place to fix it actually now that i think about it we should have added the time to that uh i don't think it will be relevant because it just doesn't it doesn't really matter but this we should have added the time stamp right there um whatever i think we're fine so let me check for ppo again cool okay so now what i will do we'll just run two more of these python 3 p2 save and i'm just going to open a tab python 3 p2 save let's check models make sure we're not screwing ourselves there look very good and then we'll check logs that's looking good so far and then finally let's check tensorboard looking good looking good i wish i didn't delete the original ppo oof whatever okay so see if i can fit this on the screen while it trains very good very good okay about an hour in some change hour and 20 30 minutes or so has gone by we are finally done training everything here and um interesting charts for sure so checking out kind of the results it does look like our best performer is this ppo model here but also curiously i believe that's this guy right here yeah so also curiously where's the talk there it is um who's our second best here see like our second best was like sort of even this like a2c for a second that for the original one that i said was really good for an a2c model by the way um that was actually our second or our uh second best for a little bit and then it appears that it kind of went bad oh well that was the well that was the one that we trained for only 300k steps um but then it's another a2c model that was actually pretty good for a little bit and then yes it throws it all away just just like the other one was clearly going to do in it and it never recovered but by the end uh the the two victors here are ppo and ppo so if you can use ppo use ppo it seems to be the best one that i've you know at least in my experience of using in in all fairness a2c i don't know anybody that's still using a2c for anything specific but anyways um yeah so we have our results we've compared models um you've seen now real examples of how that random initialization can make all the all the difference um so again you might want to actually explicitly set your random seed possibly or just simply train a handful of models in my opinion i would just again i would just kind of randomly train a few models and then whichever one is going well continue training that model i don't i don't really see any statistical reason why you want to set like random seed 0 for example for all the algorithms that you test because in theory you might be you know maybe you were that that bad ppo from before uh when really you could have been you could have been good ppo if you would have allowed that seed to you know come across your desk so anyways uh let's check out one of these ppos so it looks like um maybe around here is one of the better ones so we are where we step so let's do the 550k ppo so let's check that one out so p2c p2 load uh i already know i've i gotta figure out what was the uh 550 000 and which before i forget so let's see pb let me change this to be ppo um and which one was that ppo uh is there a way i can just yes i'll do that maybe cool come over here paste i think we can get away with that with the models there we'll save that let's run that and see if that works beautiful look at it look at it go yeah so yeah that's uh probably solved we'll look at a few more but i would say at this point yes that is indeed um that is indeed solved so the other thing that i wouldn't mind checking real quickly while that continues to play is what was our best a2c it looks like a2c like our original a2c is is the best one that went for 300 000 steps so the a2c at about not quite 200k steps but maybe 190.

[00:30:02]yeah so a2c0 at 190k looked pretty good so let me um i guess i'll move this over here you can watch that and we'll just check it out so a2z a 2c 0.

[00:30:20]is that going to be right for the modelster let me check that that'd be right for the logs but for models yeah it's just a2c i already forgot what i said was it 190k okay so this is our a2c um yeah i mean it's not it's not quite as good as the ppo it did learn at least but at least for sure on this environment i see no reason to use it ppo is faster and ppo is better so hopefully now you've got an even better understanding of reinforcement learning stable baselines three and you're coming to grips with how do you change your algorithms and all that now up to this point we've been using these pre-made environments and like i've been saying the really true hard thing is to figure out which environment do you want to use ppo on so so i'm just kidding there are other algorithms and obviously ppo is no good for there is continuous ppo but again we will we'll talk more i promise after the custom environments we will talk more about uh continuous versus uh discrete and all that kind of stuff um but i think the next best thing to do is i think anybody is getting into reinforcement learning if you're not just doing it just for the kicks is you want to apply this to your own environment so in the coming videos what i'm going to do is we'll just take some environment i can find online um i'm thinking some sort of pie game game or something like that convert that to a a gym environment so show you kind of the steps that are involved to do that and then actually train a model in that environment again because this is an environment that we're going to have to make up the observations so you'll see that it's not you know in this case we've just been handed observations so how do you come up with your own quality observations how do you start thinking in that way and then also how to come up with your rewards because again these two things are actually deceptively complicated sometimes so i think i think you'll see a few good examples as we as we go through that so anyways that's all for now thanks for watching and i will see you guys in another video you

Up Next

Create Your Own Reinforcement Learning Environments: Gridworld Tutorial

@MachineLearningwithPhil

39.1K views•2019-04-02

Building Real-Time ML Pipelines with Feature Stores and MLOps Frameworks

@ODSCAI

5.1K views•2022-02-20

Building a Custom Reinforcement Learning Environment with Q-Learning

@sentdex

116.5K views•2019-06-06

Neural Networks Explained: Math, Layers, and Learning Fundamentals

@3blue1brown

21.9M views•2017-10-05

Related Study Plans & Knowledge Roadmaps

Structured learning paths in Artificial Intelligence

View All Artificial Intelligence Topics

artificial-intelligence

Building Recommenders: Two-Tower & Vector DBs

01Foundations of Python & Machine Learning Basics

02Introduction to Recommender Systems

03Deep Learning & Representation Learning

+ 3 more learning modules

Explore Roadmap

artificial-intelligence

Latent Diffusion: PyTorch, UNet & CLIP

01Deep Learning & PyTorch Foundations

02Convolutional Networks & Autoencoders

03The Math and Mechanics of Diffusion Models

+ 3 more learning modules

Explore Roadmap

artificial-intelligence

NeRFs: 3D Reconstruction & Instant-NGP

01Math & Python Foundations for 3D Vision

02Photogrammetry & Camera Pose Estimation

03Neural Radiance Fields (NeRF) Core Concepts

+ 2 more learning modules

Explore Roadmap

artificial-intelligence

TFT Forecasting: Energy Demand & Deep Learning

01Programming & Data Science Foundations

02Introduction to Machine Learning & Time Series

03Deep Learning & Sequential Models

+ 3 more learning modules

Explore Roadmap