Train a Humanoid to Walk: Stable Baselines3 & Soft Actor-Critic

The sim-to-real gap represents one of the most significant challenges in deploying learned policies to physical robots. Policies trained in simulation often fail dramatically in the real world due to unmodeled dynamics, sensor noise, and parameter mismatches. Three complementary approaches address this challenge: (1) Domain randomization trains policies across varied simulation parameters (friction, mass, center of mass), creating robust policies that perform reasonably across different conditions but sacrificing peak performance on the training distribution; (2) Online system identification allows policies to infer necessary domain parameters from interaction data without explicit estimation; (3) Policy cloning transfers knowledge from teacher policies with privileged information to student policies using only observable states. The combination of domain randomization, online system identification, and curriculum learning enables policies that achieve 3.8 m/s on the mini cheetah—significantly outperforming previous results. Curriculum learning progressively increases task difficulty by expanding the sampling region only when success rates exceed thresholds, preventing agents from being overwhelmed by difficult tasks early in training. Adaptive curricula that consider dependencies between dimensions (e.g., faster linear velocities may allow higher angular velocities) outperform independent box curricula.

Sim-to-real transfer is a methodology for training robot control policies in simulation and deploying them on physical hardware. Five critical factors determine success: active modeling (creating accurate system representations), accurate simulation (realistic physics-based environments), accurate state estimation (correctly determining robot state), adaptive control (environment adaptation), and dynamic randomization (robustness training). The reality gap—the discrepancy between simulated and real systems—arises from communication delays, gearbox friction, lubrication effects, and unmodeled dynamics. Traditional cascaded modeling approaches fail because small errors accumulate across multiple modules. Hybrid simulation combining physics-based mechanics with learned neural networks achieves superior accuracy by preserving fundamental physical laws while capturing complex nonlinearities.

Domain randomization is a technique for bridging the gap between simulation and real-world robotics by training models on highly randomized simulated environments rather than trying to create an accurate simulator; this approach works because exposing models to extreme variation during training forces them to learn generalizable features that can adapt to the unpredictable real world, as demonstrated by successful applications in robotic grasping, manipulation, and computer vision tasks.

Deploying trained policies to real robots requires: (1) State estimator for real-world observations; (2) Trained policy model; (3) Action controller translating policy outputs to robot commands; (4) Observation extractor for feedback loops. Domain randomization during training ensures generalization to real-world conditions with sensor noise. Boston Dynamics Spot demonstrates successful deployment where simulation-trained policies transferred directly to physical hardware without modification, achieving locomotion capabilities in real-world environments.

The reality gap is the fundamental challenge where agents trained in simulation fail in real-world deployment due to differences in textures, physics, and environmental conditions. Training in simulation offers advantages: higher throughput, safety, and cost-effectiveness. Three main approaches address this: system identification (making simulators match reality), domain randomization (training across diverse simulated environments), and domain adaptation (adapting agents to real-world observations). Domain randomization trains agents across a distribution of simulated environments by randomizing visual parameters (positions, colors, textures, lighting) and physical parameters (mass, volume). A key challenge is learning object properties from single observations, solved by training simple classification networks. Training in simulation is dramatically faster than real-world training, achieving high-quality performance in hours versus years.

Optuna is a Python library for automated hyperparameter tuning that systematically searches for optimal hyperparameter values by defining an objective function, creating a study with specified parameters and ranges, running multiple trials with cross-validation scoring, and visualizing results through optimization history, parallel coordinate plots, slice plots, and parameter importance charts to improve machine learning model performance.

Optuna provides a modern, user-friendly framework for hyperparameter optimization with a clean API. Unlike other libraries, Optuna defines parameter spaces directly within the objective function using trial.suggest_* methods: suggest_categorical for categorical choices, suggest_int for integers, and suggest_float for continuous parameters. The study.optimize method manages the optimization process, tracking trials and their results automatically. Optuna supports various optimization algorithms and integrates well with different machine learning frameworks, offering flexibility for both simple and complex hyperparameter search problems while maintaining simplicity in implementation.

Optuna is a Python library for hyperparameter optimization that uses efficient search algorithms to automatically find the best hyperparameters for machine learning models by defining an objective function, creating a study, and running multiple trials; the optimization process can be visualized through parameter importance plots, slice plots, parallel coordinate plots, and optimization history to understand which hyperparameters most significantly impact model performance and how different combinations affect results.

Hyperparameter tuning is essential for maximizing machine learning model performance because library default settings are merely safe general-purpose values, not optimized for specific datasets. Optuna implements Bayesian optimization, an intelligent search method that builds a surrogate model to predict parameter-performance relationships and efficiently explores the parameter space by balancing exploitation of promising regions with exploration of untested areas. The workflow involves defining an objective function that takes a Trial object, suggests hyperparameter values, trains the model, evaluates performance using cross-validation, and returns the score. After optimization, the best parameters are used to rebuild the model on all training data, and final evaluation on held-out test data confirms the model's true generalization performance.

After establishing a baseline, hyperparameter tuning improves model performance. Optuna is a flexible Python library for conducting optimization experiments. For KNN, key hyperparameters include the number of neighbors (K, explored from 2-30) and weighting scheme (uniform or distance-based). Distance-based weighting gives closer neighbors higher importance. The video demonstrates defining Optuna trials that sample K and weight combinations, running 50 trials to explore the hyperparameter space.

Isaac Gym is a high-performance GPU-based physics simulation platform developed by NVIDIA that enables efficient reinforcement learning by performing all computational processes on the GPU, eliminating data transfer bottlenecks between CPU and GPU; this approach dramatically reduces training time—research that previously required 384 high-performance PCs can now run on a single PC—making large-scale robot learning feasible through parallelized simulation environments where each environment's state is stored as a tensor containing robot joint positions, end-effector poses, object positions, actions, rewards, and episode status.

Isaac Sim is Nvidia's simulation platform that accelerates physics simulation to 1000 times faster than real time. This enables AI agents to learn complex skills rapidly—for example, a character can learn martial arts through 10 years of simulated training in just 3 days of real-time. Additionally, Isaac Sim can procedurally generate infinite world variations, ensuring no two simulated environments are identical.

Genesis achieves a simulation speed of 430,000 times faster than real-time, compared to existing GPU-accelerated stacks like NVIDIA's Isaac Gym which achieves up to 10,000 times faster simulation while retaining real-world physics accuracy. This dramatic speed improvement enables rapid training of robotic policies on consumer-grade hardware.

Newton is an open-source, GPU-accelerated physics simulation engine built upon NVIDIA Warp and integrated with MuJoCo, designed for roboticists and simulation researchers to enable efficient, differentiable physics simulations with user-defined extensibility and OpenUSD interoperability.

Isaac Gym is a simulation framework developed by NVIDIA that enables training thousands of robots in parallel on a single GPU, achieving massive speedups over sequential approaches. Originally requiring 40 hours to train manipulation policies, Isaac Gym reduced this to just 40 minutes by exploiting GPU parallelism. The framework supports vectorized environments where actions, observations, and rewards are returned as tensors, eliminating Python overhead from individual environment calls. This parallelization approach fundamentally changes the scale at which reinforcement learning can be applied, making it feasible to train complex robotic policies that would otherwise be computationally prohibitive.

This video demonstrates how hierarchical reinforcement learning enables quadruped robots to learn complex locomotion and navigation skills from pixel inputs alone, by splitting control into high-level policies that process visual information and issue abstract commands, and low-level policies that translate these commands into precise motor actions; this hierarchical structure allows efficient learning across multiple navigation tasks while enabling knowledge transfer between different environments.

This video demonstrates how hierarchical reinforcement learning can be applied to quadruped robot locomotion by combining pre-existing low-level controllers (such as Central Pattern Generators for basic walking) with high-level RL policies that learn only small modifications to achieve complex behaviors like turning, thereby significantly reducing training time compared to learning from scratch.

DeepLoco is a hierarchical deep reinforcement learning framework that enables dynamic locomotion skills without requiring prior knowledge of physics or high-level features; it uses a high-level controller (deep convolutional network) that provides short-term footstep plans based on terrain and task goals, combined with a low-level controller (fully connected network) that generates desired joint angles to follow those plans, allowing the system to learn diverse locomotion styles, navigate challenging terrains, avoid obstacles, and perform complex tasks like soccer dribbling through reward-based training.

The common theme across projects is a hierarchical motion policy where the high-level predicts gripper actions at a higher level of temporal abstraction than the low-level. The high-level is not doing language reasoning but predicting gripper actions (sub-goals) that the low-level executes. Future directions include scaling up through large-scale simulation data and visual human demonstrations, combining learning and planning for long-horizon tasks, and tool use reasoning.

HiDe is a novel hierarchical reinforcement learning architecture that decouples high-level planning from low-level control by explicitly separating state-action spaces across hierarchy layers, enabling efficient long-horizon task solving and generalization to unseen environments through an RL-based planner that generates sub-goals using a valley map and attention mask, combined with a goal-conditioned control policy; this functional decomposition allows modular transfer of policy layers across different agents and scales to 300% larger environments while maintaining consistent success rates.