Robotic Leg RL: PyBullet Sim to Hardware

Learning Goal: Train a 2-DOF robotic leg to perform dynamic jumping in a PyBullet physics simulation using Deep Reinforcement Learning (PPO) and execute the policies on physical hardware.

Prerequisites

• Basic understanding of high-school-level physics (forces, torque).
• Familiarity with Python programming concepts (loops, functions, lists).
• Comfortable with basic command-line navigation.

Estimated Total Study Time

75 Hours (including video instructions, independent coding, simulation training, and hardware assembly/troubleshooting).

Module 1: Robotics Foundations: Python, Physics, and Linear Algebra

Module Overview

To control a physical robot and simulate its kinematics, you must master the fundamental mathematical languages of robotics: linear algebra, coordinate transformations, and Python. This module builds a solid baseline from absolute beginner Python programming to university-level robot kinematics and coordinate geometry.

Recommended Videos

Learn Python Programming: A Complete 13-Hour Tutorial

@DataWithBaraa13:02:5354,339

Learn Python Programming: A Complete 13-Hour Tutorial

This 13-hour course teaches Python from absolute beginner level to advanced application, covering fundamentals such as installing Python, using comments and print(), variables, and user input. It progresses through Python data types, string operations including indexing, slicing, search, validation, and case conversion. The course details numeric operations with integers and floats, mathematical functions, rounding, random number generation, and validation. Logic and operators are explored throu

@DataWithBaraa · 13:02:53

• Why this video is valuable: Writing custom reinforcement learning environments and hardware control scripts requires a robust foundation in Python. This comprehensive masterclass covers everything from data structures to object-oriented programming (OOP), which is essential because PyBullet and Gym environments are structured as classes.
• Knowledge Checkpoint:
  • Write a clean Python class with initialization methods and internal state tracking.
  • Use libraries such as NumPy to perform basic array operations and vector slices.
  • Debug standard execution errors, syntax exceptions, and environment pathing issues.

Understanding Vectors: Essence of Linear Algebra, Chapter 1

@3blue1brown09:5210,982,910

Understanding Vectors: Essence of Linear Algebra, Chapter 1

A vector is fundamentally a mathematical object that can be understood through three perspectives: as an arrow in space (physics perspective) with specific length and direction, as an ordered list of numbers (computer science perspective), or as any entity supporting addition and scalar multiplication (mathematician's perspective). In linear algebra, vectors are typically represented as arrows rooted at the origin within a coordinate system, where their coordinates specify how to reach from the

@3blue1brown · 09:52

• Why this video is valuable: Robotic states—such as joint positions, velocities, and foot locations—are represented as vectors. This video visualizes linear algebra concepts intuitively, establishing the mathematical groundwork for coordinate transforms and link-state matrices.
• Knowledge Checkpoint:
  • Define a vector from both a physical perspective (arrow in space) and a computer science perspective (ordered list of numbers).
  • Graphically and algebraically compute vector addition and scalar multiplication.
  • Visualize how changing bases alters coordinate values of your robotic end-effector.

Lecture 1 | Introduction to Robotics

@stanford58:11744,428

Lecture 1 | Introduction to Robotics

Robotics relies on mathematical models including kinematics (describing robot geometry and motion) and dynamics (modeling forces and torques) to enable precise control of robotic systems, with applications ranging from industrial automation to human-robot interaction and surgical assistance.

@stanford · 58:11

• Why this video is valuable: Delivered by Stanford University, this introductory lecture connects abstract mathematical models (kinematics, dynamics) directly to real-world control of multi-jointed mechanical systems, clarifying how joint torques generate spatial trajectories.
• Knowledge Checkpoint:
  • Distinguish between kinematics (geometry of motion) and dynamics (forces and torques causing motion).
  • Describe the difference between forward kinematics and inverse kinematics.
  • Map out how a 2-DOF arm or leg represents its joint space versus task space.

Lecture 02: Robotics: Primer-Kinematics and Dynamics

@IITRoorkeeJuly27:471,602

Lecture 02: Robotics: Primer-Kinematics and Dynamics

This lecture introduces fundamental concepts in robotics, covering kinematics (the study of motion without considering forces) which includes forward kinematics (computing end-effector position from joint angles using Denavit-Hartenberg parameters and homogeneous transformation matrices) and inverse kinematics (determining joint angles from desired end-effector positions), followed by dynamics (the study of motion with forces) including forward dynamics (computing joint trajectories from torques

@IITRoorkeeJuly · 27:47

• Why this video is valuable: This lecture provides a math-heavy primer on transformations, rotation matrices, and joint coupling. It teaches you how to compute end-effector (foot) coordinates based on joint angles.
• Knowledge Checkpoint:
  • Construct and calculate 2D translation and rotation matrices.
  • Calculate the position of a 2-DOF leg's foot relative to the hip using forward kinematics.
  • Formulate why multi-link systems suffer from complex coupling forces during acceleration.

Module 2: Robot Modeling and PyBullet Simulation

Module Overview

Before deploying a controller on hardware, you must build and test it in a physics simulator. This module teaches you how to construct a robot model from scratch using the Unified Robot Description Format (URDF) and program a simulation environment utilizing the PyBullet physics engine.

Recommended Videos

Understanding URDF: Unified Robot Description Format for ROS

@ArticulatedRobotics27:34124,047

Understanding URDF: Unified Robot Description Format for ROS

URDF (Unified Robot Description Format) is an XML-based standard in ROS that describes robot physical characteristics using links (rigid bodies) and joints (connections between links), where links define the robot's physical structure and joints specify how links relate to each other through different motion types (revolute, continuous, prismatic, fixed), with visual, collision, and inertial properties enabling accurate simulation and visualization.

@ArticulatedRobotics · 27:34

• Why this video is valuable: The simulation engine needs to know your robot's physical layout: link lengths, joint limits, masses, and rotational inertias. This tutorial provides a step-by-step breakdown of how to build structured XML-based URDF files to represent these properties accurately.
• Knowledge Checkpoint:
  • Construct a simple URDF file containing links (rigid bodies) and joints (revolute/continuous).
  • Define the <inertial>, <visual>, and <collision> tags for any given robot link.
  • Understand the parent-child relationship of frames in joint definitions.

Spot Micro Pybullet Simulation - Installation, Demo, and Code Review

@kevinwoodrobotics25:598,974

Spot Micro Pybullet Simulation - Installation, Demo, and Code Review

This tutorial demonstrates how to run the open-source Spot Micro quadruped robot simulation using Pybullet physics engine in Python within VS Code, covering the complete workflow from installing Python 3.8, cloning the spot_mini_mini repository, setting up a virtual environment with pip requirements, and executing the simulation in various modes including basic movement, foot path visualization, elevated rack testing, and auto-reset functionality for when the robot gets stuck; the code review ex

@kevinwoodrobotics · 25:59

• Why this video is valuable: This video serves as a template for setting up legged robots inside the PyBullet environment. It reviews real Python scripts that initialize PyBullet, load URDF models, apply joint commands, and extract simulation state variables.
• Knowledge Checkpoint:
  • Install the PyBullet library and write a basic script to run a real-time GUI physics loop.
  • Load a custom 2-DOF URDF leg into PyBullet using loadURDF().
  • Retrieve joint states (positions and velocities) using PyBullet APIs.

openDog V2 Part 6 - IT WALKS ! & Kinematic Model

@jamesbruton24:09117,498

openDog V2 Part 6 - IT WALKS ! & Kinematic Model

Inverse kinematics is a mathematical method that calculates the joint angles (shoulder and knee) required to position a robot's foot at specific X, Y, Z coordinates in space. The process involves using trigonometric functions and the Pythagorean theorem to work backwards from the desired foot position to determine the necessary joint configurations, enabling the robot to move its legs in three axes of translation (up/down, left/right, forward/backward) and three axes of rotation (pitch, roll, ya

@jamesbruton · 24:09

• Why this video is valuable: James Bruton walks through the geometric equations for 2D inverse kinematics (IK) on a physical multi-link robotic leg. It shows you how to translate desired spatial movements (e.g., foot moving straight down) into specific motor angles using the Law of Cosines.
• Knowledge Checkpoint:
  • Derive the inverse kinematics equations for a 2-DOF leg to determine knee and hip angles from a target coordinate $(x, y)$.
  • Use the Law of Cosines to solve joint configurations for a leg of arbitrary length.
  • Differentiate between joint-space control (controlling motor angles directly) and task-space control (controlling foot position).

Module 3: Deep Reinforcement Learning & PPO Basics

Module Overview

Deep Reinforcement Learning (DRL) allows a robot to learn behaviors through trial-and-error interaction. This module covers Markov Decision Processes (MDPs), policy structures, and the mathematical framework of Proximal Policy Optimization (PPO), culminating in setting up custom gym environments using Stable Baselines3 (SB3).

Recommended Videos

Deep Reinforcement Learning: Q-Learning, Policy Gradients & AlphaGo

@stanfordengineering31:58384,738

Deep Reinforcement Learning: Q-Learning, Policy Gradients & AlphaGo

This comprehensive section establishes the complete theoretical and algorithmic foundation of reinforcement learning. It begins by contrasting RL with supervised and unsupervised learning, introducing the agent-environment interaction paradigm where agents maximize cumulative rewards through sequential decision-making. The Markov Decision Process (MDP) formalizes RL problems with states, actions, rewards, transition probabilities, and discount factors. The lecture explains value functions (expec

@stanfordengineering · 31:58

• Why this video is valuable: To write high-performance RL agents, you must grasp the underlying mathematics. This Stanford lecture builds the formal foundations of MDPs, states, actions, transition models, and policy optimization.
• Knowledge Checkpoint:
  • Define the formal components of a Markov Decision Process (MDP).
  • Describe the difference between value-based methods (like Q-learning) and policy gradient methods.
  • Explain how a policy maps a state observation into an action probability distribution.

Proximal Policy Optimization (PPO) is Easy With PyTorch | Full PPO Tutorial

@MachineLearningwithPhil1:02:4785,676

Proximal Policy Optimization (PPO) is Easy With PyTorch | Full PPO Tutorial

Proximal Policy Optimization (PPO) is an actor-critic reinforcement learning algorithm that addresses the instability problem in traditional actor-critic methods, where small neural network parameter updates can cause dramatic performance drops. PPO achieves stability by constraining policy updates through a clipped objective function that limits the ratio of new to old policy probabilities within a range of [1-ε, 1+ε], preventing large parameter jumps. The algorithm uses Generalized Advantage E

@MachineLearningwithPhil · 1:02:47

• Why this video is valuable: PPO is the gold standard algorithm for dynamic robotic actions like jumping. This video explains PPO's clipped surrogate objective function and implements the entire Actor-Critic framework from scratch in PyTorch.
• Knowledge Checkpoint:
  • Explain why standard policy gradient methods suffer from training instability.
  • Describe how PPO's clipped objective function prevents destabilizing policy updates.
  • Contrast the roles of the Actor network (action selection) and the Critic network (value estimation).

Creating A Reinforcement Learning (RL) Environment - Reinforcement Learning p.4

@sentdex32:13116,520

Creating A Reinforcement Learning (RL) Environment - Reinforcement Learning p.4

This comprehensive section covers the complete process of building a custom reinforcement learning environment from scratch. The instructor demonstrates creating a grid-based environment with three entities: a player blob (controlled by the agent), a food blob (the objective to reach), and an enemy blob (an obstacle to avoid). Key topics include: environment configuration with grid size (10x10), episode count (25,000), and reward structures (move penalty -1, enemy penalty -300, food reward +25);

@sentdex · 32:13

• Why this video is valuable: PyBullet requires a structured API wrapper to communicate with RL libraries. This video shows how to build a custom OpenAI Gym/Gymnasium class from scratch, illustrating how to set up states, actions, step functions, and resets.
• Knowledge Checkpoint:
  • Write a custom Python environment class inheriting from gym.Env.
  • Define bounded custom action_space and observation_space fields using Gym spaces.
  • Implement the reset() and step() methods to update the simulation and return standard RL outputs.

Module 4: Training a 2-DOF Jumping Leg in PyBullet

Module Overview

With your URDF model configured and PPO fundamentals understood, you will now integrate them to train your robotic leg to perform dynamic vertical jumps. This module focuses on Gym-PyBullet integration, designing robust reward functions, and utilizing Stable Baselines3.

Recommended Videos

AI Mecha-Spider Learns to Jump or Die (PPO Reinforcement Learning)

@Dojo_AI_Channel09:376,135

AI Mecha-Spider Learns to Jump or Die (PPO Reinforcement Learning)

Proximal Policy Optimization (PPO) is a reinforcement learning algorithm that enables robots to learn complex physical behaviors through trial and error, where the robot observes its state (joint positions, angular velocities, and environmental distances) and receives rewards for successful actions (jumping across gaps) while learning from failures (falling), ultimately mastering the task through iterative optimization of its neural network policy.

@Dojo_AI_Channel · 09:37

• Why this video is valuable: This video demonstrates dynamic training of multi-link limbs using PPO to perform jump behaviors. It visualizes the transition from random joint flailing to coordinated, explosive leg extension, helping you structure your training observations.
• Knowledge Checkpoint:
  • List the observations (joint angles, velocities, trunk height, base orientation) necessary for a jumping task.
  • Design an termination condition to reset the environment when the leg falls or slips.
  • Identify physical behaviors indicative of localized minima (e.g., leg staying bent to avoid penalty).

Shawn Hymel: Tiny Reinforcement Learning in Robotics

@hackaday01:251,452

Shawn Hymel: Tiny Reinforcement Learning in Robotics

A reward function in reinforcement learning quantifies how well an agent is performing by assigning numerical values to different states or actions. The design involves squaring key variables (angle, angular velocity, motor position, motor velocity) and applying constants to prioritize certain objectives. For the inverted pendulum example, the reward function prioritizes getting the pendulum upright (angle near zero) while minimizing movement (low angular velocity). Negative rewards are applied

@hackaday · 01:25

• Why this video is valuable: A bad reward function leads to erratic behavior or physical self-destruction. This brief session highlights key techniques for writing smooth reward formulations, such as using squared penalties to discourage excessive torque changes and violent shaking.
• Knowledge Checkpoint:
  • Explain how sparse rewards differ from shaped rewards in dynamic robotics.
  • Implement an explosive height-reward term combined with torque-minimization penalties: $R = w_1 \cdot \text{height}_{\text{max}} - w_2 \sum (\tau^2)$
  • Prevent reward hacking where the robot learns to wiggle or exploit unmodeled contact physics.

Robot Dog Learns to Walk - Bittle Reinforcement Learning p.3

@sentdex01:0159,616

Robot Dog Learns to Walk - Bittle Reinforcement Learning p.3

Stable Baselines 3 is a Python library that provides implementations of various reinforcement learning algorithms (PPO, SAC, TD3, etc.) in a modular framework. To use it effectively, custom environments must inherit from the Gym interface, implementing standard methods like __init__, step, reset, and close. This standardization enables easier comparison between different RL algorithms and accelerates experimentation cycles by abstracting common training infrastructure.

@sentdex · 01:01

• Why this video is valuable: This video guides you through training robotic configurations using Stable Baselines3 (SB3). It demonstrates how to initialize the PPO agent, connect it to your custom robot environment, monitor rewards, and save the resulting policies.
• Knowledge Checkpoint:
  • Initialize a PPO model from the Stable Baselines3 library.
  • Set up logging callbacks to track episode lengths and cumulative reward curves over time.
  • Save trained models and reload saved policy checkpoints to run inferences.

Module 5: Actuator Control and Hardware Assembly (BLDC & ODrive Control)

Module Overview

To translate simulation results to a physical robot, you must construct robust hardware. This module covers the electrical assembly of a 2-DOF robotic leg using Brushless DC (BLDC) motors, magnetic encoders, a Teensy microcontroller, and ODrive motor controllers.

Recommended Videos

CNC Mill Conversion Part 4 - Setting up the Brushless DC Motor

@TheSolarPoweredEngineer29:584,711

CNC Mill Conversion Part 4 - Setting up the Brushless DC Motor

Setting up a Brushless DC Motor (BLDC) with ODrive requires installing Anaconda for Python, configuring motor parameters including pole pairs and encoder resolution (2048 pulses per revolution for AMT-102 encoder), setting brake resistance (0.5 ohm), and enabling closed-loop control mode for position control; proper configuration includes saving settings and rebooting the system, with velocity limits requiring tuning to prevent motor vibration.

@TheSolarPoweredEngineer · 29:58

• Why this video is valuable: This video guides you through configuring brushless motors using the ODrive platform via Python/Anaconda shell tools. It covers setting calibration parameters, pole pairs, and encoder resolutions.
• Knowledge Checkpoint:
  • Connect a BLDC motor to ODrive and launch the interactive odrivetool terminal.
  • Set critical safety bounds, including current limits and overvoltage protections.
  • Execute an automatic motor-calibration routine (AXIS_STATE_MOTOR_CALIBRATION).

Robot Arm Shoulder with Brushless motors?

@Skyentific10:1161,439

Robot Arm Shoulder with Brushless motors?

This video demonstrates how to construct a two-degree-of-freedom (2DoF) robot shoulder joint by combining two planetary brushless actuators, using an ODrive motor controller and Raspberry Pi for closed-loop control. The assembly process involves mounting the actuators with modified aluminum pulleys and printed parts, configuring encoder CPR settings (4000 counts per revolution), performing full calibration sequences, and implementing closed-loop position control to enable smooth coordinated move

@Skyentific · 10:11

• Why this video is valuable: This video showcases the design and assembly of a 2-DOF brushless joint using two planetary BLDC motors controlled by an ODrive. It provides mechanical inspiration and wiring strategies for your physical leg build.
• Knowledge Checkpoint:
  • Assemble a 2-axis articulation using brushless motors and compact planetary or cycloidal gearboxes.
  • Mount magnetic encoders securely to ensure feedback for closed-loop control.
  • Wire multi-actuator assemblies without binding or tensioning the cables during motion.

openDog V2 Part 4 - Electronics and Power up!

@jamesbruton16:1392,772

openDog V2 Part 4 - Electronics and Power up!

This video demonstrates the integration of ODrive brushless motor drivers with a Teensy 4.1 microcontroller for controlling a robot dog's six motors, using magnetic encoders for position feedback and a separate power distribution system to ensure stable 5V power for sensitive electronics while powering the motors from a 6-cell LiPo battery.

@jamesbruton · 16:13

• Why this video is valuable: James Bruton maps out the entire electrical architecture of a legged system, showing how a central microcontroller (Teensy 4.1) communicates commands to ODrive units. This setup serves as the primary template for your hardware wiring.
• Knowledge Checkpoint:
  • Wire a high-current power distribution system containing LiPo batteries, fuses, and emergency stop switches.
  • Establish communication between a central microcontroller (Teensy/Arduino) and an ODrive via UART/CAN bus.
  • Implement a safe startup routing sequence to home and arm the motors safely.

Module 6: Sim-to-Real and Policy Deployment

Module Overview

Deploying a neural network trained in simulation directly to real-world hardware often fails due to unmodeled physics, friction, and delays (the Sim-to-Real gap). This module teaches you how to bridge this gap using domain randomization and export your PyTorch policies to run in real-time on physical hardware.

Recommended Videos

Ilya Sutskever: OpenAI Meta-Learning and Self-Play | MIT Artificial General Intelligence (AGI)

@lexfridman03:27383,247

Ilya Sutskever: OpenAI Meta-Learning and Self-Play | MIT Artificial General Intelligence (AGI)

To transfer policies from simulation to physical robots, train policies under randomized physics conditions without telling them how the physics was randomized. This teaches the policy to infer physical properties during deployment. The policy learns to adapt to unknown environments by trying actions and observing responses, making it robust to systematic differences between simulation and reality.

@lexfridman · 03:27

• Why this video is valuable: OpenAI co-founder Ilya Sutskever discusses the core philosophy of Sim-to-Real transfer. He explains how training policies under randomized physics parameters forces the neural network to learn adaptive, robust behaviors that generalize to physical hardware.
• Knowledge Checkpoint:
  • Describe the conceptual goal of Domain Randomization (DR) in physics simulations.
  • List variables to randomize in PyBullet (e.g., link mass, friction coefficients, motor latency, joint noise).
  • Explain why a policy trained on randomized physics can infer real-world dynamics without explicit identification.

Sim2Real for Robotics - Opening Remarks (RSS 2020 Workshop)

@sim2real08:38601

Sim2Real for Robotics - Opening Remarks (RSS 2020 Workshop)

Sim2Real transfer is a robotics research paradigm that enables robots to learn skills in simulation environments and then apply these skills to real-world robotic systems with minimal or no fine-tuning required; however, a significant challenge exists because simulation environments and real-world conditions differ, creating what is known as the 'reality gap' that researchers must address to achieve effective skill transfer.

@sim2real · 08:38

• Why this video is valuable: This academic overview frames the key challenges of Sim-to-Real transfer. It discusses system identification, state estimation errors, and actuator limits, preparing you for the issues you may encounter when deploying policies to hardware.
• Knowledge Checkpoint:
  • Identify sources of discrepancy between simulator models and physical systems (the "Sim-to-Real gap").
  • Describe how latency in your state-action loop affects control loop stability.
  • Explain the trade-off between conservative, robust behavior and highly dynamic policies.

Quadruped Robot Control | From Beginner to Pro Episode 1

@deeprobotics860100:4441,450

Quadruped Robot Control | From Beginner to Pro Episode 1

Trained policies are exported as ONNX files (Open Neural Network Exchange format), which provides platform-independent representation of machine learning models. These files are copied to the deployment folder on the target system. Before real deployment, policies must undergo sim-to-sim testing in magical simulation, which replicates the exact API of the real robot to ensure safe behavior transfer.

@deeprobotics8601 · 00:44

• Why this video is valuable: This video shows the final deployment phase of a trained policy. It illustrates exporting your PyTorch neural network to an open, platform-independent ONNX file format to run in real-time on real-world controller hardware.
• Knowledge Checkpoint:
  • Export a trained PyTorch PPO model to the ONNX (Open Neural Network Exchange) format.
  • Run real-time forward-pass inference of your policy inside a C++ or Python deployment script.
  • Map the floating-point outputs of your network directly into real-time joint torque/position commands.

💡 Sim-to-Real Deployment Note: Standard microcontrollers like an Arduino Uno or ESP32 do not run heavy PyTorch libraries natively. For deployment, compile your trained policy network to ONNX or convert it to light C++ matrices. Run this model on a single-board computer (such as a Raspberry Pi 4/5) connected to your ODrives, or implement simple matrix-multiplication scripts directly on a Teensy 4.1 to execute your policy network's forward pass at $200\text{ Hz}$ or higher.

Course Map

Key People Index

• Ilya Sutskever
  • Context: Co-founder and former Chief Scientist of OpenAI. A pioneer of modern deep learning and meta-learning techniques, his work heavily influenced robust simulation-to-reality transfer models through self-play and randomized environments.
• James Bruton
  • Context: Former toy designer and independent robotics researcher. His open-source "openDog" and custom actuator design videos provide invaluable practical templates for DIY legged robotic systems.
• Benjamin Rosman
  • Context: Professor of Computer Science and Robotics. His lectures provide clear mathematical structures for decision-making models, reward mechanisms, and reinforcement learning agent evaluations.

Final Self-Assessment

Execute this final checklist to verify you have fully mastered the curriculum material and successfully completed your 2-DOF dynamic jumping robot leg project:

• Math & Kinematics: You can write down the 2D forward and inverse kinematic equations for your 2-DOF leg on paper and solve them inside your control script.
• URDF Configuration: You have authored a custom URDF file that defines your robot leg's link mass, joint limits, visual properties, and collision meshes.
• Simulation Environment: You have built a custom Gymnasium-compliant PyBullet environment class that wraps around your URDF model.
• PPO Mechanics: You can explain how the clipping factor in PPO stabilizes gradient descent steps during policy updates.
• Reward Function Design: Your reward function uses shaped rewards (with squared joint-torque penalties) to prevent high-frequency oscillations and hardware self-destruction.
• Successful Training: Your policy successfully trains in PyBullet, showing an upward learning curve that converges on a dynamic jump behavior.
• ODrive Configuration: You have configured your ODrives using Python, set the correct current limits, calibrated the BLDC motors, and achieved closed-loop position control.
• Hardware Wiring: Your physical robotic leg is assembled with a central controller (Teensy/Raspberry Pi) communicating commands directly to the ODrives.
• Domain Randomization: You have integrated domain randomization (varying mass, friction, and command latency) into your PyBullet training pipeline.
• Deployment Pipeline: You have exported your trained policy model to ONNX, loaded it onto your physical controller, and successfully executed a vertical jumping sequence on real hardware.