Academic Databases
Explore Our Semantic Databases
STEM & Tech
Math, Physics, CS & Engineering...
Economics
Macro, Micro, Econometrics...
Business
Management, Marketing, Finance...
Health & Life Sciences
Biology, Medicine, Neuroscience...
Humanities & Social Sciences
Psychology, Law, Philosophy, History...
General Studies
General & interdisciplinary studies...
Explore Topics
Features

Quadruped Robot Control with Deep RL: Training and Deployment on Lite3

Added: 2026-08-07

Deep Dive

Fundamental concepts of Reinforcement Learning, including Markov Decision Processes (MDPs), policy gradient methods (such as PPO), and reward function design.

Темирчев П.Г. | "Обучение с подкреплением" | Лекция 1 | ВМК МГУ

Темирчев П.Г. | "Обучение с подкреплением" | Лекция 1 | ВМК МГУ

307

MDPs consist of states, actions, transition probabilities, and reward functions. The transition function defines how the environment changes in response to actions, while the reward function provides feedback about action quality. Trajectories are sequences of states and actions generated by the agent, with probability distributions factored using the Markov property. Policy gradient methods optimize policies directly using gradient ascent, sampling trajectories to estimate gradients of expected cumulative reward. Reward design involves trade-offs: sparse rewards align with true goals but are harder to learn from, while dense rewards provide more signals but risk reward hacking. The discount factor γ determines how agents value future rewards: low γ makes agents short-sighted, high γ makes them consider long-term consequences. γ is not just a hyperparameter but fundamentally changes the task being solved.

Click to watch →

RL 6: Policy iteration and value iteration - Reinforcement learning

RL 6: Policy iteration and value iteration - Reinforcement learning

59K

Markov Decision Processes (MDPs) provide the mathematical framework for reinforcement learning, consisting of five core components: agents (decision-makers), environments (transition dynamics), states (situations the agent encounters), actions (choices available to the agent), and rewards (feedback signals). A policy is a function mapping states to actions, which can be deterministic or probabilistic. The environment provides transition dynamics as a joint probability distribution of next state and reward given current state and action. Discounted reward introduces a discount factor γ (0<γ<1) that weights future rewards exponentially, giving more importance to immediate rewards. The value of a state under a policy represents the expected total discounted reward achievable from that state onward.

Click to watch →

Principios Fundamentales del Aprendizaje por Refuerzo

Principios Fundamentales del Aprendizaje por Refuerzo

118

The Markov Decision Process (MDP) provides the mathematical foundation for reinforcement learning. The Markov property states that the future depends only on the current state, not on how that state was reached. An MDP consists of states, actions, transition probabilities, and reward functions. The discount factor (gamma) calibrates the trade-off between immediate and long-term rewards. Policies (pi) define decision rules mapping states to actions, which can be deterministic (always same action) or stochastic (probabilistic choices). The goal is finding the optimal policy that maximizes expected cumulative reward. Value functions estimate how good states or actions are: state value functions (V) estimate expected rewards from states, while action-value functions (Q) estimate rewards from state-action pairs.

Click to watch →

Stanford CS234 Reinforcement Learning I Introduction to Reinforcement Learning I 2024 I Lecture 1

Stanford CS234 Reinforcement Learning I Introduction to Reinforcement Learning I 2024 I Lecture 1

129.5K

A Markov Decision Process (MDP) extends Markov reward processes by adding actions that influence state transitions. The dynamics model specifies the probability distribution over next states given current state and action. A policy is a mapping from states to actions (deterministic) or distributions over actions (stochastic). Two central problems are evaluation (assessing a given policy's performance) and control (finding the optimal policy). Planning assumes known dynamics and reward models, while learning requires discovering these through interaction. Value functions quantify expected returns, serving as the foundation for policy improvement. The distinction between model-based (using learned dynamics) and model-free approaches represents a fundamental architectural choice in reinforcement learning system design. The discount factor γ (0 ≤ γ < 1) ensures mathematical tractability, reflects psychological reality of preferring immediate rewards, and enables meaningful policy comparison.

Click to watch →

[EEML'24] Andreea Deac & Ognjen Milinković - Reinforcement Learning Tutorial

[EEML'24] Andreea Deac & Ognjen Milinković - Reinforcement Learning Tutorial

467

Reinforcement learning problems are formalized using Markov Decision Processes (MDPs), which consist of four core components: (1) State space - the set of all possible states the agent can encounter; (2) Action space - the set of all possible actions the agent can take; (3) Transition function - determines the next state given the current state and action; (4) Reward function - determines the reward received given the current state and action. The agent interacts with the environment through a cycle: observe state, take action, receive reward and new state, then update behavior. A policy is a function that maps states to actions, typically implemented as a neural network that outputs probability distributions over possible actions. The learning objective is to find policy parameters that maximize the expected cumulative reward (return). In deep RL, policies are designed to be differentiable, enabling gradient descent optimization.

Click to watch →

Deep Dive

Basics of robot kinematics and control, specifically joint-level PD (Proportional-Derivative) control and the mechanics of legged locomotion.

openDog Dog Robot #1 | Planning & Explanation | James Bruton

openDog Dog Robot #1 | Planning & Explanation | James Bruton

212.7K

This video demonstrates how to design and control a robot dog using mathematical kinematics. The key concepts include: (1) Robot legs with three joints per leg (hip, knee, foot) require 12 motors total for a four-legged robot; (2) Ball screw actuators provide precise linear motion for joint control with minimal backlash; (3) Inverse kinematics calculations use trigonometry to determine joint angles needed to position the foot at desired coordinates in 3D space; (4) Motor encoders provide position feedback for accurate control; (5) The ODrive motor driver enables precise control with encoder feedback. The design ensures that joint movements produce straight-line foot trajectories by offsetting ball screws to create perfect triangles in the mechanism.

Click to watch →

Modelo Dinámico de un Robot de 3 Grados de Libertad en Simulink

Modelo Dinámico de un Robot de 3 Grados de Libertad en Simulink

943

This segment covers the implementation of a Proportional-Derivative (PD) control system for the robot. The control architecture consists of several key components: (1) A Demux block that separates the position, velocity, and acceleration vectors into individual joint signals for independent processing; (2) Gain blocks that apply the proportional gain (KP) and derivative gain (KD) to the position error and velocity signals respectively; (3) Sum blocks that combine the proportional and derivative terms for each joint; (4) A separate Function block that calculates gravity compensation forces based on the robot's position. The control equation for each joint is: Control = KP * Position_Error + KD * Velocity + Gravity_Compensation. The Demux is essential because each joint requires independent control, and the three outputs from the Demux feed into separate gain and sum blocks for each joint.

Click to watch →

Introduction to Robotics | Stanford CS223A Lecture 1

Introduction to Robotics | Stanford CS223A Lecture 1

Proportional-Derivative (PD) and Proportional-Integral-Derivative (PID) controllers are fundamental control strategies for robotics. PD controllers use proportional error and derivative of error to generate control inputs, while PID adds integral action to eliminate steady-state error. These controllers can be augmented with dynamic models to account for robot inertia, centrifugal forces, Coriolis forces, and gravity effects in both joint space and task space.

Click to watch →

ELO377- Ayudantía Q1 2

ELO377- Ayudantía Q1 2

216

This comprehensive segment covers three interconnected topics in robotics and control systems. First, solving trigonometric equations using the unit circle method: for sin(θ) = 0.5, solutions are θ = π/6 and θ = 5π/6; for sin(θ) = √3/2, solutions are θ = π/3 and θ = 2π/3; for sin(θ) = -√3/2, solutions are θ = 4π/3 and θ = 5π/3. Second, robot kinematics: forward kinematics uses homogeneous transformation matrices (T₀² = T₀¹ * T₁²), while inverse kinematics solves for joint angles given end-effector position (x = L₁*cos(θ₁) + L₂*cos(θ₁+θ₂), y = L₁*sin(θ₁) + L₂*sin(θ₁+θ₂)). The geometric approach involves constructing a triangle with sides L₁, L₂, and distance d = √(x² + y²), applying the law of cosines. Inverse kinematics typically has two solutions (elbow up and elbow down configurations). Third, decentralized PD control for multi-articulated systems: each joint has its own error signal and receives control input based on proportional and derivative gains (uᵢ = Kpᵢ*eᵢ + Kdᵢ*ėᵢ). For a double integrator plant (1/s²), the controller transfer function is C(s) = Kp + Kd*s, with closed-loop transfer function T(s) = (Kp + Kd*s)/(s² + Kd*s + Kp). The gains are chosen to place poles at desired locations for stability and performance.

Click to watch →

MIT Robotics - Marco Hutter - Robots in the wild

MIT Robotics - Marco Hutter - Robots in the wild

12.4K

Effective legged locomotion requires torque/impedance controllable systems rather than pure position control. The control architecture consists of interconnected modules: navigation algorithms determining walking paths, motion planners specifying stepping timing and locations, whole-body controllers tracking movements, and joint-level control for actuation. Two critical feedback loops exist: proprioceptive feedback (internal sensor measurements) and exteroceptive feedback (cameras, LIDAR for environmental localization). Modern approaches use Model Predictive Control (MPC) to optimize complete movement strategies including stepping locations and timing, incorporating whole-body dynamics and constraints like velocity limits, torque limits, and friction cones.

Click to watch →

Deep Dive

Familiarity with the PyTorch deep learning library for building, training, and running neural network policies.

(3/3)RL Journey to Trust Region Policy Optimization. TRPO implementation using pytorch

(3/3)RL Journey to Trust Region Policy Optimization. TRPO implementation using pytorch

205

Policies are implemented as neural networks outputting logits for discrete actions or means/log-standard deviations for continuous actions. PyTorch autograd requires retain_graph=True for intermediate gradient calculations and create_graph=True for higher-order derivatives. Gradients of log probabilities are flattened for optimization. KL divergence computation differs between discrete (categorical) and continuous (Gaussian) action spaces, with formulas accounting for distributional differences. This infrastructure enables efficient gradient-based policy optimization while managing computational graph complexity.

Click to watch →

Tutorial: ¡PYTORCH DESDE CERO!

Tutorial: ¡PYTORCH DESDE CERO!

25.4K

PyTorch is one of the most widely used Python libraries for implementing various types of Deep Learning models, including neural networks, convolutional neural networks (CNNs), LSTM networks, and Transformer networks. This tutorial provides a comprehensive guide to creating, training, validating, and testing Deep Learning models using PyTorch, suitable for those with basic Deep Learning knowledge who want to understand how to implement different models using this library.

Click to watch →

Deep Learning Einführung in Neuronale Netze mit Pytorch

Deep Learning Einführung in Neuronale Netze mit Pytorch

1.8K

PyTorch is a Python library used for deep learning and building neural networks. It was originally developed by Facebook and provides tools for creating and training neural networks.

Click to watch →

Redes Neurais em PyTorch | Programando em 10 minutos #1

Redes Neurais em PyTorch | Programando em 10 minutos #1

35.9K

PyTorch is a Python-based framework for building and training neural networks. It provides essential libraries including torch (core operations) and torch.nn (neural network modules). These libraries contain pre-built components like layers and activation functions that simplify the implementation of complex neural network architectures. The framework supports GPU acceleration through CUDA, enabling faster computation during training.

Click to watch →

PyTorch Full Course for Deep Learning

PyTorch Full Course for Deep Learning

923

PyTorch provides several key libraries for deep learning: torch.nn for neural network modules, torch.optim for optimizers (SGD, Adam), torch.utils.data for data loaders and datasets, and torchvision for computer vision tasks. These libraries work together to build, train, and evaluate deep learning models efficiently.

Click to watch →

Deep Dive

Understanding of physics simulation principles and the concept of the 'Sim-to-Real' gap in robotics.

Spring 2025 GRASP Seminar Hersh Sanghvi, University of Pennsylvania

Spring 2025 GRASP Seminar Hersh Sanghvi, University of Pennsylvania

174

Physics simulation enables data collection for robotics but suffers from domain shift—the 'Sim-to-Real Gap.' Simulations often lack accuracy in modeling real-world factors like actuation constraints and time delays. For example, deformable object simulators allow unrealistic stretching of cloth. Consequently, optimal control parameters found in simulation often fail on real robots, requiring manual experiments. This gap motivates approaches that can leverage simulation-derived priors while adapting to real-world variations.

Click to watch →

The Era of Embodied AI - From modular design to end-to-end system

The Era of Embodied AI - From modular design to end-to-end system

72.5K

The sim-to-real gap represents a fundamental challenge in robotics where behaviors trained in simulation fail to transfer to real-world environments. Simulators are computer programs with written physics that may approximate reality reasonably well for simple objects (like rigid mugs), but struggle with complex physical phenomena including contact physics, deformable objects, and motor dynamics. When training occurs in simulation, the resulting policies often fail dramatically when deployed in reality because the actual physics differ from simulated physics. One solution approach involves heavy randomization—training across many possible simulated scenarios with varied physics and terrains so that the learned policy becomes robust to variations. While this approach has been successful for locomotion tasks, manipulation remains more challenging due to the complexity of contact physics. Simulation provides significant benefits for visual and geometric diversity (lighting, object shapes, viewpoints) that would be difficult to collect in reality.

Click to watch →

Karen Liu: The New Role of Physics Simulation in AI

Karen Liu: The New Role of Physics Simulation in AI

2.4K

Karen Liu presents her research on creating 'learnable' physics engines that integrate data-driven learning with fundamental physics models to overcome the sim-to-real gap in robotics. Her approach involves three key innovations: (1) hybrid simulators that combine neural networks with physics equations to learn state-action dependent parameters like friction coefficients and actuator properties, (2) differentiable contact handling that enables gradient-based optimization through linear complementarity programs by exploiting contact physics structure, and (3) learnable human body models that capture complex biomechanical constraints including state-dependent joint limits and muscle-inspired torque limits. These techniques enable robots trained in simulation to successfully transfer to real-world human-robot interaction tasks such as assistive dressing, demonstrating that improving physics simulation processes can be more effective than solely optimizing control policies for bridging the sim-to-real divide.

Click to watch →

[LabMeeting] Generalized Force Model Fine-Tuning to Bridge Sim-to-Real Gaps in Bipedal Robots

[LabMeeting] Generalized Force Model Fine-Tuning to Bridge Sim-to-Real Gaps in Bipedal Robots

261

The sim-to-real gap represents the fundamental challenge of deploying policies trained in simulation to real-world robots. This gap arises because simulation environments cannot perfectly replicate real-world physics, sensors, and environmental conditions. Five main sources contribute to this gap: physics engine approximations (simplified contact models), sensor noise, terrain representation difficulties, distributional shifts between simulation and reality, and privileged information access. Three primary approaches address this challenge: system identification (estimating actual dynamics from measurements), domain randomization (randomizing simulation parameters to include real-world variations), and randomized force injection (adding disturbances to improve robustness). Understanding these causes and solutions is essential for developing reliable robotic systems that perform consistently across simulated and real environments.

Click to watch →

UC Berkeley Professor Ken Goldberg shapes the future of robotics | AI In Motion

UC Berkeley Professor Ken Goldberg shapes the future of robotics | AI In Motion

48.3K

The sim-to-real gap refers to the challenge of transferring robot behaviors learned in simulation to real-world environments. This problem varies significantly across different types of robots: locomotion robots (walking, running) have smaller gaps because they can approximate ground contact reasonably well, while manipulation robots face much larger gaps due to complex deformations in materials and objects. The gap exists because simulations often cannot accurately capture the precise forces, torques, and material deformations that occur during physical interactions with the real world.

Click to watch →

Setup

Training

Sim Tests

Deploy

Control

Hardware

Real Test

Setup

0:00

Playing Section

1
Install Isaac Lab simulator via Anaconda and PyTorch for CUDA acceleration.
2
Verify Isaac Sim installation with the empty window test.
3
Clone and install Isaac Lab, then test with a simple robot training.

Deep Dive

Advanced Sim-to-Real adaptation techniques, such as dynamics randomization, system identification, and domain adversarial training.

MIT Robotics - Pulkit Agrawal - Coming of Age of Robot Learning

MIT Robotics - Pulkit Agrawal - Coming of Age of Robot Learning

4.2K

The sim-to-real gap represents one of the most significant challenges in deploying learned policies to physical robots. Policies trained in simulation often fail dramatically in the real world due to unmodeled dynamics, sensor noise, and parameter mismatches. Three complementary approaches address this challenge: (1) Domain randomization trains policies across varied simulation parameters (friction, mass, center of mass), creating robust policies that perform reasonably across different conditions but sacrificing peak performance on the training distribution; (2) Online system identification allows policies to infer necessary domain parameters from interaction data without explicit estimation; (3) Policy cloning transfers knowledge from teacher policies with privileged information to student policies using only observable states. The combination of domain randomization, online system identification, and curriculum learning enables policies that achieve 3.8 m/s on the mini cheetah—significantly outperforming previous results. Curriculum learning progressively increases task difficulty by expanding the sampling region only when success rates exceed thresholds, preventing agents from being overwhelmed by difficult tasks early in training. Adaptive curricula that consider dependencies between dimensions (e.g., faster linear velocities may allow higher angular velocities) outperform independent box curricula.

Click to watch →

Closing the Reality Gap in Sim2Real

Closing the Reality Gap in Sim2Real

180

The reality gap is the fundamental challenge where agents trained in simulation fail in real-world deployment due to differences in textures, physics, and environmental conditions. Training in simulation offers advantages: higher throughput, safety, and cost-effectiveness. Three main approaches address this: system identification (making simulators match reality), domain randomization (training across diverse simulated environments), and domain adaptation (adapting agents to real-world observations). Domain randomization trains agents across a distribution of simulated environments by randomizing visual parameters (positions, colors, textures, lighting) and physical parameters (mass, volume). A key challenge is learning object properties from single observations, solved by training simple classification networks. Training in simulation is dramatically faster than real-world training, achieving high-quality performance in hours versus years.

Click to watch →

Lecture 22 Sim2Real and Domain Randomization -- CS287-FA19 Advanced Robotics at UC Berkeley

Lecture 22 Sim2Real and Domain Randomization -- CS287-FA19 Advanced Robotics at UC Berkeley

10.3K

Domain adaptation addresses the sim-to-real gap through supervised and unsupervised approaches. Supervised methods include fine-tuning, progressive networks, inverse dynamics learning, and Bayesian priors. Unsupervised approaches leverage image-to-image translation to map simulated images to real-world distributions. Domain randomization represents a paradigm shift: instead of creating accurate simulators, create maximally varied ones. Historical milestones include the 1997 Radical Envelope of Noise Hypothesis and 2016 CAD-to-RL. Vision applications demonstrate success: training on randomized textures, backgrounds, lighting, and distracting objects produces models achieving state-of-the-art performance on real-world object localization tasks. Domain randomization enables one-shot imitation learning for complex manipulation tasks and generic object grasping through procedural generation of unrealistic objects.

Click to watch →

Michiel van de Panne (UBC): MPC and RL, two different roads to legged locomotion, and that's OK

Michiel van de Panne (UBC): MPC and RL, two different roads to legged locomotion, and that's OK

5.1K

Transferring policies from simulation to real robots requires addressing several factors: state estimation completeness (including velocity information), PD controller gain parameters, and feedback delay characteristics. Dynamics randomization (training across parameter variations like mass and friction) improves robustness but introduces conservatism. Randomization is most beneficial when addressing specific known issues (like feedback delays) rather than serving as a default solution. Understanding true transfer barriers through careful diagnosis proves more effective than blanket randomization. Successful transfer requires matching both dynamics accuracy and control loop timing characteristics between simulation and reality.

Click to watch →

Real-World Robot Learning (TensorFlow Dev Summit 2018)

Real-World Robot Learning (TensorFlow Dev Summit 2018)

28.1K

Models trained in simulation typically fail to transfer to real-world robots due to systematic differences in visual appearances and physics. A model achieving 90% success in simulation dropped to 20% in reality. Sim-to-Real transfer addresses this through two main approaches: randomization (varying textures, colors, lighting, and geometries) and domain adaptation. Feature-level domain adaptation trains models on both domains and uses similarity loss at intermediate layers to encourage domain-invariant features. Domain-Adversarial Neural Networks implement this by training a classifier to predict domains while the main model confuses it. Pixel-level methods use GANs to transform simulated images into realistic ones, with both generator and task model trained simultaneously.

Click to watch →

Deep Dive

Perceptual (vision-based) locomotion, which integrates depth cameras or LiDAR data into the RL policy to navigate complex, uneven terrains.

Jitendra Malik: Vision and Robotics for Embodied AI

Jitendra Malik: Vision and Robotics for Embodied AI

1.1K

Perceptive locomotion (using vision for terrain navigation) is achieved by: (1) collecting RGB video of humans, (2) 3D reconstruction using computer vision, (3) physics simulation, (4) reinforcement learning for robustness. Only about 100 trajectories are needed because variations are created in simulation. The human trajectories are retargeted to the robot's body. This approach is widely deployed for humanoid locomotion, where the robot learns from human demonstrations but practices in its own body.

Click to watch →

Robust RL-Based Locomotion for Resource-Constrained Quadrupeds with Exteroceptive Sensing

Robust RL-Based Locomotion for Resource-Constrained Quadrupeds with Exteroceptive Sensing

2.2K

This video presents a robust reinforcement learning-based locomotion system for resource-constrained quadruped robots that integrates Intel RealSense and PMD Flex 2 time-of-flight cameras to create elevation maps for terrain perception, using concurrent training of control policies and odometry re-estimators with sensor fusion via Kalman filtering to achieve reliable traversal of uneven terrains including stairs up to 22cm height and challenging outdoor environments like vegetation and gravel.

Click to watch →

[Lab Meeting] BOLT humanoid development update

[Lab Meeting] BOLT humanoid development update

525

Advanced robot locomotion combines multiple sensing modalities and sophisticated training approaches. Vision-based terrain perception uses cameras to measure terrain height and detect obstacles, with neural networks processing visual data to handle noise and improve reliability. The system combines vision information with motor encoder data for robust terrain assessment. Teacher-student policy training bridges simulation and reality: the teacher policy is trained in simulation with detailed terrain information to create optimal reference policies, while the student policy is trained using only real robot sensors (vision and encoders). This approach creates policies that handle real-world sensor limitations while maintaining simulation-learned performance. The speaker plans to implement bidirectional movement, tilted plane navigation, and end-to-end vision-based locomotion using these techniques.

Click to watch →

Perception and Footstep Control for Bipedal Walking on Rough Terrain (Brian Acosta Thesis Defense)

Perception and Footstep Control for Bipedal Walking on Rough Terrain (Brian Acosta Thesis Defense)

443

A typical perception pipeline for legged locomotion includes: (1) Depth camera/LiDAR input; (2) Elevation mapping framework stitching point clouds over time to form an elevation map of terrain; (3) Plane segmentation procedure finding planar features and extracting them as planar polygons; (4) Polygon simplification or convex decomposition to prepare for MPC with convex polygon constraints. This pipeline was found to be brittle, particularly the plane segmentation step, which caused inconsistencies in terrain segmentation even when elevation maps changed little between frames.

Click to watch →

[Lab Meeting] Constrained RL and Egocentric Vision Policy Learning for Humanoid Locomotion

[Lab Meeting] Constrained RL and Egocentric Vision Policy Learning for Humanoid Locomotion

231

Terrain scanning (scan) is a vector-based representation of terrain height at multiple points relative to the robot's position (e.g., 11x19 grid with 0.1m spacing). This approach is computationally more efficient than processing RGB images or depth images. The scan-based approach uses an oracle model (teacher) that learns terrain embeddings from scan data, then a student policy learns to generate depth images from scans for distillation. Training with 4 GPUs produces 432 million transitions in 24 hours, compared to only 4 million with a single GPU - a 100x improvement in data efficiency. This enables vision-augmented locomotion where the robot can perceive and navigate terrain features like stairs.

Click to watch →

Deep Dive

Hybrid control approaches that combine Model Predictive Control (MPC) with Reinforcement Learning to achieve robust and safe locomotion.

Benjamin Recht: Optimization Perspectives on Learning to Control (ICML 2018 tutorial)

Benjamin Recht: Optimization Perspectives on Learning to Control (ICML 2018 tutorial)

16.4K

Model predictive control (MPC) is essentially Q-learning applied to continuous control, solving optimal control over finite horizons and applying only the first action before re-planning. Longer horizons provide exponential decay in sensitivity to terminal cost accuracy. The feedback nature of MPC allows correction of both environmental disturbances and model errors. Hybrid approaches initialize with simple controllers (like PID) to collect data, then learn terminal cost functions from observed costs, enabling safe learning where planning ensures safety while learning improves performance.

Click to watch →

Michiel van de Panne (UBC): MPC and RL, two different roads to legged locomotion, and that's OK

Michiel van de Panne (UBC): MPC and RL, two different roads to legged locomotion, and that's OK

5.1K

Model Predictive Control (MPC) and Reinforcement Learning (RL) are complementary approaches for legged locomotion that should be used collaboratively rather than competitively. MPC provides planning capabilities through online optimization over finite horizons, making it suitable for novel situations and time-indexed motion sequences, while RL offers efficient policy lookup through offline pre-computation, excelling at learning complex behaviors through reward shaping. The key insight is that effective legged locomotion requires combining cached solutions (RL policies) with online search (MPC planning), similar to how AlphaGo combines policy networks with search. Practical implementation involves using multi-level abstractions in MPC to reduce computational complexity, employing curriculum learning to progressively increase task difficulty in RL training, and carefully designing reward functions to guide learning. Additionally, sim-to-real transfer requires understanding specific mismatches between simulation and reality rather than defaulting to dynamics randomization, as accurate state estimation and appropriate controller gains are equally critical.

Click to watch →

RI Seminar: Byron Boots : Machine Learning and Model Predictive Control for Adaptive Robotic Systems

RI Seminar: Byron Boots : Machine Learning and Model Predictive Control for Adaptive Robotic Systems

4.3K

Short planning horizons cause MPC to become myopic and fail on tasks requiring long-term reasoning. Blended Q-learning addresses this by combining MPC with reinforcement learning: shorter predictions rely on the MPC model while longer predictions use learned Q-functions. The parameter λ controls the trade-off, and as agents gain experience, λ can be decayed to reduce dependence on potentially biased models. Learning policies of MPC controllers abstracts the problem to choosing what MPC algorithm or settings to use at any point, enabling zero-shot transfer to new domains while maintaining computational efficiency.

Click to watch →

DTC: Deep Tracking Control

DTC: Deep Tracking Control

19.6K

This video introduces DTC (Deep Tracking Control), a hybrid control architecture that combines trajectory optimization with reinforcement learning to achieve versatile and robust perceptive legged locomotion. The approach uses a model-based planner to generate optimized reference motions at approximately 2 Hz, while a deep neural network policy trained in simulation learns to track these optimized footholds, addressing the limitations of pure model-based methods (which struggle with model mismatch and sparse reward environments) and pure reinforcement learning (which lacks accuracy and terrain generalization). The system demonstrates superior robustness on slippery or deformable ground and successfully handles challenging terrains including stepping stones with height differences, moving objects, and gaps up to 60 cm in length, with the capability to clear objects up to 48 cm using an enhanced kinodynamic MPC method.

Click to watch →

Se Hwan Jeon - PhD Thesis Defense

Se Hwan Jeon - PhD Thesis Defense

293

This PhD thesis presents a hybrid approach to legged robot locomotion control that combines model-based methods (Model Predictive Control) with learning-based methods (reinforcement learning). The key innovation is using learned neural networks to provide warm starts for MPC optimization, significantly reducing computation time from 0.5-5 seconds to 10-20 Hz. Additionally, the research introduces a residual architecture where a learned corrective network is added on top of an MPC controller, providing the physical grounding and interpretability of model-based methods while enabling the adaptability and robustness of learning-based approaches. This hybrid strategy addresses the fundamental trade-off between the computational expense and model mismatch issues of pure model-based control and the erratic behavior outside training distributions of pure learning-based control.

Click to watch →

Deep Dive

Optimization of RL policies for real-time edge computing on embedded hardware using ONNX or TensorRT.

Using RLLib in an Enterprise Scale Reinforcement Learning Solution

Using RLLib in an Enterprise Scale Reinforcement Learning Solution

1.2K

Deployment strategies depend on latency requirements. For manufacturing controllers where seconds of latency are acceptable, Ray Serve creates cloud-based APIs. For real-time applications like drones requiring millisecond responses, trained checkpoints are optimized for embedded hardware using ONNX or TensorRT, particularly for NVIDIA DRIVE hardware. For regulatory requirements mandating offline operation, standard laptops or workstations can be deployed directly to systems like smart grids or wind farms. This comprehensive approach ensures trained models can be deployed across diverse hardware environments.

Click to watch →

Tiny Reinforcement Learning for Robotics: Edge AI & Control Theory

Tiny Reinforcement Learning for Robotics: Edge AI & Control Theory

Deploying reinforcement learning models to microcontrollers requires transforming complex neural networks into efficient implementations. Proximal Policy Optimization uses an actor-critic architecture: the actor outputs action probabilities (three discrete choices for the pendulum), while the critic predicts future rewards during training. After training, the critic is discarded, leaving only the actor for deployment. Model compression tools like Edge Impulse optimize these networks for microcontrollers like ESP32 S3, which feature hardware acceleration for neural inference. The deployed model achieves approximately 8ms inference time, demonstrating that complex RL policies can run autonomously on embedded hardware without GPU/CPU connections.

Click to watch →

Export & Optimize YOLO-NAS to ONNX & TensorRT for Real-Time Performance

Export & Optimize YOLO-NAS to ONNX & TensorRT for Real-Time Performance

10.4K

This section details the complete TensorRT optimization pipeline for YOLO NAS models. It covers ONNX to TensorRT engine conversion using trtexec with explicit batch size configuration (critical since TensorRT doesn't support dynamic batching), version matching between conversion and inference environments, and comprehensive benchmarking with warmup iterations and percentile calculations. The section demonstrates TensorRT inference implementation including engine loading, input image formatting (RGB, 640x640, normalized), and output parsing for bounding boxes, confidence scores, and class IDs. The workflow enables real-time video stream processing with optimized performance, allowing deployment on mid-range hardware while achieving 100+ fps. The section emphasizes that hardware-specific optimization enables real-time performance without requiring expensive high-end GPUs.

Click to watch →

Open Neural Network Exchange (ONNX) in the enterprise: how Microsoft scales ML - BRK3012

Open Neural Network Exchange (ONNX) in the enterprise: how Microsoft scales ML - BRK3012

2.2K

ONNX Runtime serves as a unified inference engine enabling consistent model execution across diverse hardware platforms. It maintains a single application interface while supporting CPU, GPU, and specialized accelerators through execution providers. The architecture parses ONNX graphs, applies optimizations, and partitions subgraphs to appropriate execution engines based on available hardware capabilities. Intel's AVX-VNNI instruction set demonstrates this strategy, fusing three vector operations into one to achieve 4x compute acceleration with minimal accuracy loss (0.41 top-1, 0.21 top-5). Combined with graph fusion techniques, this enables 3.4-3.67x performance scaling across batch sizes. The approach treats edge computing as an extension of the cloud, enabling consistent model execution from cloud-scale deployments to lightweight edge devices while maintaining identical application code across all targets.

Click to watch →

ONNXCommunityMeetup2023: Olive: a user-friendly toolchain for hardware-aware model optimization

ONNXCommunityMeetup2023: Olive: a user-friendly toolchain for hardware-aware model optimization

588

Olive is a user-friendly hardware-aware model optimization toolchain that composes industry-leading techniques across model compression, optimization, and compilation to automatically generate the most efficient model(s) for cloud or edge inference, given a model and targeted hardware along with constraints like accuracy and latency; it works with ONNX Runtime as an end-to-end inference optimization solution.

Click to watch →

Setup

Training

Sim Tests

Deploy

Control

Hardware

Real Test

Setup

0:00

Playing Section

1
Install Isaac Lab simulator via Anaconda and PyTorch for CUDA acceleration.
2
Verify Isaac Sim installation with the empty window test.
3
Clone and install Isaac Lab, then test with a simple robot training.

41.5K views10617:11@deeprobotics8601Original Release: 2025-09-06

This video demonstrates the complete pipeline for training and deploying reinforcement learning policies to control quadruped robots, covering the installation of simulation environments (NVIDIA Isaac Lab and Omniverse), training policies using reward functions with multiple components, performing sim-to-sim testing for safety validation, exporting policies as ONNX files, and deploying them on real robots through SSH and wireless connections, while highlighting the inherent sim-to-real gap that affects policy performance in real-world conditions.

[00:00:00][Music] Hi, my name is Percy and I'm a control engineer at Deep Robotics. Today I'm going to teach you guys how to train IO policy from scratch to control a live streaming project robot. First, we'll start by installing Aset lab. Aset lab is built by Nvidia. It's currently the most popular simulator for robotics around the world.

[00:00:24]To start installing as lab we are using an Anaconda virtual environment and we'll work on that. It's very easy to install Anaconda virtual environment. We simply just copy and paste command and let it run. And here we have our virtual environment.

[00:00:43]After that we need to install PyTorch for CUDA acceleration. You can also install PyTorch on the PyTorch official website if you have more upto-ate GPUs like their code may install like a more upto-date version of CUDA and PyTorch.

[00:01:00]But here we I'm going to use the command here on their official website cuz it's enough for me.

[00:01:08][Music] Okay, after installing PyTorch, we can start installing SIM. It's very simple.

[00:01:28]We just copy and paste this one line command and it will start installing. It may take some time, but uh we just let it run.

[00:01:41][Music] After installing ISX scene, we need to verify the installation. We can simply just type IS6 here and type yes.

[00:02:05][Music] And if you guys can see this empty window pops up, uh that means your i6 has been successfully installed. The first time start is going to take some time. Uh I have already run multiple times here. So it's pretty fast for me.

[00:02:23]Now we can start installing lab. First, let's get clone the code from the ripple to our local computer and [Music] let's install some build essentials.

[00:02:40]We need to fix some broken dependencies.

[00:02:43]Okay, [Music] now we just use the use their script to install it. Islab.sh SH install.

[00:02:56]I'll just let it run.

[00:02:58][Music] So there may be some bugs coming up but it's totally fine. Okay. Now we have installed as lab. Let's verify the installation.

[00:03:15][Music] Also, if you guys can see this empty window pops up, it means that ISAC lab has been successfully installed.

[00:03:28]And we can also use lab to train a simple robot. We'll choose the ISX end example here to see whether we can start training. Okay, so now you guys can see the code is starting to run and we are currently training an SM robot and now we'll start diving into our live three training code.

[00:03:50]Okay, so let's go to our deep robotics GitHub center and go to this IO training repo. First, let's get clone the code to our local computer and install this package.

[00:04:06]To verify the installation, we can use this line of code to list the environments.

[00:04:12]If you guys see these four environments, it means your our training code has been successfully installed.

[00:04:20]And now we can start training our light three robot. We just use this command.

[00:04:26]Specify the device [Music] and the code is starting to train.

[00:04:40]Now the training has finished and we can check what we got from the results. So let's go to the logs folder and go to this rsl deeper library 3 rough. We can use tensorboard to check the training process. Just type tensorboard log directory and just click this link and we can see the training process smooth curve reload data and increase the number of road components. So there is uh there are many reward components here because the reward functions consists of different reward different parts of the rewards. We'll dive deeply into this in the next video.

[00:05:27]Okay. So now we can we can also check the training results uh uh visually and in an interactive way. Just copy this command type keyboard so we can inter interact with it. And you can also specify which checkpoint you want to use. Going to choose the last checkpoint. So the default number of apples we train is 20,000. And you guys can change it if you want to train more.

[00:05:59][Music] And now here's a small live 3 robot here. Just click console and uh we can start controlling the robot.

[00:06:10]Oops, this is not working well. And so now you guys can see that the live three robot can be controlled to traverse on uneven terrains uh in Isc lab simulation and it's working pretty well [Music] now. We can also control it to traverse on some stairs. But um this is a very simple policy. It is only for tutorial purposes. Uh this this the performance is not that good. So of just kind of falls to the ground.

[00:06:54]So this is this part is called simtosim test. It is very necessary before you deploy the robot deploy the policy into the real robot.

[00:07:04]uh you need to check whether the API is correct, whether you send the correct data. Otherwise, if you just like send a very large torque, it's going to burn the motors or it's going to cause some very dangerous behavior. So, it's very necessary, very important to do the simp uh process. After playing lab autonomically, just export the policy as an onyx file. We we're going to use this Onyx file on the real life 3 robot.

[00:07:34]It is also very easy to use this ONX file. We can just copy and paste this policy to the IO deploy folder which we'll go back which we will dive into later.

[00:07:47]We just check the train policy in ISC lab. Now we're going to check it in uh in the modical simulation because um we built our magical simulation to have the exact same API as a real robot. uh if it if the robot in magical simulation can move uh normally as expected it won't cause any like dangerous behaviors in the real system. So this part is called sim to sim test. It's very important and it's very important for safety.

[00:08:18]Before deploying before doing the sim to sim test I want to introduce a very useful tool. It is for segmentation fault. So when working with C++ working with C uh sometimes we just get a segmentation fault and we don't know where the bug is coming from. This tool can tell you which function in which file caused the problem. And to install this function is very easy. will simply just copy these commands and uh start installing and our our code has already integrated with it. So if you install this tool and run our code, our code will just autonomously use this tool.

[00:08:57]Afterwards has installed pi bullet and magical code.

[00:09:02]I I created cond environment to work with is just uh to separate it from other code.

[00:09:08]Okay. Oh, I have already have a like real IO point. Let's just remove it and start over again.

[00:09:15]Let's click get get clone it recursively cuz we have a lot of third parties for communication with the hardware uh for UIF models. So, it's a it's a pretty complicated folder.

[00:09:29]Okay. Now, let's go into our live 3 IO deploy folder and just make build folder and go to it. check the CMake uh flags.

[00:09:40]We'll also dive deeply into the the meaning of this code, the meaning of the flags. Uh but in this video, we're just going to run the process so you guys can deploy it and we'll start compiling.

[00:09:55]So the compiling has succeeded.

[00:10:00]To run the code to run the program to run the executable program uh we need to open two terminals. So then in the first terminal let's go to the simulation and uh run our module simulation file Python file and in other in the other terminal we run our executable IO deploy file. So, we also have this uh nice colorful debugging information in Mojuko. Uh it will print it out uh one FPS. You guys can uh debug it if you didn't if the sims test didn't goes very well.

[00:10:39]And to control the robot in Magical, first press Z to enter standup mode. The robot will just stand up by itself. And then press C to go into control mode.

[00:10:53]Now we can start controlling the robot with our keyboard.

[00:10:59]So now we can just press WD to control it to move around and Q and E to control it to make turns.

[00:11:11]And we can also control it to go upstairs. It works pretty well in simulation at least for now. But usually the stairs seem to real gap and in the real robots doesn't work as smoothly as this.

[00:11:31]This is the package of a live 3 quart robot.

[00:11:35]This is the charging base.

[00:11:39]This is the charger and this is the cable.

[00:11:44]You can connect them like this.

[00:11:48]And here's the light three battery. Can just put it there and it will start charging. This is the retray game pad. We're going to use this to control the robot. Here are some p some parts for the foot pad of the CPA robot.

[00:12:03]If it's broken, you can just use it to replace it. Sorry.

[00:12:09][Music] And this is the light 3 robot. Here's the battery. To get the battery out of the robot, you should lift it up a little bit and there's a button there and just put your fingers at the bottom and click it. the battery will just slip out. And to put it back, just simply do that. Okay. To to start the battery, to start the robot, just short click and long click the battery.

[00:12:58]And if you see the lights of the head is starting to turn blue, it means the robot has uh uh has been started.

[00:13:05]The next step is to connect both the laptop and the game pad to the robot.

[00:13:10]Uh you can find the name of the Wi-Fi from the bottom. You can just flip it and you can see it. And we're going to connect it with the Wi-Fi. The password is usually 1 2 3 4 5 6 7 8.

[00:13:24]And we have already connected.

[00:13:26]We also need to connect the game pad to the Wi-Fi.

[00:13:32][Music] Okay, we have connected both of them. So the next step is to transfer the files and the policy to the robot. We have already done that. If you guys haven't, you can use SAP file to transfer the files to to the to the robot. The password is a single quote.

[00:13:53]So now let's just enter the onboard computer by using SSH.

[00:14:01][Music] Okay. Now we and to run the file. So we have already finished compiling. Uh if you guys haven't just follow the readme command and you will use that uh cm make flags and start compiling. After that just run the IO deploy file exactly the same as to sim.

[00:14:30]So you can see the robot is starting to move. The default uh controlling of the robot is to click this one to stand up and to click this one to enter IO control mode. Okay, so now we are inside control mode.

[00:14:48]Let's take it for a walk.

[00:14:56]So you can use the joystick to control it.

[00:15:04]And you can also traverse on uneven terrains as where we uh train it on.

[00:15:11]So this is very normal to happen. When it happens, it will just enter a safety mode. You can just uh take it up, put it on a flat terrains, [Music] put the legs like this. Uh okay we better like there's a slope here just in the same way control it to stand up enter the control mode. Okay now we can just control it to walk back.

[00:15:45][Music] So the performance of this policy is not very good. It's only to uh give you guys a tuto tutorial to know how to control the robot.

[00:15:57]You can also let it go upstairs a little bit.

[00:16:12]So it can uh go over like one or two stairs, but the policy is not trained to go over multiple stairs at the same time. So if you guys remember in the simtosim test, there are some stairs that the robot can go up to. Now we're going to see whether it can go up in the real in the real world.

[00:16:32]So as you guys can see it cannot go up.

[00:16:35]Uh this is very normal cuz the synal gap is definitely bigger than the sync scene gap and Mojuko simulator it doesn't have any noise doesn't have any uh turbulence. So it is uh very clear that the simp gap is going to be bigger and this is very normal to happen. So this is the end of the tutorials. Uh good luck with your own robot development.

[00:16:58]Thank you.

[00:17:00][Music]

Up Next

Robotics 2: PD Control and Stability Analysis (Lecture 14b) | Prof. De Luca

@videodiag-sapienzauniversi4757

2.5K views•2020-04-23

RatSLAM: Biologically Inspired Robot Mapping and Navigation

@milfordrobotics

20.9K views•2012-08-03

How to Build a Self-Balancing Robot: Arduino Nano & MPU6050

@easytechzones

16.8K views•2022-03-09

Introduction to Robotics | Stanford CS223A Lecture 1

@stanford

744.4K views•2008-07-22

Related Study Plans & Knowledge Roadmaps

Structured learning paths in Robotics

View All Robotics Topics

CDPR: Kinematics, Arduino & Winch Control

01Foundations of Arduino & Basic Electronics

02Stepper Motors and Winch Mechanics

03Kinematics of 2D Cable-Driven Robots

+ 2 more learning modules

Explore Roadmap

Rocker-Bogie Rover: Build, Stereo Vision & ROS

01Physics and Kinematics of Rocker-Bogie Suspension

02CAD Design, 3D Printing, and Assembly

03Actuators, Power, and Microcontroller Integration

+ 3 more learning modules

Explore Roadmap

Building AGVs: QR Navigation & Lifter Systems

01Introduction to AGVs & Embedded Electronics

02Chassis Design & Lifter Mechanics

03Computer Vision & QR Code Localization

+ 2 more learning modules

Explore Roadmap

EMG Exoskeletons: Design, Control & Actuation

01Electronics & Microcontroller Foundations

02Biomechanics & EMG Sensor Fundamentals

03Actuators & High-Torque Mechanisms

+ 3 more learning modules

Explore Roadmap