Sim-to-Real Transfer in Robotics: RSS 2020 Workshop Opening

Robot simulation tools can be categorized by hardware requirements: low-end tools like PyBullet and Webots run on standard laptops without GPUs, mid-range tools like Gazebo and MuJoCo require entry-level to mid-range GPUs, and high-end tools like NVIDIA Isaac Sim need RTX 4070+ GPUs for photorealistic rendering and parallel learning; beginners should start with lightweight tools matching their current hardware, and cloud computing offers an alternative for accessing high-end simulation capabilities without expensive hardware.

Physics engine selection depends on five key dimensions: scalability, differentiability, dynamics realism, visual fidelity, and documentation. PyBullet and MuJoCo excel in physics realism, while Unity leads in visuals. Differentiable physics enables gradient-based optimization methods. The choice depends on whether visual fidelity matters and whether differentiable capabilities are needed. For robot table tennis, the system architecture includes shared components (physics, state machine, reward module) and environment-specific components (real-world perception, safety layer). The transition from PyBullet to MuJoCo improved dynamics modeling, particularly for fluid dynamics needed to model ball spin.

MuJoCo is an open-source physics simulation framework (originally commercial, acquired by Google DeepMind) well-suited for robots with joints and contacts. Scenes are constructed as kinematic trees where coordinate frames attach to each other. Bodies have properties and joints connect them (free joints allow free movement, constrained joints connect bodies). The mj_step function advances simulation state. States include joint angles and velocities; control inputs are actions applied. Pre-built environments include real robots like Atlas and Dexter's hand.

MuJoCo (Multi-Joint Dynamics with Contact) is a high-performance physics engine designed for robotics, machine learning, and biomechanics simulation, featuring physics accuracy with joint kinematics and dynamic analysis, flexible robot modeling via XML files, cross-platform support (Windows, Linux, Mac), and extensive API integration with Python and C++, enabling researchers to simulate various robot types including bipeds, humanoids, quadrupeds, drones, and manipulators for applications in reinforcement learning, prototype building, and biomechanics research.

Robotic simulations are essential tools for designing controllers because real-world robot data collection is expensive, time-consuming, and poses safety risks; simulation environments like MuJoCo and PyBullet provide safe, scalable, and cost-effective alternatives for training and testing robot algorithms before real-world deployment. The Universal Robot Description File (URDF) is an XML-based format used to describe robot kinematics and physical properties, consisting of links (physical components) and joints (connections between links), with common joint types including revolute (rotational), prismatic (linear), continuous (unlimited rotation), and fixed joints.

A policy is a mapping from agent states to actions (or probabilities over actions for stochastic policies). It defines the agent's behavior strategy - what action to take in each situation. Policies can be deterministic (selecting a single action per state) or stochastic (selecting actions probabilistically). The goal of reinforcement learning is often to find the optimal policy that maximizes expected cumulative reward. Policies can be represented using various methods including lookup tables, neural networks, or other function approximation techniques.

Reinforcement learning is a branch of machine learning where agents interact with unknown environments to maximize cumulative reward over time. Deep reinforcement learning extends this by using neural networks as function approximators for high-dimensional inputs. The field sits on a spectrum of increasing complexity: supervised learning provides clear input-output pairs, contextual bandits provide inputs without correct outputs, and reinforcement learning involves stateful environments where inputs depend on previous actions, requiring long-term planning due to delayed effects. The Markov Decision Process (MDP) provides the formal framework, defined by state space S, action space A, transition probabilities P(s'|s,a), reward function R(s,a,s'), and initial state distribution μ. Policies determine how agents choose actions: deterministic policies map states directly to actions, while stochastic policies output probability distributions. Neural networks represent policies similarly to supervised learning—outputting probability vectors for discrete actions or Gaussian distributions for continuous actions. Policy gradient methods directly optimize policy parameters θ to maximize expected reward. The score function gradient estimator provides the mathematical foundation: ∇_θ E[f(X)] = E[f(X) × ∇_θ log p_θ(X)]. Applied to RL, this yields the policy gradient formula: ∇_J(θ) = E[∑_t ∇_a log π_θ(a_t|s_t) × G_t], where G_t is total discounted reward from time t. This treats good trajectories as positive examples, training the policy to reproduce successful actions.

Reinforcement learning is a machine learning paradigm that learns through environment interaction rather than supervised training data. It mirrors how children learn new games by experimenting with actions and learning from rewards/penalties. The core components include actions (available choices), rewards (feedback on action quality), and game states (current situation). A policy represents the decision-making strategy, determining which action to take given a state. The fundamental goal is to maximize cumulative rewards over time through iterative improvement of this policy.

Machine learning enables machines to learn from experience without explicit programming. It encompasses three main branches: supervised learning (using labeled data for prediction tasks like regression and classification), unsupervised learning (finding structure in unlabeled data through clustering and dimensionality reduction), and reinforcement learning (learning from environmental interactions). Reinforcement learning distinguishes itself by having agents learn optimal behaviors through trial and error, receiving rewards for appropriate actions and penalties for inappropriate ones. Key elements include the agent (decision-maker), environment (interactive setting), state (current situation), action (interaction method), reward (environment feedback), policy (action selection strategy), and value function (action evaluation). Model-based agents maintain explicit environmental models, while model-free agents directly compute value functions and policies. This foundational understanding prepares learners for advanced topics in sequential decision-making.

Reinforcement learning (RL) is a type of machine learning where an agent learns to make sequential decisions in an environment by receiving occasional rewards as feedback, rather than being provided with labeled training data like in supervised learning or unlabeled data like in unsupervised learning; the agent observes the environment state, takes actions, receives rewards, and aims to maximize expected cumulative rewards through a policy function that maps states to actions, facing key challenges including the credit assignment problem (determining which past actions led to outcomes) and the exploration-exploitation trade-off (balancing trying new actions versus using known effective ones).

Robot dynamics incorporates gravitational forces, centrifugal forces, Coriolis forces, and moments of inertia into equations relating joint torques to angular accelerations. Position control uses feedback loops comparing reference positions with encoder measurements to generate error signals and corrective actuator commands. Force control regulates interaction forces for tasks like grinding and enables safe human-robot collaboration through impedance control. The integrated control system architecture combines path planning, trajectory generation, inverse kinematics, and joint controllers in a feedback loop, with forward and inverse kinematics solved repeatedly during operation.

A control system consists of a plant, reference input (R), and measured output (Y), with the goal of making Y match R using control input U. Feedback control measures the current state, compares it to the desired state, and adjusts the control input based on the error. PID (Proportional-Integral-Derivative) control combines three components: P responds to current error, I accumulates past errors to eliminate steady-state error, and D predicts future error based on rate of change. The controller output is U(t) = KP × e(t) + KI × ∫e(τ)dτ + KD × de(t)/dt. Robotic manipulators are multi-input multi-output (MIMO) systems where inputs are joint torques and outputs are joint positions and end-effector positions. Robot dynamics are described by τ = M(θ)θ̈ + C(θ, θ̇)θ̇ + G(θ), relating joint torques to positions, velocities, and accelerations. A second-order linear system is described by mẍ + bẋ + kx = F, where the characteristic equation mλ² + bλ + k = 0 determines system behavior. The damping ratio ζ = b/(2√(mk)) determines stability and response characteristics.

Control loops enable robots to perform tasks automatically with fine control, reducing driver workload. Feedback uses sensors to gather information about the robot's state and environment. Without sensors, there is no feedback, and the robot cannot know its actual state. The computer gives instructions, and the robot checks if it is actually doing what was commanded based on sensor information.

Robot dynamics is governed by Newton's second law (F=ma), where applied force is consumed in three ways: accelerating links, overcoming Coriolis/centrifugal forces, and countering gravity. This creates n coupled nonlinear second-order differential equations for n-jointed robots. Inverse dynamics converts desired trajectories to torque commands for position control, while forward dynamics predicts trajectories from torques for torque-controlled applications. These concepts form the mathematical foundation for robotic motion control systems.

Feedback control compensates for errors by reacting to measured quantities. Different sensors are used: visual sensors on end effectors for vision-based feedback, force sensors for contact tasks. The default minimum measurements include joint position encoders. Controllers are designed at the torque level, outputting torques actuators should produce to achieve desired behavior. This compensates for errors and handles uncertainties in the robot's physical properties.