Teleoperating a UR5 Robot with Oculus Quest 2 VR

Added:

The Limitations of VR-Based Teleoperation: Haptic Deficit and Cognitive Fatigue

While teleoperating a UR5 robot using consumer VR headsets like the Meta Quest 2 offers an immersive 3D interface, roboticists highlight several critical limitations compared to alternative paradigms. First, consumer VR controllers lack bilateral force feedback (haptics). Without tactile resistance, operators cannot sense contact forces, increasing the risk of damaging the robot or its environment during delicate tasks. Specialized haptic devices provide active force feedback, offering superior precision. Second, latency between the physical robot, the ROS network, and the Unity digital twin, combined with prolonged headset wear, can cause cognitive fatigue, eye strain, and motion sickness. Finally, many experts advocate for 'Shared Autonomy' over direct Cartesian control. By delegating path planning and obstacle avoidance to the robot's local controller while the human provides high-level supervision, shared autonomy reduces operator workload and mitigates the latency issues inherent in real-time VR tracking.

Basics of ROS (Robot Operating System), including nodes, topics, and the publisher-subscriber communication model.
Fundamental concepts of robotic kinematics, specifically the difference between Joint-space control and Cartesian (task-space) control.
Introduction to the Unity game engine, including 3D environment setup, game objects, and basic C# scripting.
Basic principles of Virtual Reality (VR) development, such as tracking, controllers, and integrating SDKs like OpenXR.
Implementing haptic feedback (force feedback) to allow the VR operator to feel physical contact forces experienced by the UR5.
Mitigating network latency in teleoperation using predictive displays and safety-focused virtual fixtures (geofencing).
Utilizing the collected VR teleoperation data for Learning from Demonstration (LfD) to train autonomous robot behaviors.
Integrating computer vision or 3D point cloud streaming (using LiDAR or RGB-D cameras) back into the Unity digital twin for enhanced situational awareness.
516 views10likes2:01@erwinkendoOriginal Release: 2022-08-31

This video demonstrates seamless teleoperation of a UR5 CM3 robotic arm using the Meta Quest 2 VR headset, where users control the robot's end effector position through a virtual marker in a digital twin environment created in Unity and connected to a ROS system via tcp-connector, enabling wireless Cartesian control of the physical robot.