Teleoperating a UR5 Robot with Oculus Quest 2 VR

Joint space represents the configuration of all robot joints, with one dimension per joint (e.g., 6 dimensions for a 6-DOF robot). Cartesian space represents the end effector's position and orientation in 3D space, with maximum 6 degrees of freedom (3 translations + 3 rotations). Planar robots have only 3 Cartesian DOF (x, y, and rotation about z). Controlling a robot involves either joint control (specifying joint angles) or Cartesian control (specifying end effector position/orientation). The PUMA 560 exemplifies a 6-DOF industrial robot capable of reaching any position and orientation in 3D space.

This comprehensive section distinguishes between joint space and task space control paradigms. Joint space control directly commands joint angles, while task space control specifies desired end-effector positions (x,y) in Cartesian coordinates. The video explains how forward kinematics X = f(q) maps joint angles to end-effector positions, and how the Jacobian J = ∂f/∂q relates velocities. The critical insight is that task space control requires computing joint references through inverse kinematics and Jacobian inversion: q̇_ref = J⁻¹Ẋ_ref and q̈_ref = J⁻¹(Ẍ_ref - J̇q̇_ref). The section demonstrates this with a lemniscate (figure-eight) curve, showing how symbolic computation automates derivative calculations. The implementation shows how the same feedback linearization framework applies, with the control law using the computed joint references to achieve precise Cartesian trajectory tracking despite gravitational and inertial effects.

Robot kinematics studies functional relationships between joint space variables (thetas and ds) and task space variables (position and orientation). Joint space contains joint variables, task space contains position (x, y, z) and orientation (rotation matrix elements). Actuator space differs from joint space due to mechanical linkages and gearboxes between motors and joints. If actuator space dimension exceeds Lambda, the manipulator is redundant; if less, it is underactuated.

Joint space control generates trajectory errors directly in joint coordinates, while Cartesian space control starts with Cartesian trajectories and inverts kinematics to obtain joint space references. Even with Cartesian data, if error is generated in joint space, it is classified as joint space control. For square non-singular Jacobians, velocity-level inversion is needed initially, followed by continuous acceleration-level computation. The feedback linearization control law can be implemented efficiently with complexity linear in the number of joints, using actual measurements instead of desired acceleration for real-time computation.

Joint space control converts desired Cartesian positions to joint coordinates using inverse kinematics, then generates trajectory commands. Cartesian space control generates Cartesian trajectory commands directly, requiring forward kinematics to convert joint measurements. The controller generates forces in Cartesian coordinates, which must be transformed to joint coordinates using the Jacobian transpose for robot actuation.

Unity 3D is a powerful game development tool that supports both 2D and 3D games, using C# as its primary programming language (more powerful and stable than JavaScript). The Unity interface consists of several key views: Hierarchy (shows objects in the current scene), Game (displays the final game view), Inspector (shows properties and components of selected objects), Project (contains all game assets like images, sounds, and scripts), and Console (displays error messages and debug output). The Transform component is fundamental for controlling object position, rotation, and scale. This first module introduces the basic Unity interface and C# programming concepts necessary for game development.

Unity is a free, multi-platform game engine developed by Unity Technologies that supports 2D, 3D, and VR/AR development, featuring a C# API, component-based architecture with GameObjects and Scenes, and a vast community providing free and paid assets like models, textures, and scripts to accelerate game development.

This comprehensive lesson covers Unity's interface and core development concepts. The interface includes the menu bar for file operations, Hierarchy panel for object management, Inspector panel for property editing, and the central view area for scene visualization. Objects are created by right-clicking in the Hierarchy panel. The Inspector displays components like Transform (position, rotation, scale), Mesh Filter (shape), Mesh Renderer (visual appearance), and Box Collider (physics). The 3D coordinate system uses red for X-axis, blue for Z-axis, and green for Y-axis. The Gizmo provides visual handles for manipulation. C# scripts are created by right-clicking in the Project panel and selecting 'Create > C# Script'. All scripts inherit from MonoBehaviour, which provides lifecycle methods: Start() runs once when the object is enabled, Update() runs once per frame. The Print() function outputs text to the Console for debugging. Scripts must be attached to objects in the scene by dragging them in the Inspector. To move objects with code, access the Transform component and modify its Position property. The Update() method runs every frame, creating continuous behavior. Vector3 is a 3D vector used for position and direction. C# requires semicolons to end statements, and float numbers must end with 'f'.

Unity is a game engine released in 2005 that revolutionized game development by making it accessible to individuals. Before Unity, game engines were either closed or overly complex. Unity enables single developers to create games for mobile phones, PCs, and consoles. The engine supports cross-platform deployment across Windows, Linux, macOS, Android, iOS, tablets, and gaming consoles. Unity's visual design system uses object dragging and dropping, requiring no programming skills to start. C# scripting controls object behavior including movement and physics. The engine supports user interfaces, modern graphics, animations, cutscenes, and audio. Notable games created with Unity include Firewatch, Portal, Morty, The Forest, and Insurgency. To download Unity, visit unity.com and select the free Personal Edition. Install Unity Hub first, which manages installations and projects. Create a Unity account that syncs across all services. During installation, select Visual Studio Code for C# scripting and target platforms like Android (requiring SDK and GDK) or iOS. Create new projects by selecting templates (3D recommended), naming the project, choosing storage folder, and optionally enabling Unity Cloud for team collaboration.

Unity is a game engine providing all standard development functionality, with over 13% of Steam games created using it. Installation involves downloading from unity.com, running the setup, and accepting terms. Unity Hub manages installations and projects. The editor consists of Scene view (game map), Game view (camera perspective), Inspector (object properties), Hierarchy (object list), and Project window (assets). Navigation uses scroll wheel for camera movement, right mouse button for perspective changes, and Alt key for scene movement. Objects are added via right-click, selecting 3D Objects, and choosing shapes. The Inspector displays position, rotation, and scale values. Materials define visual appearance. Components like Box Collider enable collision detection, Rigidbody enables physics, and Animator controls animations. Objects can be organized into folders. Building a complete game level involves creating the ground, adding visual elements like grass, and combining objects into organized game objects.

OpenXR provides standardized XR development through three main platforms: OpenXR playground, Apple playground, and PlayStation VR. Sony participates in OpenXR consortium, and PSVR2 can operate in OpenXR mode. OpenXR 1.1 introduced the generic controller with trigger, grab, primary/secondary buttons, joystick, and menu buttons. This replaces the original simple controller. Developers should support the generic controller profile and only add additional interaction profiles after physical testing. Extension availability varies by headset, runtime, and engine, requiring technical producers familiar with platform-specific capabilities.

VR development requires understanding hardware and software decisions. Headsets are evaluated by degrees of freedom (3 DOF tracks head rotation, 6 DOF adds body tracking) and connectivity (wired vs wireless). Three major engines exist: Godot, Unity, and Unreal Engine, all free and open source. Unity is recommended for VR due to flexibility, community support, and Android optimization. SDKs like SteamVR and Oculus SDK enable headset integration, but Unity's XR Plug-in Management provides unified platform support. Unity installation requires downloading the editor and configuring components including Visual Studio integration and Android build support. The Unity interface consists of Hierarchy, Inspector, Game window, and Project panels. Transform controls (WASD for position, E for rotation, R for scale) enable object manipulation. VR camera setup requires creating an XR Rig component with camera offset and tracking source configuration. Camera tracking origin determines the reference point for camera movement. VR controllers are configured using the XR Interaction Toolkit, with controllers created as empty GameObjects as children of the camera.

This tutorial demonstrates how to create a functional VR game project in Unity by installing OpenXR and XR Interaction Toolkit plugins, configuring project settings for VR optimization, setting up an XR Rig with controllers and locomotion system, and implementing grabbable objects with XR Grab Interactable components. The process involves enabling preview packages, installing OpenXR and XR Interaction Toolkit, configuring quality settings (pixel light count to 1, anti-aliasing to 4), adding input actions for controllers, creating a basic environment with directional light and ground plane, and implementing locomotion systems like continuous movement with snap turns to prevent motion sickness.

This comprehensive section covers the core foundations of VR development using Unreal Engine's OpenXR template. It begins with an overview of the template's capabilities including teleportation movement, snap turning, component-based grab systems, hand menus with one-handed and laser pointer input, and spatialized audio. The session then progresses through essential scene setup requirements: configuring Player Start capsules or VRPawns with proper Auto Possess Player settings, setting up Nav Mesh Bounds Volumes, ensuring floor meshes have correct collision properties, and properly configuring OpenXR runtime environments for supported devices. The latter portion demonstrates advanced interactive systems including the Grab Component for object manipulation (Free Grab vs Snap Grab modes), the Variant Actor system for cycling between different Actor variants using event dispatchers and custom Blueprints, and the extensible Hand menu system for adding custom functionality through button duplication and On Clicked event implementation.

OpenXR is a cross-platform VR standard that allows the same blueprint code to work across multiple VR devices (Meta, Vive, Valve, Pico, Apple Vision Pro). Building with OpenXR rather than device-specific plugins ensures your VR features work across all supported devices. This is particularly important for hand tracking and other input features that should work consistently across different VR hardware.