Unity XR Toolkit: Configuring Action-Based Input Effectively

Creating interactable objects involves adding Rigidbody components, Materials, and XR Grab Interactable components. XR Ray Interactors enable distance-based grabbing, causing objects to become tracked to controllers and interactable through physics. The World Interaction Demo showcases multiple interaction types: three Grab Interactable types (Velocity Tracked, Kinematic, Instantaneous), UI Interaction with World Space Canvas, teleportation for locomotion, Snap Interactions with sockets for object placement, and Complex Grab Interactions with state-based behaviors and secondary functionality like trigger-activated actions.

The XR Socket Interactor in Unity's XR Interaction Toolkit automatically grabs grabbable objects that fall into its trigger area, enabling hands-free object holding for inventory systems or puzzle mechanics; however, without layer masks, it cannot differentiate between object types, so developers must configure Interaction Layer Masks by assigning specific layers to both interactors and interactables to ensure only compatible objects can interact with each other.

This section covers the foundational concepts of implementing ray-based interaction in Unity VR. It explains how to create two XR Ray Interactors (left and right) attached to camera offsets, noting that only one base interactor can exist per hand. The tutorial demonstrates copying component values from existing direct interactors to properly configure ray interactors. Key attributes discussed include the Interaction Manager, Layer Mask for determining interactable objects, Force Grab for distant object selection, Raycast Configuration for line types (projectile/Bezier curves), and Select Configuration. Visual components like Line Renderer and XR Interactor Line Visual are covered, along with audio and haptic event systems for providing feedback during interactions.

To restrict XR sockets to only interact with specific objects, create a script that inherits from XR Socket Interactor, add a public string field for the target tag, and override the CanHover and CanSelect functions to check if the interactable's tag matches the target tag using Unity's CompareTag function; for newer XR Toolkit versions, create a custom inspector in an Editor folder that inherits from the appropriate XR Toolkit editor class to display the tag field in the Unity Inspector.

OpenXR provides standardized XR development through three main platforms: OpenXR playground, Apple playground, and PlayStation VR. Sony participates in OpenXR consortium, and PSVR2 can operate in OpenXR mode. OpenXR 1.1 introduced the generic controller with trigger, grab, primary/secondary buttons, joystick, and menu buttons. This replaces the original simple controller. Developers should support the generic controller profile and only add additional interaction profiles after physical testing. Extension availability varies by headset, runtime, and engine, requiring technical producers familiar with platform-specific capabilities.

For VR projects using OpenXR, additional configuration is required beyond standard input setup. Create a Player Mappable Input Config in Content Browser, set the config name, and add your mapping context to the Context section. Then navigate to Project Settings > OpenXR Input and assign your new config. Without this step, VR controllers like Valve Index may not work properly even if standard inputs function. This demonstrates how platform-specific configurations are necessary for different input systems.

This comprehensive process covers the essential Unity XR configuration steps for VR game development. First, navigate to Edit > Project Settings > XR Plug-in Management and install the XR Plug-in Management package. Then install OpenXR packages by clicking the Open XR button. Enable backends and restart Unity. Configure Oculus settings by setting Multi View to Multipass and enabling Low Overhead Mode. Set OpenXR Pass Instance to Multipass. Add VR device interaction profiles including Oculus, Meta Quest Pro, Valve Index, and HTC Vive, plus hand interaction. Finally, set Android Render Mode to Multipass. These configurations establish the foundational VR infrastructure needed for Gorilla Tag locomotion implementation.

Unity's 2021 VR development updates introduce two major changes: the new Action-Based Input System and OpenXR support. The new input system organizes VR interactions into action maps (head, left hand, right hand) with actions and bindings, replacing the traditional input system for better multi-device compatibility. OpenXR support, added in Unity 2020.2, enables VR development across multiple headsets including Valve Index and HTC Vive through the XR Plugin Management system. To implement, developers must enable the new input system in project settings, create an Action-Based XR Rig using the XR Toolkit, assign default input actions, and attach an Input Action Manager script to access controller inputs programmatically.

This segment covers OpenXR cross-device development and Unreal Engine's Enhanced Input system. OpenXR enables developers to implement functionality once and have it work across different VR headsets without writing device-specific code. The runtime handles controller offsets and hardware differences. Enhanced Input uses action mappings to bind controller inputs to game actions, abstracting away device-specific details. Developers should only add bindings for controllers they have access to and tested on. The system handles dead zones and input modifiers automatically. The Input Mapping Context allows customization of input behavior for specific interaction profiles, such as different dead zone thresholds for different controllers.

Force feedback systems are categorized as passive (brakes that resist user-applied force) or active (motors that actively move fingers). HaptX chose passive architecture for safety—nothing can hurt the user unless dropped. The complete system integrates electromagnetic motion tracking (sub-millimeter precision), pneumatic valve arrays (144 valves controlling tactors), and force feedback ribbons behind fingers. Software models virtual object surfaces and commands appropriate pressure patterns based on user hand calibration. This integrated approach recreates the physical deformation of real objects on the user's skin rather than simulating sensations through vibration, achieving unprecedented realism in VR haptic experiences.

This section covers interactive XR elements. XR Controller tracks position/orientation with system-determined values; XR Camera tracks user gaze and head position. Input Actions enable reacting to controller inputs with Action IDs, signals, boolean properties, and analog values for thumb sticks and triggers. Different platforms require lists for compatibility. For 2D UI, Qt Quick controls work inside 3D worlds with XR Item specifying physical size in centimeters. XR Virtual Mouse enables mouse events from controller pointing and button presses. Hand tracking represents hands as controllers with pinch and grip gesture IDs. XR Hand Model shows hand models in VR (Quest-specific). Touch events use Process Touch function with hand position and touch ID, returning offset for surface interaction illusion.

SteamVR now supports native hand tracking on Quest 3 devices through the Steam Link and SteamVR beta channels, enabling users to experience hand presence in VR games without physical controllers by utilizing skeletal hand input; this feature works best in games like Half-Life Alyx that support skeletal input, though users may need to customize controller bindings and encounter limitations such as reduced accuracy when reaching over shoulders or using palm-facing movements.

F1 2016 implements haptic feedback in the controller where the controller vibrates on the side corresponding to the direction the car is turning. When turning left, the left side of the controller vibrates, and when turning right, the right side vibrates. This provides players with tactile feedback about the car's movement and direction.

Update 16 introduces significant controller customization options, allowing users to swap button positions between left and right controllers and adjust vibration feedback intensity for personalized haptic experiences. The Guardian boundary system now supports hand tracking for setup, eliminating controller dependency. New home environments include a spaceship bridge theme, while the spatial menu system overlays navigation directly within the user's physical room using testroom cameras. These features enhance user personalization and provide more intuitive ways to interact with the VR environment.

The Unified Menu Container (UMC) is Story Framework's centralized widget container for managing game menus using input-based navigation (A/D keys) instead of mouse controls. Each UMC entry consists of a tab name, tab widget, and tab input mapping context. To add custom widgets, create a User Interface Widget Blueprint with a Canvas Panel containing an Overlay and centered Image, then link it to the UMC through Story Framework values. Create Input Mapping Contexts (IMCs) to define navigation inputs. Custom widgets can implement UMC Info interface events ('On Tabbed To', 'On Tabbed Away') for custom logic. Navigation inputs (navigate_up, navigate_down, navigate_left, navigate_right) control movement between menu items. To enable navigation between custom widgets, create custom events in the Inventory Manager that accept float direction inputs, allowing bidirectional communication between the menu system and custom widgets.

Spatial computing supports multiple input types: tap gestures (look-and-tap selection, up to two simultaneous actions), hand and head pose tracking (precise joint and position data via Hands package and Input System), AR data (planes, world mesh, image markers via ARKit/AR Foundation), and Bluetooth devices (keyboards, controllers via Input System). Tap gestures require input colliders on objects. Hand tracking and AR data are only available in unbounded volumes and require permissions. Developers must choose between bounded volumes (coexisting with other apps) and unbounded volumes (full immersion with environmental interaction). Existing touch interactions can adapt by adding input colliders and using tap as primary input.

Android XR is a new operating system for Virtual, Mixed Reality, and Spatial computing that enables familiar 2D Android applications to exist within spatial windows, allowing users to move and resize them with natural hand and voice inputs; powered by Gemini AI, it provides multimodal conversational experiences including spatial video playback, immersive map navigation, contextual information retrieval about landmarks, and automatic spatialization of photos and videos to create immersive digital reconstructions of physical spaces.

Overlay menus require disabling mouse input to the game while the menu is open. This is achieved by hooking two functions: (1) MouseInputEnabled - returns false when the menu is open to block mouse input, and (2) SetRelativeMouseMode - stores the current mouse mode and returns false when the menu is open. When the menu closes, the stored mouse mode is restored. Both hooks use fastcall calling conventions and require the address-of operator on function pointers. The mouse mode restoration must occur both when the menu is closed via the insert key and when the close button is clicked.

The Enhanced Input system enables seamless multi-platform interaction through dynamic mapping contexts. Base contexts handle universal inputs (movement, rotation, zoom), while mode-specific contexts (Villager Mode, Build Mode) add platform-appropriate controls. Contextual input handling distinguishes between selecting villagers versus empty space, triggering appropriate responses. Build mode validates actor placement through overlap checks and floor collision traces. This architecture supports keyboard, mouse, gamepad, and touchscreen inputs within unified blueprints, enabling intuitive player interactions across diverse hardware.