Robotic Hand-Eye Calibration: Precision Alignment Demo

Visual servoing uses visual feedback to control robot motion, compensating for kinematic errors, sensor inaccuracies, and execution errors. Two main configurations exist: external cameras (fixed in environment) and onboard cameras (mounted on robot arm). Position-based visual servoing determines 3D object positions from visual features and transforms them to robot coordinates, generating motion commands to reach target positions. Image-based visual servoing controls motion based on differences between current and desired image configurations, abstracting away explicit 3D reconstruction. The image Jacobian matrix relates 3D point velocities to image plane velocities, enabling control law derivation as v = J⁺(x)ᵀe. Modern systems combine visual, force, and joint encoder data for robust manipulation in unstructured environments.

Two main visual servoing approaches exist: image-based and position-based. Image-based control uses features directly in image space, calculating errors between desired and actual positions to generate velocity commands. This approach is robust to calibration errors but produces non-intuitive movements. Position-based control converts features to 3D coordinates before generating commands, requiring accurate calibration but producing more intuitive movements. The interaction matrix maps camera motion to feature motion in image space, enabling appropriate control command generation.

Two main approaches exist for visual servoing: Position-Based Visual Servoing (PBVS) reconstructs 3D object geometry from multiple images taken at different viewpoints, then controls the robot based on this 3D model; Image-Based Visual Servoing (IBVS) controls the robot directly using 2D image features without explicit 3D reconstruction. PBVS uses structure from motion techniques to triangulate 3D feature points and create surface models, while IBVS focuses on specific feature points and defines desired positions for them. Blob detection calculates centroid location and orientation using moments of inertia. Feature point matching algorithms like SIFT and SURF detect distinctive points robust to rotation, scaling, and translation, enabling correspondence between images. IBVS avoids the computational complexity of 3D reconstruction while still achieving precise visual guidance.

This video demonstrates two visual servoing approaches for controlling a 2-degree-of-freedom planar robot arm to center a specific world point in the camera image. In Image-Based Visual Servoing (IBVS), the system directly uses pixel coordinate errors to compute end-effector velocity commands through the interaction matrix L, requiring direct velocity control. In Position-Based Visual Servoing (PBVS), the desired image coordinates are back-projected into world coordinates to determine the desired camera pose, then joint angles are computed using inverse kinematics and controlled via feedback linearization with PD control. IBVS offers faster computation and better tolerance to model errors, while PBVS provides higher accuracy in world coordinates and works with torque-controlled robots lacking direct velocity control.

Traditional fuselage assembly uses laser trackers requiring 2+ hours per setup. Airbus developed visual servoing with multiple cameras for real-time point tracking at 10Hz, enabling dynamic positioning to achieve 0.1-0.5mm tolerances. The system calculates errors directly in the image plane without camera calibration, simplifying setup. Combined with profilometers for flush verification and global measurements for warping detection, setup time reduced from hours to under 2 minutes. This represents a fundamental shift from point-by-point scanning to continuous visual feedback control.

The Iterative Closest Point (ICP) algorithm is a local registration technique used to estimate the pose (position and orientation) of an object point cloud relative to a scene point cloud by iteratively finding nearest-point correspondences and computing transformations; unlike global registration, ICP requires a reasonable initial guess of the object's position, and its performance can be significantly improved by using point-to-plane ICP (which utilizes surface normals) rather than point-to-point ICP, achieving better alignment with fewer iterations.

Geometric perception enables robots to estimate object poses by matching 3D point clouds from depth sensors (like Intel RealSense cameras) to known object models, using mathematical optimization techniques such as Singular Value Decomposition (SVD) to solve the inverse kinematics problem of finding the best rigid transformation that aligns observed scene points with model points, with the Iterative Closest Point (ICP) algorithm providing an iterative refinement approach that alternates between assigning correspondences and optimizing the pose estimate.

This lecture continues geometric perception, connecting to kinematics work on robot motion. The major theme is estimation with messy point clouds. RGBD sensors (like D415) provide color and depth images transformed into point clouds. The Iterative Closest Point (ICP) algorithm alternates between optimizing pose given correspondences and finding correspondences given pose. Each subproblem has a global optimum, but alternation can cause local minima. This is a common pattern in optimization, seen in EM algorithm and bilinear optimization.

The Iterative Closest Point (ICP) algorithm is a popular method for aligning point clouds by iteratively finding correspondences between points in two sets and optimizing the translation and rotation to minimize distances between matched points, though it may converge to local minima if initial estimates are poor.

The Iterative Closest Point (ICP) algorithm is a fundamental technique for aligning 3D point clouds by iteratively finding correspondences between points and computing the optimal transformation (rotation and translation) to minimize the squared error between corresponding points; the algorithm works by first computing the center of mass of both point clouds, then using Singular Value Decomposition (SVD) to solve the orthogonal Procrustes problem for the rotation matrix, and finally applying the transformation to align the point clouds, with the process repeating until convergence or until the error falls below a threshold.

This video demonstrates how artificial vision systems enable automated metal processing through two key applications: 3D bin picking uses 3D scanning and matching to locate and coordinate robotic arms for picking metal pieces, while 3D profile measurement combined with Deep Learning enables comprehensive weld inspection by capturing complete weld images and detecting defects, and pericentric optics with Deep Learning allows inspection of internal thread features without physical access.

Calibration is a critical challenge in bin picking applications. Modern vision systems provide automated calibration interfaces that simplify the process: users simply take approximately 15 pictures of the workspace, and the system calculates and validates robot and scanner parameters. After capturing reference points, the interface indicates whether calibration was successful or if adjustments are needed, significantly reducing setup time and complexity.

Vision calibration in pick and place machines improves placement accuracy by using computer vision to detect and correct for component position and rotation errors that occur during the picking process, compensating for tolerance stack-up from feeders, nozzles, and mechanical systems; this is achieved through three main calibration types: lens distortion correction (flattening image warping), fiducial calibration (aligning the board coordinate system), and bottom vision (detecting component position and rotation on the nozzle).

A 3D vision system for robotic bin-picking uses CAD-based 3D shape matching to precisely locate randomly oriented parts in bins, enabling robots to identify and pick parts efficiently by calculating optimal gripping positions based on part geometry and verifying clear access points before execution.

Vision system calibration establishes the mapping between pixel coordinates (vision system output) and millimeter measurements (robotics coordinate system). Since vision systems operate in pixels while robots work in millimeters, calibration creates a transformation matrix that converts between these coordinate systems. The calibration process requires providing corresponding pixel and millimeter data points, after which the vision software computes the transformation. This mapping enables accurate position reporting from vision detection to robot motion commands, forming the foundation for all guided placement applications.