Stereo Camera Mapping with RTAB-Map ROS Tutorial

Searching for loop closures among thousands of nodes is combinatorial and exponentially grows. Practical solutions include: (1) Using a map to limit search to the robot's neighborhood; (2) Using semantic knowledge (like recognizing rooms or areas) to guide data association; (3) Visual place recognition methods like bag of words, which convert images to visual word histograms and compare against a library to find matches.

RTAB-VSLAM (Realtime Appearance-Based Visual Simultaneous Localization and Mapping) is an open-source library that creates 3D maps while simultaneously tracking robot position in real time. The system employs intelligent memory management with two distinct memory types: short-term memory handles continuous data matching for real-time map building, while long-term memory detects loop closures when the robot revisits previously seen areas, ensuring fixed-time processing for large-scale environments. The feature-based mapping algorithm extracts distinctive visual features from camera images, matching them across timestamps to detect environmental changes. Short-term memory updates positioning points dynamically, while long-term memory integration improves accuracy when matching future data. Best practices for optimal mapping include reducing robot speed to minimize odometry error, starting mapping near straight walls or enclosed areas, performing complete 360° scans, and completing full mapping loops before focusing on details. The system automatically saves maps upon session termination.

Autonomous navigation requires creating a map first. The mapping process involves launching RTAB-Map alongside the move_base node from the navigation stack, with the grid map topic remapped to the move_base node's map topic parameter. The robot explores the environment using keyboard teleop, and RViz visualizes the map creation process. After mapping, all data is stored in a database containing images with yellow disks indicating key features (corners, intersections, distinctive objects). The database viewer displays images chronologically. Pink disks indicate loop closures - when two images share matching key features, the system detects the robot has returned to a previously visited location, enabling position correction and accurate long-term mapping.

Back-end optimization addresses accumulated drift errors using loop closure information. The system identifies when the robot returns to previously visited locations and uses this constraint to globally optimize the trajectory. Two main approaches exist: filter-based methods (Kalman Filters, Particle Filters) and optimization-based methods (Bundle Adjustment, Pose Graph Optimization). Optimization-based methods have become dominant due to improved computing power and the ability to exploit sparsity in the problem structure. Loop closure detection compares current sensor data with historical data to identify matching locations, enabling global trajectory correction and consistent map generation.

When the robot returns to an area it has previously scanned, overlapping submaps can be registered together to create loop closure constraints. This registration process determines the offset between poses from different submaps, providing additional constraints that eliminate drift. The resulting graph is called a pose graph because it only involves poses and constraints between them, without explicitly modeling landmarks, reducing optimization complexity.

Kimera-VIO implements Visual-Inertial Odometry (VIO) by fusing stereo camera data with IMU measurements to estimate 3D robot poses and reconstruct 3D landmarks. The system operates as two components: a standalone C++ library containing core mathematical algorithms, and a thin ROS wrapper for interfacing. The pipeline processes stereo images, extracts visual features, tracks them across frames, and estimates the camera trajectory in real-time. Feature states are visualized with color coding: red for undetected features, blue for tracked features below threshold, and green for reliable tracked features. Users can switch between point cloud and mesh visualization modes during runtime. The output provides sparse point cloud representations suitable for obstacle avoidance applications.

Visual-Inertial Odometry (VIO) combines visual SLAM with IMU (Inertial Measurement Unit) measurements through tight sensor fusion in a unified optimization framework. The system jointly optimizes camera poses, 3D landmarks, IMU biases, and velocities within a factor graph. This approach leverages the complementary strengths: cameras provide absolute position and landmark observations, while IMUs provide reliable short-term orientation and acceleration measurements. Estimating IMU biases continuously is critical because uncorrected biases cause rapid divergence. The scale ambiguity of monocular vision is resolved using accelerometer measurements over time.

Visual-inertial fusion combines camera and IMU measurements to estimate robot state (position, velocity, biases). The mathematical formulation involves solving for unknowns including absolute scale and IMU biases using inertial and visual measurements. Martinelli's work enables closed-form initialization from three consecutive views with sufficient acceleration. Two implementation strategies exist: filter-based methods (MSCKF) tracking ~20 features for efficiency versus optimization-based approaches maintaining sliding windows or full trajectories for higher accuracy. The choice depends on real-time performance versus accuracy trade-offs, with optimization approaches using iSAM achieving real-time operation at 50Hz through incremental updates.

This video demonstrates a stereo Multi-State Constraint Kalman Filter (MSCKF) based Visual-Inertial Odometry (VIO) system that achieves robust state estimation for autonomous flight, showing successful navigation through unstructured environments with drift of only 0.5% over 700-meter trajectories while maintaining computational efficiency comparable to state-of-the-art monocular solutions.

This section introduces the core motivation and methodology for combining deep learning with classical state estimation in VIO. Monocular VIO uses one camera and IMU to improve performance and robustness over classical methods that rely on manually crafted pipelines prone to errors in rapid motion, exposure changes, and low texture environments. The proposed approach uses a robocentric EKF where the IMU kinematic model provides prediction, while learned relative poses and covariances enable measurement updates. This hybrid architecture leverages deep learning's strength in handling high-dimensional image data while maintaining the physical plausibility of classical EKF frameworks.

The Jetson Nano is NVIDIA's smallest computer at $99, running the entire CUDA-X stack including computer vision and speech recognition. It enables 200,000 developers across 2,000 companies to build robots for warehouse logistics, agriculture, retail, and industrial applications. The Isaac robotics platform provides three reference robots (Kaya, Carter, Link), a robotic simulator for virtual training, and tools for reinforcement learning. NVIDIA DRIVE is an autonomous vehicle platform consisting of: DGX Saturn 5 supercomputer for AI training, ensemble AI algorithms for perception/localization/path planning, DRIVE AV Constellation simulation platform for testing corner cases, and DRIVE IX for human-machine interface. Release 9 provides Level 2 capabilities including off-ramp/on-ramp, auto lane change, real-time mapping, and driver monitoring.

This video demonstrates how NVIDIA Jetson enables deployment of large language models (LLMs) and vision transformers (VITs) at the edge through optimized software stacks (Jetpack 6), quantization techniques (4-bit quantization reduces 70B LLM memory from 140GB to ~35GB), and integrated tools like TensorRT, Riva for speech, and Isaac ROS for robotics, achieving real-time performance (95 FPS for VITs, interactive rates for LLMs) while maintaining low latency, privacy, and offline operation for applications like robotics and autonomous systems.

This video demonstrates how to build an autonomous AI robot using an NVIDIA Jetson Nano single-board computer, where the robot learns to avoid obstacles by training on approximately 600 images using PyTorch deep learning framework; the robot uses its camera to perceive its environment and make decisions about movement based on learned patterns, illustrating the fundamental concept of artificial intelligence where machines can perceive their surroundings and take actions to achieve goals.

The NVIDIA Jetson Orin Nano is a compact edge computing device featuring 6 ARM cores and 1024 CUDA cores that enables powerful AI applications like real-time vehicle detection and large language model inference at affordable prices, demonstrating how edge devices can perform sophisticated machine learning tasks locally without cloud dependency.

NVIDIA's Jetson Orin platform offers two developer kits (Nano and AGX Orin) featuring ARM-based processors with Ampere generation GPUs, delivering 40-275 TOPS of AI performance while maintaining low power consumption (7-60W), making them suitable for edge AI applications and robotics development.