Computer Vision: 3D to 2D Projection with Camera Matrices

Epipolar geometry describes the geometric relationship between stereo images, where a point in one image corresponds to a line (epipolar line) in the other image, enabling efficient feature matching and 3D reconstruction; this is implemented using the fundamental matrix (F) computed from matched points via SIFT detection and RANSAC algorithm, allowing visualization of corresponding epipolar lines across stereo image pairs.

Epipolar geometry describes the geometric relationship between two images of the same scene captured from different viewpoints. Unlike optical flow which allows correspondences anywhere in the image, epipolar geometry constrains correspondences to specific lines called epipolar lines. The baseline connects the two camera centers, and a family of planes pivoting around this baseline generates conjugate epipolar line pairs. Every point on one line corresponds to a point on its conjugate line in the other image. This transforms the 2D correspondence problem into a series of 1D searches, making correspondence finding computationally more efficient and enabling more accurate matching algorithms.

Epipolar geometry describes the geometric relationship between two camera views, where corresponding points in stereo images must lie on specific epipolar lines; this constraint reduces the search for matching points from a 2D problem to a 1D search along these lines, with the fundamental matrix elegantly capturing all the geometric relationships between camera intrinsics and extrinsics without depending on the specific 3D scene being observed.

Stereo 3D vision uses two cameras positioned at known locations to reconstruct 3D scenes by finding corresponding points between left and right images; the key mathematical concept is epipolar geometry, which restricts the search for matching points to specific lines (epipolar lines) rather than the entire image, making the correspondence problem computationally tractable.

Epipolar geometry describes the geometric relationship between two camera images, where corresponding points lie on epipolar lines determined by the epipolar plane (defined by the 3D point and two camera centers) and epipoles (projections of camera centers onto each other's image planes). The fundamental matrix F is a 3x3 matrix that maps a point in one image to its corresponding epipolar line in the other image, satisfying rank 2 and determinant 0 constraints with 7 degrees of freedom. The essential matrix is a special case of the fundamental matrix for normalized image coordinates, representing the camera pose as rotation and translation. These concepts enable stereo vision by constraining correspondence search to epipolar lines, reducing computational complexity and enabling depth estimation through triangulation.

Several lens distortion models are available: radial and quadratic (solves for pin cushion or barrel distortion), radial and division quadratic (solves for both high and low distortion values), and radial and tangential (for off-center film scans). The division quadratic method is technically more accurate. After solving, the solve tab displays average solve error, focal length, lens distortion value, and seed frames used. To refine the solve, adjust track filtering settings and re-solve. Lowering the maximum solve error and re-solving can improve accuracy. The focal length and distortion values may change slightly with each solve attempt.

Real-world cameras exhibit nonlinear errors primarily due to lens imperfections. Barrel distortion causes straight lines to appear curved outward in wide-angle lenses. Two approaches address nonlinear distortions: (1) Physically-motivated models use mathematical descriptions of physical phenomena (like radial distortion polynomials), and (2) Empirical models fit correction parameters to observed distortion patterns. Both approaches apply location-dependent pixel shifts, where correction magnitude depends on radial distance from the principal point. The iterative inversion process handles these nonlinearities by refining estimates until convergence.

Real-world cameras exhibit lens distortion, primarily radial distortion where points appear to bulge outward from the image center. The distortion depends on the radial distance r from the center: x_distorted = x(1 + k₁r² + k₂r⁴) and similarly for y. To correct for this, one captures images of a known planar pattern (like a checkerboard), observes the distorted pattern, and estimates the distortion coefficients k₁ and k₂ by solving a least-squares optimization problem that minimizes the difference between expected and observed corner positions.

Lens distortion correction exploits the radial symmetry of optical systems. Since distortion patterns are symmetric around the image center, measuring distortion along a single line from the center is sufficient to characterize the entire pattern. The correction algorithm applies a radius-based correction formula that depends on the distance from the image center. Polynomial formulas approximate the distortion pattern, and look-up tables provide efficient pre-calculated corrections instead of real-time computation for each pixel.

Real cameras require extending the ideal pinhole model to include radial and tangential lens distortions. Radial distortion causes bending of light rays near lens edges, modeled with coefficients k1, k2, and optionally k3. Tangential distortion arises from lens-image plane misalignment, requiring only p1 and p2 parameters. Together with intrinsic and extrinsic parameters, these form a complete camera model for accurate real-world representation.

Structure from motion reconstructs 3D scenes from 2D image collections by detecting features across multiple views, matching descriptors between images, and estimating camera geometry and scene structure. Visual Simultaneous Localization and Mapping (SLAM) extends this by combining stereo matching (to get 3D points from left-right camera pairs) with temporal matching between consecutive frames to estimate camera motion and build dense disparity maps for real-time navigation.

This section introduces Structure from Motion (SfM), which recovers 3D structure and camera motion from a sequence of images taken by a single moving camera. Unlike stereo vision (two cameras at the same time), SfM uses temporal variation. The goal is recovering camera positions and scene structure from point correspondences. The problem assumes rigid objects that don't deform. The mathematical formulation involves recovering camera matrices A_j and 3D points P_i from correspondences P_i^j across M cameras. The section establishes the fundamental problem statement for SfM.

Visual SLAM (V-SLAM) uses cameras instead of LIDAR, providing only bearing angles without range information. Monocular V-SLAM loses range entirely, while stereo/multi-camera systems recover depth through disparity. Monocular V-SLAM is mathematically equivalent to structure from motion in computer vision - estimating camera positions and reconstructing 3D scenes from multiple 2D images. Unlike deterministic structure from motion, V-SLAM incorporates probabilistic frameworks including process noise and measurement noise. In V-SLAM, the camera pose has more degrees of freedom: position (X, Y, Z), orientation (unit quaternion), linear velocities, and angular velocities. Landmarks are 3D points (X, Y, Z) rather than 2D line segments. Unit quaternions represent orientation to avoid gimbal lock. The constant acceleration motion model describes how velocities change linearly and positions change quadratically over time.

Structure from Motion (SfM) reconstructs 3D scenes from multiple 2D images using camera motion. The process involves detecting feature points, finding correspondences, and computing camera poses (rotation and translation matrices). The Rome reconstruction used 150,000 photographs over 26 hours, with 18 hours on stereo matching and 8 hours on reconstruction using 496 processors. SfM can reconstruct both large scenes and small objects like busts (20 images). The result is a 3D point cloud showing camera poses and scene geometry, importable into 3D modeling software like Blender.

Structure from motion is the process of reconstructing 3D world coordinates from 2D image observations. SLAM (Simultaneous Localization and Mapping) is a related concept that combines estimating the camera's position (localization) with building a map of the environment (mapping). These techniques are fundamental to autonomous vehicles, which must determine their position while simultaneously creating a representation of their surroundings.