Colmap Photogrammetry Tutorial: From Free Images to 3D Models

This comprehensive lecture covers two revolutionary technologies in AI and computer graphics: NeRF (Neural Radiance Fields) and Gaussian Splatting. Both aim at novel view synthesis - generating views from angles never captured. NeRF discretizes 3D space into voxels, each with RGB color and opacity, using rays cast from camera origins through image pixels. The rendering formula computes final color by summing contributions from all sampled voxels, weighted by transmittance (based on Lambert-Beer law). NeRF uses a two-branch MLP where the first branch predicts density from position, and the second predicts color from position and viewing direction. Fourier features (sine/cosine functions) solve high-frequency modeling limitations. NeSDF improves mesh extraction by predicting Signed Distance Function instead of density. Instant Neural Graphics Primitives (INGP) achieves faster training through multi-resolution encoding. Gaussian Splatting represents scenes using 3D Gaussians defined by center, scale, rotation, color, and opacity. The projected 2D Gaussian is derived using first-order Taylor expansion. Depth ordering and front-to-back compositing handle occlusion. Covariance matrices must be symmetric and positive semidefinite, enforced through eigenvalue decomposition. Rotation uses quaternions. Spherical harmonics determine color dependence on view direction, enabling realistic rendering where objects appear differently from different viewing angles.

Gaussian splatting is a novel 3D reconstruction technique that represents scenes as collections of Gaussian distributions (ellipsoids) instead of using neural networks like Neural Radiance Fields (NeRF). This method achieves dramatically faster rendering speeds (approximately 100 FPS versus NeRF's ~0.2 FPS) by using traditional rasterization techniques with depth buffering and alpha blending, rather than ray marching. The Gaussians are optimized through gradient descent, with cells dynamically splitting, duplicating, and adjusting their size and opacity to better fit the scene geometry. This approach provides more flexibility for real-time applications and interactive manipulation since Gaussians are explicit geometric entities rather than neural network parameters.

This section covers the paradigm shift from implicit to explicit scene representations. MegaDepth (2018) used existing 3D reconstructions as training data to teach neural networks to predict depth from single photos. The Mannequin Challenge provided ideal training data for depth networks on moving people. Neural Radiance Fields (NeRF, 2020) encoded entire scenes into neural network weights, capturing geometry, colors, and lighting. NeRF in the Wild (2021) adapted this to internet-scale photos, successfully disentangling buildings from lighting conditions. However, NeRFs were computationally expensive. 3D Gaussian Splatting (2023) replaced implicit networks with explicit ellipsoidal primitives, enabling real-time rendering at 100 FPS in browsers. Wild Gaussians (2024) applied the NeRF lighting separation technique to this efficient substrate, allowing interactive time-of-day tracking in real-time.

AI-generated 3D technology has evolved from traditional photogrammetry to Neural Radiance Fields (NeRF) and Gaussian Splatting, enabling photorealistic 3D scene generation from multiple 2D images; while NeRF revolutionized 3D representation in 2020 by using neural networks to predict new viewpoints from existing images, Gaussian Splatting (2023) dramatically improved computational efficiency, reducing rendering time from hours to minutes and enabling real-time 1080p rendering at 30fps, though current applications primarily convert existing 2D content rather than generating entirely new 3D scenes from text prompts.

Neural Radiance Fields (NeRF) represent scenes using fully connected deep networks that take 5D coordinates (spatial location and viewing direction) as input and output volume density and view-dependent radiance. The network is trained using images with known camera poses and volume rendering techniques to project density and colors onto images. This approach enables photorealistic novel view synthesis of complex scenes with complicated geometry and appearance. The 3D Gaussian Splatting paper represents a recent advancement that enables real-time rendering of such scenes.

3D Gaussian Splatting is a photorealistic 3D reconstruction technique that converts video footage into detailed point clouds, which can be imported into Unreal Engine 5 using plugins like Kenji's 3D Gaussians Plugin to convert them into Niagara particles for real-time rendering; this method excels at preserving fine details like humans and small objects that NeRF struggles with, though it has limitations such as faded backgrounds and floating ellipsoids that require post-processing and cropping tools to enhance visual quality and performance.

Oculus has invested years in building world-class VR integration for Unity and Unreal Engine game development platforms. These integrations now support Oculus Quest development, meaning developers using these engines can build for Quest with existing tools and workflows. The Superhot demo shown at Connect was ported from Rift to Quest by the Superhot team in just a few weeks, demonstrating the efficiency of this cross-platform approach. Additionally, games can be developed using Unity for Quest and then built for Rift with no code changes, streamlining the development process.

The production digitizes real-life game boards by taking hundreds of detailed photographs and reconstructing them into dense, high-polygon 3D models. These models are then integrated into Unreal Engine 5, allowing the team to render scenes efficiently while maintaining the authentic visual details of the original tabletop setup.

This tutorial demonstrates how to export 3D models from games running on Unreal Engine and Unity engines. For Unreal Engine, use UModel to extract assets, convert PSKX format to GLTF2, import into Blender, export as FBX, and import into Unreal Engine with proper material and collision configuration. For Unity games, use AssetStudio to locate and extract .unity3d files, export objects as OBJ format, and import into Unreal Engine. The process involves proper texture mapping, collision setup, and material creation to ensure the exported models function correctly in the target engine.

Unreal Engine is a powerful 3D game engine developed by Epic Games in 1998 that has evolved beyond game development to become a versatile platform used across multiple industries including architectural visualization, film and television production, broadcasting, training and simulation, and VR applications. The engine's key strengths include its real-time rendering capabilities, physically based rendering (PBR) technology for photorealistic results, user-friendly interface, and extensive toolset that supports developers from indie studios to major AAA studios like Ubisoft and EA.

Ground Control Points (GCPs) are marked physical reference points on the ground with known geographic coordinates that allow drone-captured photogrammetry data to be tied to real-world locations, enabling highly accurate orthomosaic maps without requiring expensive RTK-enabled drones; the process involves surveyors setting GCPs using GPS equipment, drone operators placing visible aerial targets over these points during flight missions, and then using photogrammetry software to align the captured images to the known coordinates, which can achieve accuracy comparable to RTK systems at a significantly lower cost.

To add ground control points in Drone to Map: (1) Access the Control Manager through the import controls option; (2) Import GCP coordinates (automatically detects coordinate system); (3) Use the Image Link Editor to associate each GCP with corresponding image positions; (4) Click on each GCP location in the image and select the matching ground control point; (5) Repeat for multiple images (typically 3-5 GCPs per image position). Important: Use different image positions for each GCP rather than clicking the same image repeatedly, as all photos from one position have identical angles and won't provide proper photogrammetric constraints.

Ground Control Points (GCPs) are identifiable spots with known coordinates used to anchor photogrammetric data to a coordinate system; they must be placed according to specific rules including even distribution across the site (at least 3 GCPs), visibility from the air, and placement at corners, center, and elevation extremes; their positions are measured using RTK or PPK-capable GNSS receivers like the Reach RS2, with the workflow involving configuring RTK links, creating survey projects, setting pole height, measuring each GCP with the pole leveled, and exporting coordinates for import into photogrammetry software.

Ground Control Points are essential reference markers used in aerial mapping to achieve high accuracy. Surveyors use total stations, levels, and GNSS receivers to collect precise coordinates for these control points. These physical targets are placed on the ground at known locations and serve as reference points that aerial images can be geo-referenced to. The science of combining aerial images with ground control points to produce high-accuracy imagery is called photogrammetry. GCPs allow the drone-captured imagery to be aligned with real-world coordinates, transforming raw images into accurate maps.

Ground Control Points (GCPs) are reference points with known real-world coordinates that are used to georeference drone imagery and improve the accuracy of 3D models. In OpenDroneMap, GCPs are processed by creating a text file containing the coordinate system (e.g., WGS84 UTM Zone 23S), followed by lines with point names and X, Y, Z coordinates separated by commas. The workflow involves loading photos into the GCP interface, linking each GCP to its corresponding location in each photo, exporting the linked data, and then processing the images with the GCP file to correct positional errors, particularly in the Z-axis (elevation) where drone GPS data typically shows significant inaccuracies.