Photogrammetry vs NeRF: Key Differences in 3D Reconstruction Explained

Integrating assets into Unreal Engine involves launching the Photo Studio template, configuring project settings, and importing FBX files. Imported meshes receive default materials with diffuse textures on base color and normal maps on normal inputs. Manual texture assignment connects imported files to material graphs. Positioning and scaling prepare assets for scene integration. This pipeline demonstrates complete photogrammetry-to-real-time rendering workflow, transforming mobile phone photographs into interactive game assets ready for visualization and presentation.

Effective NeRF creation follows photogrammetry best practices: take clear stills with incremental movement, maintain significant photo overlap, walk around objects in dome patterns, and capture from multiple angles including top-down views. The COLMAP pipeline processes image datasets through structure from motion to estimate camera poses. Users run a Python script that analyzes images and generates a transform.json file containing metadata for each photo including filename, sharpness, and 3D camera position. Training time depends on resolution and count, with approximately 99 photos taking under 10 minutes on mid-range hardware.

This workflow covers integrating scanned models into Unreal Engine: (1) Import the exported FBX model into Unreal Engine, (2) Remove specular highlights from the material to prevent unwanted shine, (3) Install and configure the Metahuman plugin, (4) Create a new Metahuman identity and select components from mesh, (5) Choose an appropriate body type and set to neutral pose, (6) Properly frame the subject with approximately 20-degree field of view to minimize distortion, (7) Promote the frame for tracking and solve for the Metahuman character, (8) Upload the solved character to Metahuman Creator for further customization.

Photogrammetry is a technique where real-world objects are photographed and converted into 3D digital assets for game development. In Unreal Engine 5, developers can scan actual items, render them, and integrate them into virtual environments, creating highly realistic scenes. The process involves photographing real objects, processing them through specialized software, and placing them within the 3D world. This technique is particularly effective for creating realistic textures and environmental details that would be difficult to model manually. The presenter demonstrates this with examples of villages and other environments, showing how photogrammetry creates convincing realism by capturing authentic surface details and lighting characteristics from the real world.

This video demonstrates the complete photogrammetry workflow for creating photorealistic 3D environments in Unreal Engine 5, covering the entire pipeline from field scanning with a Sony Alpha 7R IV camera (3,300 photos), through RealityCapture mesh generation (1.3 billion polygons), ZBrush retopology and UV unwrapping, Marmoset Toolbag texture baking, Substance Painter cleanup with Nano Banana AI assistance, to final UE5 environment assembly using PCG vegetation, Foliage Mode, material blending, and landscape sculpting.

Dynamic NeRF extends traditional NeRF to handle scenes where objects move or change over time. Traditional NeRF can only render static scenes, but Dynamic NeRF incorporates time as an additional input dimension (6 inputs instead of 5). The system uses two key components: Canonical Space (a time-invariant reference representation at t=0) and Deformation Field (which predicts offsets to map points from actual time t to canonical space). This separation allows learning stable geometry and appearance in the canonical space while modeling dynamics through the deformation field. The system can render novel views while accounting for temporal changes, essential for applications like autonomous driving where environments constantly change.

NSF (Neural Scene Flow Fields) extends NeRF to handle dynamic scenes where objects move between frames. While NeRF works well for static objects captured from multiple viewpoints, NSF addresses the limitation of handling scenes with moving objects. NSF takes video sequences as input (multiple frames showing the same scene from the same camera position but at different times) and learns to synthesize images from arbitrary time points within the video. NSF extends NeRF's input structure by adding a temporal dimension, including 3D position coordinates, viewing direction vector, frame index, and scene flow vector. Scene flow is the 3D extension of optical flow, describing how 3D points move between consecutive frames. The output includes RGB color values, density values, and scene flow vectors. NSF uses a photometric loss function that compares rendered images with actual video frames, computing the loss by transforming points using scene flow vectors and comparing rendered pixel values with actual pixel values in the target frame.

Neural Radiance Fields (NeRF) represents scenes as a continuous volumetric function defined over 5D coordinates (3D position + viewing direction + time). The model outputs volume density and viewing-dependent RGB color for each sample point. For dynamic scenes, NeRF is extended by including time in the domain and explicitly modeling 3D motion through scene flow fields. The model predicts time-dependent reflectance and densities at input time indices, along with forward and backward 3D scene flow fields that denote 3D offset vectors showing where a point moves to at the next and previous time steps. This allows modeling of moving objects and their trajectories across time.

The model consists of dynamic and static feature structures and a template NeRF module. For the dynamic feature structure, both 3D position X and time t are fed as inputs, along with cell identification Z. For the static feature structure, only 3D position X is used as input. These treated features and address information are then fed to the template NeRF that outputs RGB color C and volume density Sigma. This separation allows the model to capture both time-varying and time-invariant characteristics of dynamic scenes.

Neural Radiance Fields can be extended to model dynamic scenes by incorporating time dependency into the network architecture. This approach, called Neural Radiance Fields for Dynamic Scenes, uses a semi-static field component to model large changes that occur over time (such as moving objects), and a dynamic component to model motion occurring within individual frames. The reference frame can be attached to either the scene or the moving actor, depending on the application requirements.

Neural Radiance Fields (NeRF) represent a breakthrough in computer vision that synthesizes novel 3D views from 2D image collections. This technology uses neural networks to optimize continuous volumetric scene functions, enabling reconstruction of complex environments from minimal input—typically 4-6 carefully captured images. Unlike traditional photogrammetry requiring hundreds of images, NeRF dramatically reduces capture time while maintaining high fidelity. The technology approximates real-world light behavior through inverse rendering, allowing automatic conversion from raw images to renderable 3D assets without manual point cloud processing. This represents a paradigm shift in 3D content creation, potentially replacing conventional scanning workflows with AI-driven reconstruction.

By appending the viewing direction to the input coordinates, NeRF can model non-Lambertian reflectance (shiny, specular surfaces). Without view direction, the model would assume Lambertian behavior where surface color appears the same regardless of viewing angle. With view direction, the model learns that the same physical point in space can appear red under one viewing angle and shiny under another, enabling realistic rendering of glossy and reflective surfaces.

NeRFs can be created from three sources: photos of real things, rendered CG images (using Blender plugins that automatically generate camera poses), or edited input images (painting people out, adding CG elements). The first NeRFs generated were all synthetic data. You can render Blender objects and overlay them on photos, then feed back into NeRF to create photographic scenes with CG elements. Research is combining NeRFs with photogrammetry - using NeRFs for surface reconstruction (hair, carpet fuzz) and instructing networks to optimize surface properties. The goal is creating 3D geometry in shaders that matches what NeRFs saw, essentially creating traditional 3D assets from NeRF data. This represents the convergence of NeRF and photogrammetry approaches - they work from opposite ends and meet in the middle to create fantastic 3D scenes that can be rigged, animated, and edited in traditional 3D software.

NeRF (Neural Radiance Fields) represents a volumetric neural rendering approach where a 3D scene is represented as a volume (like a cube filled with smaller cubes). Neural networks decide what type of block/pixel should be placed at each coordinate based on viewing direction. By training these networks to match real images, the system learns to reconstruct 3D scenes that can be viewed from new camera positions not present in the original training data.