NVIDIA Instant NGP Tutorial: Creating a NeRF Movie from Video

NeRFs can be created from three sources: photos of real things, rendered CG images (using Blender plugins that automatically generate camera poses), or edited input images (painting people out, adding CG elements). The first NeRFs generated were all synthetic data. You can render Blender objects and overlay them on photos, then feed back into NeRF to create photographic scenes with CG elements. Research is combining NeRFs with photogrammetry - using NeRFs for surface reconstruction (hair, carpet fuzz) and instructing networks to optimize surface properties. The goal is creating 3D geometry in shaders that matches what NeRFs saw, essentially creating traditional 3D assets from NeRF data. This represents the convergence of NeRF and photogrammetry approaches - they work from opposite ends and meet in the middle to create fantastic 3D scenes that can be rigged, animated, and edited in traditional 3D software.

NeRF (Neural Radiance Fields) reconstructs 3D scenes by treating space like pixel art, assigning colors to 3D coordinates so that viewing from specific angles reproduces original photographs. The system must predict view-dependent colors due to lighting and reflections. Traditional approaches stored all scene information within neural network parameters, requiring backpropagation across millions of weights and taking hours or days to train. The breakthrough came by distributing parameters between the neural network and external memory structures associated with each spatial coordinate, enabling localized updates that reduced training time from hours/days to mere seconds.

This segment covers the complete 3D asset pipeline from Blender to Unreal Engine. The workflow includes: creating indented logo effects by contracting selections (3 pixels) and painting white centers; exporting UV layouts as PNG files; applying UV-mapped textures in Blender by changing mapping coordinates from Generated to UV; assigning different materials to different UV islands; exporting the model as an FBX file; importing the FBX into Unreal Engine's Content Browser; scaling the model (by factor of 50) since Unreal uses different unit scales (1 unit ≈ 1 cm); creating materials by dragging UV textures into the material editor; connecting textures to Base Color inputs; adding Constant nodes for metallic values; and applying materials to the imported model. The tutorial demonstrates the complete 3D asset pipeline including UV unwrapping, texture creation, material assignment, and engine integration.

NeRF's capabilities extend beyond basic reconstruction to sophisticated visual effects and semantic manipulation. Blender integration enables compositing by exporting camera trajectories to Blender, then rendering and compositing with NeRF scenes for effects like portals and transitions. Language-guided scene editing represents an emerging frontier, allowing users to modify reconstructions through natural language instructions—adding mustaches, transforming objects into marble statues, or changing appearances. The ultimate vision involves 3D-consistent semantic integration where scenes can be searched and edited using natural language, bridging text-based and visual content understanding. These advances demonstrate NeRF's transition from research demonstration to professional production tool, enabling creative applications previously requiring complex pipeline integrations.

This video tutorial demonstrates the complete workflow for importing 3D assets from Blender into Unreal Engine 4, covering proper mesh smoothing techniques using MCT space (Mickelson Tangent space) to ensure smooth and flat areas render correctly, scale management through FBX unit settings (defaulting to centimeters), and character rigging using Rigify with the unrigify script for proper skeletal mesh import and animation retargeting.

Dynamic neural rendering extends static NeRF to handle moving subjects by combining multiple conditioning modalities. The pipeline tracks faces using 3D morphable models to reconstruct rotation, translation, camera intrinsics, and 37-dimensional expression parameters. Expression conditioning feeds these vectors into the radiance field network, providing cross-expression constraints that enable animation. Per-frame latent codes encode additional dynamics beyond face expressions, improving results for body movements. This approach enables full 4D avatar control—changing both head pose and facial expressions—while maintaining photorealistic rendering quality, solving limitations of earlier methods that couldn't handle dynamic content.

This section traces the evolution of NeRF from its 2020 origins to modern implementations. It addresses original limitations: slow training times requiring days, static object constraints, incomplete viewpoint coverage requirements, and limited animation support. The presentation details how Instant NeRF solved the speed problem using hash table-based position encoding, reducing training to approximately 10 seconds. It then explains Dynamic NeRF, which incorporates time as an input dimension to handle moving subjects by predicting how object positions shift between frames, enabling coherent reconstructions of dynamic scenes and animations.

Dynamic NeRF extends traditional NeRF to handle scenes where objects move or change over time. Traditional NeRF can only render static scenes, but Dynamic NeRF incorporates time as an additional input dimension (6 inputs instead of 5). The system uses two key components: Canonical Space (a time-invariant reference representation at t=0) and Deformation Field (which predicts offsets to map points from actual time t to canonical space). This separation allows learning stable geometry and appearance in the canonical space while modeling dynamics through the deformation field. The system can render novel views while accounting for temporal changes, essential for applications like autonomous driving where environments constantly change.

Deformable Neural Radiance Fields (D-NeRF) is a method that reconstructs photorealistic 3D human models from casually captured selfie videos by using a deformation field that transforms rays from different observations to intersect at a shared canonical coordinate frame, solving the problem of non-rigid scene reconstruction that traditional NeRF cannot handle.

D-NeRF (Dynamic Neural Radiance Field) is a neural rendering technique that extends traditional NeRF to handle dynamic objects by introducing a deformation network that predicts spatial displacement vectors (delta x, delta y, delta z) based on time, which are then added to canonical coordinates before inputting to the original NeRF network; this allows the model to reconstruct smooth 3D representations of moving objects from sparse multi-view images captured at different time points, achieving superior results compared to naive approaches that simply add time as an input dimension.

Advanced optimization includes: (1) Balancing camera AA samples with other parameters since high camera AA rebalances others, (2) Using Arnold denoiser to reduce camera AA samples (8→2) while maintaining quality, (3) Enabling adaptive sampling with minimum AA 2, maximum 10, and threshold 0.005. These techniques reduce render time from 1 minute 43 seconds to 44 seconds, achieving approximately 8x speedup while maintaining clean results.

This section covers advanced lighting configuration and render optimization. Key topics include: (1) Changing light color and intensity through the multiplier parameter (brightness) and color settings, with shadow color always remaining black; (2) Editing multiple lights simultaneously by selecting them with Shift key and creating a group for batch editing; (3) Sanca configuration as a self-illuminating element that significantly impacts render time; (4) Understanding render time factors, with sanca being the primary factor that increases render duration; (5) Acceptable render time benchmarks of approximately 1 hour for professional work; (6) PNG vs JPEG format selection, with PNG recommended for final renders to preserve image quality without compression artifacts.

Camera setup requires matching focal lengths (70-80mm for close-ups), adjusting aperture blades to eight with ratio two, and using the Lens Sim add-on for realistic lens simulation. HDRI lighting serves as effective fill light when natural references are unavailable, downloaded from Poly Haven in EXR format. Area lights simulate window illumination by rotating toward targets and adjusting spread values for sharpness. Optimal render settings include noise threshold at 0.075, samples at 1000+, and resolution at 200%. These technical parameters ensure final renders capture all accumulated detail while maintaining photorealistic quality.

This section covers the Photographer add-on for advanced camera controls including manual exposure, aperture, ISO, and shutter speed, setting real-world sun positions using the Sun Position add-on, adjusting Filter Glossy for caustics, enabling caustics for glass and water, using subsurface scattering for translucent materials, and understanding light portals for performance optimization. It also covers gobos for projected light patterns, fixing glass transparency issues, adding glass absorption for realistic tinting, and using the Blackbody node for accurate light colors based on Kelvin temperature.

This section explores advanced rendering optimization strategies. The instructor demonstrates the trade-off between render quality and speed, showing how increasing sample counts reduces graininess but extends render times. The gather type selection (Ray Trace vs Approximate) is explained, with Approximate offering faster results for complex geometry despite some inaccuracies. The video concludes this section by saving multiple comparison images (001-005.png) to document the progression from basic to optimized renders, providing visual evidence of the improvements achieved through proper ambient occlusion configuration.